Diffusion Feedback Helps CLIP See Better

## Abstract

Abstract Contrastive Language-Image Pre-training (CLIP), which excels at abstracting open-world representations across domains and modalities, has become a foundation for a variety of vision and multimodal tasks.
However, recent studies reveal that CLIP has severe visual shortcomings, such as which can hardly distinguish orientation, quantity, color, structure, etc .
These visual shortcomings also limit the perception capabilities of multimodal large language models (MLLMs) built on CLIP.
The main reason could be that the image-text pairs used to train CLIP are inherently biased, due to the lack of the distinctiveness of the text and the diversity of images.
In this work, we present a simple post-training approach for CLIP models, which largely overcomes its visual shortcomings via a self-supervised diffusion process.
We introduce DIVA , which uses the DI ffusion model as a V isual A ssistant for CLIP.
Specifically, DIVA leverages generative feedback from text-to-image diffusion models to optimize CLIP representations, with only images (without corresponding text).
We demonstrate that DIVA improves CLIP’s performance on the challenging MMVP-VLM benchmark which assesses fine-grained visual abilities to a large extent ( e . g ., 3-7% ↑), and enhances the performance of MLLMs and vision models on multimodal understanding and segmentation tasks.
Extensive evaluation on 29 image classification and retrieval benchmarks confirms that our framework preserves CLIP’s strong zero-shot capabilities.
The code will be available at https://github.com/baaivision/DIVA .

## 1 Introduction

Contrastive language-image pre-training (CLIP) (Radford et al., ) has been widely applied to various multimodal understanding and generation tasks, including open-domain image classification (Sun et al., ; Zhang et al., ; Zhu et al., ), text-to-image retrieval (Luo et al., ; Baldrati et al., ; Sain et al., ),
visual grounding (Wang et al., ; Yu et al., ; Wang et al., ; ),
and text-to-image generation (Frans et al., ; Bar-Tal et al., ; Rombach et al., ; Crowson et al., ; Ramesh et al., ; Vinker et al., ).
This widespread application is due to CLIP’s excellent visual representation ability, learned from large-scale data. Thus, enhancing CLIP’s representation and capabilities is crucial for advancing downstream tasks.

Since the introduction of CLIP (Radford et al., ), numerous subsequent studies on CLIP models have emerged in recent years.
These studies have utilized training techniques pre-train (Sun et al., ; ; Fang et al., ; Xu et al., ; Zhai et al., ; Shi et al., ) and fine-tune (Wei et al., ; Zhang et al., ) CLIP models, achieving improved performance and unlocking new abilities. However, these approaches still suffer from unavoidable limitations, as they heavily rely on image-text data pairs and cannot work on image-only data.

Figure: Figure 1: Left: The existing CLIP models mostly suffer from the inability to distinguish visual details. After enhancing the visual capabilities with our DIVA, the sensitivity of CLIP to visual details has greatly improved. Right: Our proposed DIVA consistently boosts the performance of various CLIP models (Radford et al., ; Fang et al., ; Xu et al., ; Zhai et al., ) on MMVP-VLM benchmark that evaluates the visual capabilities of vision-language models.
Refer to caption: extracted/5810727/images/introduction.png

As noted by recent works (Kim et al., ; Zeng et al., ; Zhang et al., ; Tong et al., ; ), despite its excellent zero-shot performance, CLIP suffers from certain perceptual understanding limitations due to the contrastive learning paradigm and the noisy image-text pairs used in training. These limitations include an inability to accurately comprehend long texts and to perceive fine-grained differences in similar images. While some studies have attempted to address the text comprehension issue (Zhang et al., ), research on improving CLIP’s fine-grained visual perception remains underexplored. The ability to perceive visual detail is crucial for foundation models, and the lack of this capability in CLIP directly affects the performance of vision and multimodal models that use CLIP as a vision encoder (Tong et al., ; ).

Thus, in this work, we focus on addressing CLIP’s inability to distinguish fine-grained visual details via self-supervised learning (SSL) paradigm.
Based on text-to-image diffusion models generating realistic images with much details, we explore leveraging generative feedback from diffusion models to optimize CLIP representations.
By conditioning diffusion models with CLIP’s densely recapped visual features and applying reconstruction loss to CLIP optimization,
we leverage the DIffusion model as a Visual Assistant for CLIP, thus the name of our approach, DIVA.
Our results highlight that DIVA greatly enhances CLIP’s performance on MMVP-VLM benchmark measuring visual abilities of V-L models, and improves multimodal large language models (MLLMs) and vision models on multimodal and vision understanding tasks.
Besides, DIVA maintains CLIP’s excellent zero-shot performance on 29 image classification and retrieval benchmarks.

Our main contributions can be summarized as follows:

- •
Concentrating on overcoming CLIP’s visual shortcomings in perceiving fine-grained details, we present the first work to exploit the potential of leveraging generative feedback from text-to-image diffusion models to optimize CLIP model’s discriminative representations.
- •
We propose a simple self-supervised framework DIVA for CLIP’s representation optimization. Coupled with our visual dense recap scheme, DIVA conditions diffusion models with dense visual features from CLIP and incorporates image reconstruction loss for optimization.
- •
Our DIVA greatly boosts CLIP’s visual perception capability and improves its performance on MMVP-VLM benchmark, further enhancing MLLMs and vision models on multimodal and visual understanding tasks. Meanwhile, our results on 29 image classification and retrieval benchmarks show that DIVA maintains CLIP’s original excellent zero-shot performance.

## 2 Related Work

CLIP Models & MLLMs.
The introduction of CLIP (Radford et al., ) has significantly advanced multimodal learning. Since its debut, a series of CLIP models have emerged (Sun et al., ; Fang et al., ; Xu et al., ; Zhai et al., ), enhancing performance and unlocking new capabilities through improved pre-training techniques and model architectures.
On this basis, CLIP has been widely adopted as a foundation model, serving as a backbone for various applications such as image segmentation (Li et al., ; Xu et al., ; Shan et al., ; Xu et al., ; Liang et al., ; Zhou et al., ), object detection (Gu et al., ; Li et al., ; Subramanian et al., ) and video understanding (Bose et al., ; Lin et al., ; Castro & Heilbron, ; Xu et al., ; Rasheed et al., ; Tang et al., ). Its ability to align language and vision has led to superior results on these tasks compared to traditional methods.
Moreover, CLIP has driven the development of MLLMs (Liu et al., ; ; Sun et al., ; ). Combining strong visual understanding with advanced large language models facilitates more sophisticated interactions between vision and language. Recent works have highlighted inherent visual flaws in the CLIP models and MLLMs using CLIP as the visual encoder (Tong et al., ; ). To address this, some research has incorporated multiple vision encoders to achieve more precise and comprehensive visual perception (Kar et al., ; Jiang et al., ; Tong et al., ). However, this approach increases computational costs and memory usage. There has been no research directly enhancing CLIP’s visual perception capabilities to better serve MLLMs. Thus, the main focus of our work is to fundamentally overcome CLIP’s visual perception shortcomings, directly benefiting both vision models and multimodal MLLMs that use CLIP as a backbone.

Diffusion Models for Representation Learning.
Diffusion models (Ho et al., ; Song et al., ) have made remarkable progress in various generative tasks, such as image generation (Rombach et al., ; Saharia et al., ; Betker et al., ; Zheng et al., ), video generation (Singer et al., ; Blattmann et al., ; Junhao Zhang et al., ; Ho et al., ), editing (Meng et al., ; Hu et al., ; Mou et al., ), etc.
Apart from research above, there are also many works focus on employing diffusion models for representation learning.
Some of works leverage the intermediate activation of pre-trained diffusion models for different downstream tasks, including classification (Xiang et al., ), semantic segmentation (Baranchuk et al., ), panoptic segmentation (Xu et al., ), depth estimation (Zhao et al., ), etc.
Other works (Hudson et al., ; Pan et al., ) train their own diffusion models coupled with meticulously devised modules to further boosting the representation capabilities.
Besides, Diffusion-TTA (Prabhudesai et al., ) aims to adapt pre-trained vision encoders to samples in testing set using feedback from a diffusion model.
Additionally, some methods (Guo et al., ; Trabucco et al., ; Tian et al., ; Azizi et al., ) utilize diffusion models to generate synthetic data, which is then adopted to enhance the representation capabilities of corresponding models.
In contrast, in our work, we mainly focus on exploring the potential of enhancing the original discriminative representations of CLIP models by directly utilizing generative feedback from the diffusion models.
Additionally, we aim to leverage the diffusion models to break free from the constraints of paired image-text data and construct a self-supervised framework to improve CLIP’s visual perception capabilities.

## 3 Enhancing CLIP’s Representations via Diffusion Feedback

In this section, we present our DIVA, an effective framework for boosting CLIP’s visual perception capabilities with a pre-trained conditional diffusion model.
We first discuss CLIP’s visual deficiencies in perceiving details and generative diffusion models as preliminaries in Sec. 3.1.1 and Sec. 3.1.2 respectively.
Then the overall architecture of DIVA is illustrated in Sec. 3.2, followed by our well-designed visual dense recap strategy for better unleashing the power of DIVA in Sec. 3.3.

### 3.1 Preliminaries

#### 3.1.1 CLIP’s Visual Deficiencies

Thanks to the excellent representations learned through pre-training on massive data, CLIP (Radford et al., ) exhibits outstanding generalization capabilities and is widely applied in the V-L understanding domain.
However, CLIP is not without its flaws.
As highlighted in the study (Tong et al., ), CLIP struggles to distinguish detailed differences between two images that are obviously distinct to human observers.
This deficiency mainly stems from two aspects:
1) Training Paradigm: The original contrastive learning strategy of CLIP aims to minimize the distance between positive pairs and maximize the distance between negative pairs of visual class tokens and textual semantics,
resulting in visual perception bias that mainly focuses on high-level semantic information while overlooking visual details such as orientation, quantity, color, and structure.
Consequently, CLIP sometimes encodes visually different images into similar embeddings, making it difficult to differentiate these images’ subtle variations.
2) Data Format: The text in the image-text pairs used to train CLIP is limited in length. As pointed out by (Zhang et al., ), although the length of the text token is restricted to 77, CLIP’s actual effective text length is less than 20. Therefore, the textual data in these image-text pairs inherently lacks descriptions of the visual details in the corresponding positive sample images. This fundamental limitation of the training data also leads to CLIP’s inability to adequately perceive visual detail information.

#### 3.1.2 Generative Diffusion Models

A diffusion model learns to model a probability distribution by reversing a process that progressively adds noise to the image.
Given an image sample $x_{0}$ drawn from an underlying probability distribution $p(x)$, a forward diffusion process defines a Markov chain to gradually add random Gaussian noise $\mathbf{\epsilon_{t}}\in\mathcal{N}(\mathbf{0},\mathbf{I})$ to the original sample $\mathbf{x_{0}}$:

$$ $\mathbf{x_{t}}=\sqrt{1-\beta_{t}}\mathbf{x_{t-1}}+\sqrt{\beta_{t}}\mathbf{ \epsilon_{t}},\quad t=1,\ldots,T$ (1) $$

Here, $T$ denotes the number of diffusion steps, and $\beta_{t}\in(0,1)$ is a predefined time-dependent variance schedule. As the $T$ becomes large enough, $\mathbf{x_{T}}$ is close to $\mathcal{N}(\mathbf{0},\mathbf{I})$.
The transition equation can be reformulated as follows by leveraging the additive property of Gaussian distribution:

$$ $\mathbf{x_{t}}=\sqrt{\bar{\alpha}_{t}}\mathbf{x_{0}}+\sqrt{1-\bar{\alpha}_{t}} \mathbf{\epsilon},\quad t=1,\ldots,T$ (2) $$

in which $\alpha_{t}=1-\beta_{t}$ and $\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$.
On this basis, the image sample $x_{0}$ can be iteratively generated from a random noise $\mathbf{x_{T}}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ by reversing the forward diffusion process:

$$ $\mathbf{x_{t-1}}=\frac{1}{\sqrt{\alpha_{t}}}(\mathbf{x_{t}}-\frac{1-\alpha_{t} }{\sqrt{1-\bar{\alpha}_{t}}}\mathbf{\epsilon_{\phi}}(\mathbf{x_{t}},t))+\sigma _{t}\mathbf{\epsilon},\quad t=T,\ldots,1$ (3) $$

$\mathbf{\epsilon_{\phi}}$ is a denoising neural network trained to predict $\mathbf{\epsilon}$ in the forward diffusion process and $\sigma_{t}$ is the posterior noise variance.
A commonly used training objective for a diffusion model $\mathbf{\epsilon_{\phi}}$ is:

$$ $L(\phi)=\mathbb{E}_{t,\mathbf{x_{0}},\epsilon}\left[\|\mathbf{\epsilon}- \mathbf{\epsilon_{\phi}}(\sqrt{\bar{\alpha}_{t}}\mathbf{x_{0}}+\sqrt{1-\bar{ \alpha}_{t}}\mathbf{\epsilon},t)\|^{2}\right]$ (4) $$

Besides, the diffusion models can be easily extended to conditional generation by incorporating a condition $\mathbf{c}$ into $\mathbf{\epsilon_{\phi}}$, in which $\mathbf{c}$ can be a class label, a text prompt or a reference image. Thus, the training objective should be modified to:

$$ $L(\phi)=\mathbb{E}_{t,\mathbf{x_{0}},\epsilon,\mathbf{c}}\left[\|\mathbf{ \epsilon}-\mathbf{\epsilon_{\phi}}(\sqrt{\bar{\alpha}_{t}}\mathbf{x_{0}}+\sqrt {1-\bar{\alpha}_{t}}\mathbf{\epsilon},t,\mathbf{c})\|^{2}\right]$ (5) $$

### 3.2 Overall Structure of DIVA

Figure: Figure 2: Overall architecture of our DIVA. Given an image $x_{0}$, the CLIP model $\theta$ encodes the visual features as main part of condition c, then the generative diffusion model $\phi$ predicts the added noise $\epsilon$ taking the noisy image $x_{t}$ and condition c as input. We optimize the CLIP’s representation by maximizing the image likelihood with the diffusion loss via generative feedback.
Refer to caption: extracted/5810727/images/methodology.png

As illustrated in Fig. 2, DIVA mainly consists of two parts: the CLIP model to be enhanced in terms of visual perception capabilities, and the pre-trained text-to-image diffusion model providing generative feedback.
Taking an original image as input, the CLIP model encodes the corresponding visual features, which will be combined with the empty text’s embeddings (i.e., [BOS] & [EOS]) from the diffusion model’s text encoder for diffusion’s condition.
Given the image with added noise, the diffusion model attempts to predict the noise added from the previous step to the current step with the aforementioned condition.
This process needs to be repeated N times because, for each image, we will randomly select N states (i.e., two) from the total steps (e.g., 0$\sim$1000 steps) of the diffusion model for optimization.
The corresponding loss function can be represented as Equation 5.
Keeping all parts’ weights except the CLIP visual encoder frozen, the training objective is simply to minimize the reconstruction loss (i.e., generative guidance).
In this manner, by constraining the diffusion model to more accurately predict the added noise, the CLIP’s original semantic-rich discriminative representations will be gradually optimized into representations with more visual details through diffusion feedback. On this basis, results in Sec. 4.4 demonstrate that our DIVA does not greatly damage the zero-shot performance of the original CLIP models.
The pseudo code of the specific enhancement process can be found at Algorithm 1 in Appendix.

### 3.3 Diffusion’s Condition Design

Visual Dense Recap Scheme.
In our DIVA, the diffusion model’s condition design is pivotal, as it sets the upper limit for enhancing CLIP’s visual capabilities. We introduce a simple yet effective strategy called visual dense recap. Unlike detailed re-captioning of an image’s caption in natural language, our approach performs re-captioning at the level of visual richness by incorporating features from local patch tokens along with the class token into the condition. When only the class token is present, CLIP’s visual features primarily contain strong semantic information, which is insufficient for reconstructing the original image. Consequently, the reconstruction task becomes challenging due to the lack of adequate information, and CLIP cannot learn significantly enhanced representations. By incorporating local patch features, the auxiliary function of the condition is significantly enhanced, allowing the generative feedback to effectively improve CLIP’s visual perception capabilities. We conduct ablation studies in Sec. 4.5 to demonstrate the efficacy of visual dense recap.

Visual Recap Density. Although the visual dense recap scheme appears straightforward, the density of the recap is crucial. If the density is too high (i.e., introducing too many local tokens), the richness of the condition information approaches its maximum, greatly reducing the difficulty of reconstruction task. This results in CLIP’s representation requiring minimal optimization to easily complete the reconstruction, limiting the upper bound of CLIP’s optimized capabilities. Conversely, if the recap density is too low (i.e., retaining only class token or introducing few local tokens), CLIP’s optimization process will struggle with the high difficulty of reconstruction, failing to adequately learn the expected detailed visual representations. This intuitive point is confirmed in Sec. 4.5.
Specifically, ensuring the visual class token is always present in the condition, we introduce randomly selected local token features with approximately 15% and 30% probabilities for OpenAI CLIP (Radford et al., ) at 224 and 336 resolutions.
For the SigLIP ViT-SO-14 (Zhai et al., ) at 224 and 384 image sizes, we incorporate local token features obtained through 1D average pooling with local window sizes of 6 and 10, respectively.
Except that introducing 50% randomly selected patch tokens into the condition for DFN ViT-H-14/378, for the remaining baselines (Fang et al., ; Xu et al., ), we include all local token features for the condition design.
Apart from DFN ViT-H-14/224 and SigLIP ViT-SO-14/224&384 (Zhai et al., ) only using visual class token, all other models incorporate local features consistent with the training stage conditions during inference, combining them with class token to fully leverage the detailed representations captured by the enhanced CLIP.

## 4 Experimental Results

To evaluate the effectiveness of our DIVA and demonstrate its potential to enhance CLIP representations, comprehensive experiments are conducted on multimodal understanding and visual perception tasks, which will be elaborated in the followings.

### 4.1 Implementation Details

DIVA is trained on 8 NVIDIA-A100 80GB GPUs with a global batch size of 640.
We adopt Stochastic Gradient Descent (SGD) optimizer with a learning rate of 1e-4 and momentum of 0.9 to refine the representations of CLIP models via generative feedback.
We only optimize the CLIP models with relatively high-quality Conceptual-3M dataset (Sharma et al., ) for 4600 steps (i.e., nearly 1 epoch) among the training phase, which can already boost CLIP’s performance in a training-efficient manner.
For all experiments, we adjust the parameters of the discriminative CLIP vision encoders and keep the pre-trained diffusion models’ parameters frozen through the training process.

### 4.2 Fine-grained Visual Perception Evaluation

**Table 1: Performance of CLIP based models on various visual patterns of MMVP-VLM benchmark. Our framework greatly overcomes CLIP’s original shortcomings in terms of perceiving visual details. Symbols for visual patterns as (Tong et al., ) are inherited: \faCompass: Orientation and Direction, \faSearch: Presence of Specific Features, \faSync: State and Condition, \faSortNumericUp: Quantity and Count, \faMapPin: Positional and Relational Context, \faPalette: Color and Appearance, \faCogs: Structural and Physical Characteristics, \faFont: Texts, \faCamera: Viewpoint and Perspective.**
| Method | Ours | Image Size | Params (M) | \faCompass | \faSearch | \faSync | \faSortNumericUp | \faMapPin | \faPalette | \faCogs | \faFont | \faCamera | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OpenAI ViT-L-14 |  | 224^2 | 427.6 | 13.3 | 13.3 | 20.0 | 20.0 | 13.3 | 53.3 | 20.0 | 6.7 | 13.3 | 19.3 |
| OpenAI ViT-L-14 | ✔ | 224^2 | 427.6 | 13.3 | 20.0 | 40.0 | 6.7 | 20.0 | 53.3 | 46.7 | 20.0 | 13.3 | 25.9 (+6.6) |
| OpenAI ViT-L-14 |  | 336^2 | 427.9 | 0.0 | 20.0 | 40.0 | 20.0 | 6.7 | 20.0 | 33.3 | 6.7 | 33.3 | 20.0 |
| OpenAI ViT-L-14 | ✔ | 336^2 | 427.9 | 26.7 | 20.0 | 33.3 | 13.3 | 13.3 | 46.7 | 26.7 | 6.7 | 40.0 | 25.2 (+5.2) |
| MetaCLIP ViT-L-14 |  | 224^2 | 427.6 | 13.3 | 6.7 | 66.7 | 6.7 | 33.3 | 46.7 | 20.0 | 6.7 | 13.3 | 23.7 |
| MetaCLIP ViT-L-14 | ✔ | 224^2 | 427.6 | 6.7 | 6.7 | 60.0 | 0.0 | 26.7 | 66.7 | 20.0 | 20.0 | 40.0 | 27.4 (+3.7) |
| MetaCLIP ViT-H-14 |  | 224^2 | 986.1 | 6.7 | 13.3 | 60.0 | 13.3 | 6.7 | 53.3 | 26.7 | 13.3 | 33.3 | 25.2 |
| MetaCLIP ViT-H-14 | ✔ | 224^2 | 986.1 | 13.3 | 20.0 | 53.3 | 33.3 | 13.3 | 66.7 | 33.3 | 13.3 | 40.0 | 31.9 (+6.7) |
| SigLIP ViT-SO-14 |  | 224^2 | 877.4 | 26.7 | 20.0 | 53.3 | 40.0 | 20.0 | 66.7 | 40.0 | 20.0 | 53.3 | 37.8 |
| SigLIP ViT-SO-14 | ✔ | 224^2 | 877.4 | 13.3 | 26.7 | 60.0 | 46.7 | 13.3 | 73.3 | 53.3 | 26.7 | 53.3 | 40.7 (+2.9) |
| SigLIP ViT-SO-14 |  | 384^2 | 878.0 | 20.0 | 26.7 | 60.0 | 33.3 | 13.3 | 66.7 | 33.3 | 26.7 | 53.3 | 37.0 |
| SigLIP ViT-SO-14 | ✔ | 384^2 | 878.0 | 26.7 | 33.3 | 53.3 | 26.7 | 13.3 | 80.0 | 40.0 | 26.7 | 46.7 | 38.5 (+1.5) |
| DFN ViT-H-14 |  | 224^2 | 986.1 | 20.0 | 26.7 | 73.3 | 26.7 | 26.7 | 66.7 | 46.7 | 13.3 | 53.3 | 39.3 |
| DFN ViT-H-14 | ✔ | 224^2 | 986.1 | 20.0 | 20.0 | 80.0 | 40.0 | 46.7 | 66.7 | 46.7 | 20.0 | 53.3 | 43.7 (+4.4) |
| DFN ViT-H-14 |  | 378^2 | 986.7 | 13.3 | 20.0 | 53.3 | 33.3 | 26.7 | 66.7 | 40.0 | 20.0 | 40.0 | 34.8 |
| DFN ViT-H-14 | ✔ | 378^2 | 986.7 | 26.7 | 26.7 | 53.3 | 33.3 | 26.7 | 73.3 | 26.7 | 13.3 | 60.0 | 37.8 (+3.0) |

To validate that our DIVA can effectively mitigate the inherent visual capability deficiencies of CLIP models, we first conduct experiment on various existing CLIP models (Radford et al., ; Fang et al., ; Xu et al., ; Zhai et al., ).
Despite the variations in image resolution, model size, training data and methodology among these CLIP models, our method consistently enhances their performances on the MMVP-VLM benchmark.
As presented in Table 1, our framework achieves the best performance improvements (i.e., ↑4-7%) on OpenAI ViT-L-14 and MetaCLIP ViT-H-14, and even on current best-performing DFN ViT-H-14 our framework realizes a performance gain of nearly 3-5%.
This fully demonstrates that DIVA is both general and effective in enhancing the fine-grained visual perception capabilities of CLIP models.
Notably, through the generative guidance provided by our self-supervised framework that is free from image-text constraints, the perceptual abilities of CLIP models on almost all visual patterns have the potential to be enhanced.

Figure: Figure 3: Qualitative analysis on MMVP-VLM and MMVP benchmark. Left: The prediction results from the OpenAI ViT-L-14 CLIP before & after incorporating DIVA. Right: The prediction results from LLaVA-1.5-7B before & after using our DIVA. The results on both benchmarks show that our framework can greatly enhance CLIP models’ fine-grained visual perception capability and effectively alleviate the hallucination problem.
Refer to caption: extracted/5810727/images/qualitative_mmvp.png

### 4.3 Backbone Enhancement Performance Evaluation

Next, with the help of our DIVA, we further evaluate the performance gains brought by the enhanced CLIP backbones for multimodal understanding and visual perception tasks.

**Table 2: Performance gains achieved by our enhanced CLIP visual backbone for MLLM (i.e., LLaVA-1.5-7B and LLaVA-1.5-13B) on various V-L understanding tasks. By refining the CLIP’s representation with generative feedback, our method mitigates the visual deficiencies in MLLMs (i.e., LLaVA^1.5) and improves original instruction following ability.**
| Method | Ours | LLM | Image<br>Size | MMVP | POPE | MME | MMBench | LLaVA-Wild |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| rand | pop | adv | en | cn |  |  |  |  |  |  |  |
| LLaVA^1.5 |  | Vicuna-7B | 336^2 | 24.7 | 87.3 | 86.1 | 84.2 | 1510.7 | 64.3 | 58.3 | 65.4 |
| LLaVA^1.5 | ✔ | Vicuna-7B | 336^2 | 31.3 | 87.9 | 87.0 | 84.6 | 1500.6 | 66.4 | 60.6 | 66.3 |
| LLaVA^1.5 |  | Vicuna-13B | 336^2 | 30.7 | 87.1 | 86.2 | 84.5 | 1531.3 | 67.7 | 63.6 | 72.5 |
| LLaVA^1.5 | ✔ | Vicuna-13B | 336^2 | 35.3 | 88.1 | 87.4 | 84.8 | 1522.9 | 69.4 | 63.1 | 73.5 |

Enhanced Vision Backbone for MLLMs.
Firstly, we adopt LLaVA-1.5 (Liu et al., ) as the baseline framework to explore the potential of improved visual encoders in MLLM.
LLaVA employs a pre-trained CLIP vision encoder and trains a projector to semantically align visual tokens with textual tokens from large language model (LLM).
To ensure fair comparisons, we train our model with the same setting in LLaVA and evaluate model performance on various multimodal understanding benchmarks (i.e., MMVP (Tong et al., ),
POPE (Li et al., ), MME-Perception (Fu et al., ), MMBench (Liu et al., ), MMBench-CN (Liu et al., ), LLaVA-Bench-in-the-Wild (Liu et al., )).
It can be clearly seen from Table 2 that LLaVA’s performance is greatly boosted by replacing the original CLIP vision encoder to ours.
The big accuracy gains on these benchmarks (except MME) are all thanks to the significant enhancement in CLIP’s visual perception capabilities brought by our DIVA paradigm utilizing generative feedback.

**Table 3: Performance gains achieved by our enhanced CLIP backbone with generative guidance on semantic segmentation task. * denotes the re-implemented results. Boosting CLIP’s ability to perceive fine-grained visual details results in considerable benefits for visual dense prediction task.**
| Method | Backbone | Ours | ADE20K-847 | ADE20K-150 | Pascal Context-459 | Pascal Context-59 |
| --- | --- | --- | --- | --- | --- | --- |
| SAN* | ViT-L-14/224 |  | 10.9 | 29.2 | 14.2 | 55.8 |
| SAN | ViT-L-14/224 | ✔ | 11.0 | 30.2 | 15.4 | 56.7 |
| SAN* | ViT-L-14/336 |  | 11.5 | 30.3 | 14.7 | 56.7 |
| SAN | ViT-L-14/336 | ✔ | 11.5 | 31.8 | 15.7 | 57.8 |

Enhanced Vision Backbone for Fine-Grained Visual Perception.
We also include segmentation task to evaluate the benefits brought by our enhanced CLIP backbones for visual dense prediction task.
We adopt the recent state-of-the-art model in the open vocabulary semantic segmentation field, SAN (Xu et al., ) with CLIP (Radford et al., ) at both 224 and 336 image resolutions, as baselines.
Four commonly used benchmarks (i.e., ADE20K-847/150 (Zhou et al., ) and Pascal Context-459/59 (Mottaghi et al., )
for are employed for performance evaluation.
As shown in Table 3, with the benefit of our generative tuned CLIP backbone, the baseline models achieve considerable performance improvements on most of segmentation benchmarks and do not suffer performance degradation on the remaining one (i.e., ADE20K-847).

### 4.4 Generalization Capability Evaluation

**Table 4: Summary of zero-shot image classification performance on 27 datasets for evaluating model generalization capability. O-1 and M-1 separately represent OpenAI ViT-L-14/224 and MetaCLIP ViT-H-14/224. DIVA greatly improves CLIP’s ability to perceive visual details, while preserving its outstanding generalization capabilities.**
| Method | Ours | ImageNet-1K | ImageNet-V2 | ImageNet-Adv. | ImageNet-Ren. | ImageNet-Ske. | ObjectNet | CIFAR-10 | CIFAR-100 | MNIST | Caltech-101 | SUN397 | FGVC Aircraft | Country-211 | Stanford Cars | Birdsnap | DTD | Eurosat | FER2013 | Flowers-102 | Food-101 | GTSRB | PCam | Pets | Rendered SST2 | Resisc45 | STL10 | VOC2007 | avg. top-1 acc. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| O-1 |  | 75.5 | 69.8 | 70.7 | 87.8 | 59.6 | 69.0 | 95.6 | 75.8 | 76.4 | 86.6 | 67.5 | 31.9 | 31.9 | 77.8 | 51.4 | 55.4 | 60.1 | 49.9 | 79.1 | 93.0 | 50.6 | 52.0 | 93.6 | 68.8 | 64.5 | 99.4 | 77.4 | 69.3 |
| O-1 | ✔ | 75.5 | 69.7 | 70.8 | 87.7 | 59.5 | 69.1 | 95.5 | 76.3 | 76.1 | 86.8 | 67.5 | 31.8 | 31.8 | 77.9 | 51.8 | 55.1 | 60.2 | 49.4 | 78.9 | 93.0 | 50.2 | 53.8 | 93.7 | 67.0 | 64.5 | 99.3 | 77.8 | 69.3 |
| M-1 |  | 78.5 | 72.1 | 69.6 | 91.8 | 68.1 | 73.6 | 98.3 | 86.7 | 81.3 | 89.1 | 74.1 | 48.2 | 34.7 | 87.2 | 68.5 | 69.8 | 55.6 | 54.9 | 80.7 | 92.5 | 62.3 | 56.1 | 94.2 | 71.0 | 72.6 | 99.4 | 77.6 | 74.4 |
| M-1 | ✔ | 78.4 | 71.9 | 69.1 | 91.6 | 67.9 | 73.4 | 98.3 | 86.4 | 81.0 | 89.1 | 74.3 | 47.0 | 34.7 | 87.2 | 67.8 | 69.6 | 55.0 | 55.8 | 80.7 | 92.4 | 62.4 | 54.8 | 94.1 | 70.7 | 73.0 | 99.4 | 77.6 | 74.2 |

**Table 5: Summary of zero-shot text and image retrieval performance on Flickr30K and COCO datasets for evaluating model generalization capability. Our DIVA significantly enhances CLIP’s visual detail perception ability while maintaining its excellent generalization capabilities.**
| Method | Ours | Image Size | Zero-Shot Text Retrieval | Zero-Shot Image Retrieval |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Flickr30K | COCO | Flickr30K | COCO |  |  |  |  |  |  |  |  |  |  |  |
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |  |  |  |
| OpenAI ViT-L-14 |  | 224^2 | 85.1 | 97.3 | 99.0 | 56.4 | 79.4 | 86.6 | 65.2 | 87.3 | 92.0 | 36.5 | 61.0 | 71.1 |
| OpenAI ViT-L-14 | ✔ | 224^2 | 85.3 | 97.3 | 99.0 | 56.7 | 79.7 | 87.0 | 64.4 | 86.9 | 92.0 | 36.6 | 61.0 | 71.3 |
| MetaCLIP ViT-H-14 |  | 224^2 | 89.5 | 98.8 | 99.7 | 65.5 | 85.2 | 91.1 | 76.8 | 93.9 | 96.6 | 48.2 | 72.3 | 81.1 |
| MetaCLIP ViT-H-14 | ✔ | 224^2 | 89.2 | 98.7 | 99.7 | 65.5 | 85.0 | 91.1 | 77.3 | 93.8 | 96.7 | 48.4 | 72.4 | 81.1 |

After comprehensive validation of our approach’s ability to boost CLIP models’ fine-grained visual perception abilities, we conduct a thorough assessment of CLIP model’s original generalization ability.
The details about all benchmarks can be found at Table 9 in Appendix.
Specifically, OpenAI ViT-L-14 (Radford et al., ) and MetaCLIP ViT-H-14 (Xu et al., ), which are widely used and shows the greatest performance gains on MMVP-VLM benchmark, are adopted as our baselines.
We present their zero-shot accuracies on 27 image classification benchmarks in Table 4.
It is evident that our DIVA significantly enhances CLIP models’ representations of fine-grained visual details without adversely affecting the generalization capabilities of the baselines to a large extent.
Furthermore, Table 5 illustrates the comparison of zero-shot image and text retrieval performance before and after incorporating DIVA.
The quantitative results reaffirm that optimizing CLIP models’ representations with DIVA preserves the original great generalization ability.
Given that these tasks heavily rely on the CLIP visual backbone’s global semantic understanding, it is reasonable that our generative guided CLIPs do not achieve much performance improvements on these tasks.

### 4.5 Ablation Study

By taking OpenAI ViT-L-14/224 as the baseline model, we conduct comprehensive ablation studies on MMVP-VLM regarding each pattern and average score, exploring the effect of condition designing, introduced data scale and adopted diffusion models for DIVA.

**Table 6: Ablation study on the condition design for diffusion models. G and L denote visual class token and patch tokens. Compared to using semantic matching constraints from image-text pairs as condition, taking visual features as condition for representation-level optimization is more effective. Furthermore, an appropriate degree of visual dense recap scheme is also crucial for DIVA.**
| Visual Condition | Textual Condition | \faCompass | \faSearch | \faSync | \faSortNumericUp | \faMapPin | \faPalette | \faCogs | \faFont | \faCamera | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ✘ | ✘ | 13.3 | 13.3 | 20.0 | 20.0 | 13.3 | 53.3 | 20.0 | 6.7 | 13.3 | 19.3 |
| ✘ | Real Caption | 6.7 | 13.3 | 26.7 | 20.0 | 6.7 | 53.3 | 33.3 | 13.3 | 20.0 | 21.5 (+2.2) |
| ✘ | Free-Source Phrase | 6.7 | 13.3 | 20.0 | 20.0 | 6.7 | 53.3 | 26.7 | 13.3 | 26.7 | 20.7 (+1.4) |
| only G | Empty Caption | 20.0 | 20.0 | 20.0 | 20.0 | 13.3 | 46.7 | 26.7 | 20.0 | 13.3 | 22.2 (+2.9) |
| G + part_L | Empty Caption | 13.3 | 20.0 | 40.0 | 6.7 | 20.0 | 53.3 | 46.7 | 20.0 | 13.3 | 25.9 (+6.6) |
| G + all_L | Empty Caption | 6.7 | 20.0 | 40.0 | 6.7 | 6.7 | 40.0 | 40.0 | 6.7 | 13.3 | 20.0 (+0.7) |

Effect of Condition Design for Diffusion Models.
We first probe into the influence of diffusion models’ condition designing.
As elaborated in Sec. 3.3, the condition design of diffusion models is vital as it directly determines the upper limit of CLIP models’ enhanced representation quality.
As shown in Table 6, we consider two condition settings: 1) using pure text embeddings as condition, which similar to Diffusion-TTA (Prabhudesai et al., ) (rows 3-4); 2) incorporating densely recapped visual features and empty text’s embeddings as condition (rows 5-7).
Specifically, whether using real caption matched with images from CC-3M dataset or using free-source phrases about image details, guiding CLIP through image-text matching constraints can yield performance gains.
However, since this manner does not originate from the representation level, the achieved gain is not significant.
In contrast, our condition design introduces appropriately densified visual features, constructing a framework that only uses images to achieve self-supervised optimization of CLIP representations and detaching from image-text form constraints.
DIVA helps CLIP achieve the best performance gains (↑6.6%) by introducing partial visual local features coupled with class tokens as condition. Introducing too few or many local tokens results in visual density being too low (row 5) or high (row 7), both of which reduce the achieved performance improvement.

**Table 7: Ablation study on the data scaling property of our DIVA with different data scales. Training time is measured by # gpus$\times$hours. DIVA demonstrates great potential with data scaling properties, where the increase in data volume proportionally enlarges the performance gains.**
| Data Scale | Training Time | \faCompass | \faSearch | \faSync | \faSortNumericUp | \faMapPin | \faPalette | \faCogs | \faFont | \faCamera | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ✘ | ✘ | 13.3 | 13.3 | 20.0 | 20.0 | 13.3 | 53.3 | 20.0 | 6.7 | 13.3 | 19.3 |
| 25% | 16.8 | 6.7 | 13.3 | 20.0 | 13.3 | 6.7 | 46.7 | 53.3 | 13.3 | 13.3 | 20.7 (+1.4) |
| 50% | 32.8 | 13.3 | 13.3 | 40.0 | 13.3 | 6.7 | 26.7 | 53.3 | 20.0 | 13.3 | 22.2 (+2.9) |
| 75% | 49.6 | 6.7 | 26.7 | 40.0 | 13.3 | 6.7 | 53.3 | 53.3 | 6.7 | 6.7 | 23.7 (+4.4) |
| 100% | 66.4 | 13.3 | 20.0 | 40.0 | 6.7 | 20.0 | 53.3 | 46.7 | 20.0 | 13.3 | 25.9 (+6.6) |

Data Scaling Property.
Then, we investigate the potential data scaling property of our DIVA by taking CC-3M dataset as training data.
The corresponding results are presented in Table 7.
It is obvious that the CLIP model’s performance on MMVP-VLM benchmark is consistently improved with more training samples.
As the ratios of introduced samples continue to rise, there’s no sign of diminishing gains in accuracy, suggesting that our framework has great potential with continually scaled up training data.
Noticeably, by integrating 100% training data, our method greatly boosts CLIP’s visual perception capability (i.e., approximately ↑7%) with 66.4 # gpus $\times$ hours.
It means if 8 gpus are available, DIVA only need 8.3 hours training time to realize considerable performance gains for CLIP models.
Besides, this setting results in an adaptation time of roughly 0.01 seconds per sample, which fully proves the efficacy of our method.

**Table 8: Ablation study on the adopted generative diffusion models in our DIVA. Training time is measured by # gpus$\times$hours. Our framework is not sensitive to the version of stable diffusion models, consistently brings representation enhancement for CLIP models.**
| Method | Diffusion Resolution | Training Time | \faCompass | \faSearch | \faSync | \faSortNumericUp | \faMapPin | \faPalette | \faCogs | \faFont | \faCamera | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ✘ | ✘ | ✘ | 13.3 | 13.3 | 20.0 | 20.0 | 13.3 | 53.3 | 20.0 | 6.7 | 13.3 | 19.3 |
| DiT-XL/2 | 512^2 | 80.8 | 20.0 | 13.3 | 20.0 | 6.7 | 6.7 | 20.0 | 20.0 | 6.7 | 6.7 | 13.3 (-6.0) |
| SD-1-4 | 512^2 | 71.2 | 20.0 | 13.3 | 26.7 | 20.0 | 13.3 | 40.0 | 33.3 | 26.7 | 13.3 | 23.0 (+3.7) |
| SD-2-1-base | 512^2 | 66.4 | 13.3 | 20.0 | 40.0 | 6.7 | 20.0 | 53.3 | 46.7 | 20.0 | 13.3 | 25.9 (+6.6) |
| SD-xl-base-1.0 | 512^2 | 90.4 | 20.0 | 20.0 | 26.7 | 26.7 | 6.7 | 46.7 | 33.3 | 6.7 | 26.7 | 23.7 (+4.4) |

Effect of Diffusion Model Structures.
At last, we explore the effect of incorporating various types of diffusion models for generative guidance.
Specifically, two types of diffusion models are employed as generative guidance in our DIVA, including DiT (Peebles & Xie, ) and stable diffusion (SD) series (Rombach et al., ) (i.e., 1-4, 2-1-base, xl-base-1.0).
It is clear in Table 8 that our method achieves the biggest performance gain (i.e., ↑6.6) on MMVP-VLM by integrating SD-2-1-base.
Furthermore, we observe that integrating DiT-XL/2 as generative guidance exacerbates the perceptual ability of original CLIP model in capturing visual details.
We attribute this to DiT’s relatively poor representation quality compared to SD models.
For the included SD series, the quantitative results in Table 8 also demonstrate that DIVA is not sensitive to version of SD models, which can consistently refine CLIP’s feature representations within our framework.

## 5 Conclusion

In this work, we focus on addressing the visual limitation of CLIP models that struggle with distinguishing fine-grained image details.
We present the first work to explore leveraging generative feedback from text-to-image diffusion models to directly optimize CLIP models’ representations.
Specifically, by feeding dense visual features from CLIP as condition to the diffusion models and introducing the reconstruction loss from diffusion process onto the CLIP model’s optimization, we establish a self-supervised framework DIVA.
Notably, this architecture is simple and clean, requiring no additional plugins while demonstrating significant potential.
Extensive evaluations demonstrate that our DIVA not only substantially enhances CLIP models’ performance on the MMVP-VLM benchmark that measures visual abilities of vision-language models, but also aids in improving the performance of MLLMs and vision networks respectively on multimodal and visual understanding tasks.
Furthermore, experiments on 29 benchmarks evaluating generalization capabilities confirm that our self-supervised DIVA maintains the CLIP models’ original excellent generalization capabilities.

## 6 Limitations and Future Topics

One potential limitation of this work is that the data scale for generative fine-tuning and the model capacity of our DIVA could be scaled up further to push better CLIP representations and performance.
Moreover, this work mainly focuses on designing a simple but effective framework to enhance CLIP models with generative diffusion process, which means although our DIVA demonstrates the newly exploited potential of using generative diffusion models for better CLIP models’ representation guidance, it can be integrated with finer-grained supervision scheme to further boost discriminative model capabilities.
Furthermore, exploring additional modalities beyond image-text data, e.g., video and audio, is also a promising direction for investigation.
Since this work is just a beginning in this direction, it opens up a future research perspective to develop a more general and powerful framework based on diffusion models that could enhance vision-language foundation models.

## Appendix

## Appendix A Datasets for Generalization Capability Evaluation

**Table 9: Datasets used to evaluate CLIP models’s generalization capability.**
| Dataset | Classes | Test size | Evaluation Metric |
| --- | --- | --- | --- |
| ImageNet-1K (Deng et al., ) | 1000 | 50,000 | accuracy |
| ImageNet-V2 (Recht et al., ) | 1000 | 10,000 | accuracy |
| ImageNet-Adversarial (Hendrycks et al., ) | 1000 | 7,500 | accuracy |
| ImageNet-R(endition) (Hendrycks et al., ) | 1000 | 30,000 | accuracy |
| ImageNet-Sketch (Wang et al., ) | 1000 | 50,899 | accuracy |
| ObjectNet (Barbu et al., ) | 1000 | 50,273 | accuracy |
| CIFAR-10 (Krizhevsky et al., ) | 10 | 10,000 | accuracy |
| CIFAR-100 (Krizhevsky et al., ) | 100 | 10,000 | accuracy |
| MNIST (LeCun et al., ) | 10 | 10,000 | accuracy |
| Caltech101 (Fei-Fei et al., ) | 101 | 9144 | accuracy |
| SUN397 (Xiao et al., ) | 397 | 108,754 | accuracy |
| FGVC Aircraft (Maji et al., ) | 100 | 3,333 | accuracy |
| Country-211 (Radford et al., ) | 211 | 21,100 | accuracy |
| Stanford Cars (Krause et al., ) | 196 | 8,041 | accuracy |
| Birdsnap (Berg et al., ) | 500 | 2,195 | accuracy |
| Describable Textures (Cimpoi et al., ) | 47 | 1,880 | accuracy |
| EuroSAT(Helber et al., ) | 10 | 27,000 | accuracy |
| Facial Emotion Recognition 2013 (Goodfellow et al., ) | 8 | 3,574 | accuracy |
| Oxford Flowers 102 (Nilsback & Zisserman, ) | 102 | 6,149 | accuracy |
| Food-101 (Bossard et al., ) | 102 | 25,250 | accuracy |
| GTSRB (Stallkamp et al., ) | 43 | 12,630 | accuracy |
| PatchCamelyon (Veeling et al., ) | 2 | 32,768 | accuracy |
| Oxford-IIIT Pets (Parkhi et al., ) | 37 | 3,669 | accuracy |
| Rendered SST2 (Radford et al., ) | 2 | 1,821 | accuracy |
| RESISC45 (Cheng et al., ) | 45 | 31,500 | accuracy |
| STL-10 (Coates et al., ) | 10 | 8000 | accuracy |
| Pascal VOC 2007 Classification (Everingham et al., ) | 20 | 4,952 | accuracy |
| Flickr30K (Young et al., ) | - | 1000 | recall |
| COCO (Lin et al., ) | - | 5000 | recall |

## Appendix B Pseudo code for DIVA Pipeline

Figure: Algorithm 1 DIVA

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See the weakness above
soundness: 3
strengths: 1. the paper is well written and easy to follow.
2. The improvement on the MMVP-VLM benchmark is significant.
3. The experiment is well designed and comprehensive.
summary: The paper aims to improve some CLIP's shortcoming cases, e.g., understanding the quantity,color,structure of images, since CLIP is trained by focusing on the high level semantic features. Motivated by this issue, the paper proposed to leverage a diffusion model to assist the improvement of CLIPs and inject the CLIP features as a condition into the diffusion model. The design brought a significant performance gain on the MMVP-VLM benchmark as well as the Llava 1.5 and segmentation tasks without impairing the other benchmarks.
weaknesses: 1. The hypothesis of this paper is not well explained, and the solution is a little bit ad-hoc. In detail, the failure of CLIP on several typical cases mentioned in the paper should be better studied and analyzed instead of just citing a few papers and mentioning very shortly. Are the failure cases caused by the architecture design, training strategy or just the lack of training data is unknown. Only given such analysis, we can start to think of a solution to improve. On the contrary, the paper just proposed to use diffusion model to help CLIP output more detail or low-level features directly, which is less convincing to me.

2. Several settings of this paper are also ad-hoc. For instance, in L225 - L241, the paper discussed the 'Visual Recap Density'. It seems for different CLIP variants, the density is way different, and I can hardly find any relationship across them. A more scientific way to decide the density should be considered. Similar thing for the times of the condition is used. The paper chooses 2 (N=2). I'm curious how this number is determined and how the performance will change for different numbers, e.g., N=10, 100.

3. From my understanding, the paper used CC3M dataset to tune a pretrained CLIP. I wonder whether the gains on the benchmarks are just brought by tuning the CLIP with CC3M or the diffusion model?

4. The rows 3-4 in Table 6 is confusing. When the textual condition is used, how the vision encoder can be trained?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the weakness section.
soundness: 2
strengths: - Learning a good discriminative representation from image generation loss is now new, but I think it still has a big potential and should be studied. In this regard, the proposed idea of fine-tuning a pre-trained discriminative model (CLIP) via a pre-trained generative model (Diffusion) is interesting and demonstrates its potential.

- The authors evaluate DIVA across various benchmarks, including multimodal understanding such as MMVP-VLM, the backbone of LLaVA, zero-shot classification, and segmentation.
summary: This manuscript introduces DIVA, a fine-tuning framework to enhance CLIP's fine-grained visual capabilities using a text-to-image diffusion model. DIVA tunes CLIP through diffusion loss with a frozen diffusion model conditioned by the CLIP outputs. To be specific, DIVA introduces a heuristic design for CLIP output-conditioned diffusion model, which leverages randomly dropped CLIP output sequences. Throughout experiments, it demonstrates improvements in extensive benchmarks including MMVP-VLM.
weaknesses: - Overall, this manuscript lacks detailed motivations and explanations. It is unclear why this approach is needed - e.g., Why does CLIP need diffusion feedback and how does it help? Why this particular approach is superior to others?

- The training process involves sampling multiple random states of the diffusion process for each image, which can be computationally expensive. There is insufficient discussion on whether this cost is justified and whether the gains are significant enough under these costs.

- The proposed diffusion's condition design (section 3) is quite heuristic and overall improvements are not consistent and even not significant in many cases. For example, none of the cases in Tables 1, 2, 4, 5, 7, and 8 show consistent improvements in all metrics. Moreover, many metrics indicate the improvements are marginal so I am uncertain that the proposed method effectively works and can generalize well across different types of visual challenges..

- There is a lack of analysis on potential trade-offs when using diffusion models as feedback. The paper does not clearly address whether the alignment between the CLIP's text encoder and visual encoder might degrade over time, potentially affecting performance, despite there being no consideration of the text encoder in the proposed training framework.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 1
questions: - The paper has highlighted that the CLIP is trained with limited length. Most diffusion models are also only trained with limited token lengths and have the same problem. Would it be possible that the golden fraction of local patch sampling rate (in Table 6) is just because of this?
soundness: 2
strengths: 1. The paper follows a clear intuition on how to improve the CLIP model with better visual understanding. The utilization of diffusion model to improve CLIP model is overall a nice idea.

2. The paper presented a method to use both the global tokens and local tokens to improve the visual encoder. The paper also found that the percentage of used local tokens is important and conducted experiment to validate that. 

2. The paper analyzes the method on multiple tasks and thus the improvement is better shown.

3. The paper shows ablation studies for key designs of the method (some are actually the analysis of the method).
summary: The paper utilizes the diffusion model to improve CLIP features' spatial and visual understanding. As CLIPs are only trained with global visual token, it lacks the ability to understand fine-grained visual concepts. At the same time, the diffusion model is trained to model the image distribution, thus it inherently has a better visual understanding. Given this idea, the paper tries to fine-tune CLIP model with a diffusion model guidance. 

In details, the proposed method sends the global token and random-sampled local tokens as a condition to a diffusion model. The gradient back from the diffusion loss is used as guidance to optimize the CLIP model. Based on this, the paper presents detailed ablation studies and shows improvement for visual understanding.
weaknesses: Weaknesses: 

- The paper's clarity of the method needs to be improved. Usually, the presentation (i.e., writing) of the paper would not be directly treated as weakness and some of them are put in suggestions below. However, the ambiguity in current method description does affect the reader's understanding of the paper method, and affect the assessment of the paper during review. Thus the reviewer put it as a weakness to highlight it and hope that it can be improved during rebuttal. I list some points here:
  
   1. For Sec 3.3, does it an expansion of context for the "visual features combined with the empty text’s embeddings"? If so, please help specify it in the text.
   2. From my current understanding, the visual features (patches) are feed as the text embeddings to the diffusion model pipeline. It would be better to have any analysis on the validity of doing this (at least have some discussions regarding this as it is not an natural operation). 
   3. What would be the positional embedding for the patches when feeding into the text encoder?
   4. In Sec 3.3 ("visual reacp density"), it would be better to illustrate the reason why the random selection schemas are different for different methods. 


- The paper lacks the details of the evaluation protocol. As the detailed inference method can largely affect the results, it's crucial to have a detailed description of how the method is evaluated on the MMVP dataset. 
   1. Given this, it would be hard to evaluate the validity of the ablation studies, which also affect the review of the paper. 

- The analysis needs to be improved in presentation:
   
   1. For some comparisons, the difference between baseline and proposed methods are close (e.g., 0.4 in absolute value). For these small differences, it's hard to understand its improvement. Some variance or significances analysis would be appreciated. Also, highlighting these numbers with gray colors would be helpful to readers. 
   2. In Table 3, some baseline and the proposed methods have the same score, but only the proposed methods are bold. It would be confusing to readers. 

Suggestions (will not be treated as paper weakness but would request it in the revision):
0. The algorithm description in the main text is not comprehensive enough for reproducing. It would be better to expand the descriptions. Even it sounds trivial, it would be better to extend these content for clarity. 
1. It would be better to refine the terminology in Algo 1 (Appendix B). E.g., 
   - The $\theta^*$ and $\theta$ are confusing. 
   - It's better to write CLIP feature as $f_\theta(x)$ instead of \theta(x)$

Comments:

I overall would like to trust the validness and soundness of the paper given the detailed experiments. The current details provided in the paper are not convincing enough to the readers.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - The proposed method significantly enhances the spatial awareness of CLIP. I wonder if this improvement extends to other multimodal tasks, such as open-vocabulary dense prediction tasks.
soundness: 3
strengths: - Utilizing diffusion models to enhance CLIP's dense representation presents a novel perspective that will likely interest the community.

- The proposed method is intuitive and straightforward to implement, making it compatible with various existing vision-language models

- The experiments are comprehensive, demonstrating significant improvements across various benchmarks.

- The paper is well-written and easy to follow.
summary: CLIP faces challenges related to fine-grained visual perception. To address this, the authors propose fine-tuning the CLIP image encoder by leveraging the spatial awareness of diffusion models. They use the visual tokens and the $[CLS]$ token from the CLIP image encoder as conditions for the diffusion models, backpropagating the gradients to update the CLIP encoder. The fine-tuned models are evaluated across several challenging benchmarks that require fine-grained visual perception, yielding promising results.
weaknesses: - Given that no explicit alignment constraint between the image and text encoders is applied during the fine-tuning process and only the image encoder is updated, I am concerned about the potential for misalignment between the image and text encoders. 

- The visual recap operation varies across different models, potentially necessitating careful tuning of hyperparameters. Is there a general strategy applicable to all models, or must it be tailored to each specific model?

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 4
questions: 1. It is crucial to also verify that the original cross-modal alignment capability of CLIP can also be preserved.

2. What criteria were used to decide the proportion of local tokens? Could you provide an analysis or ablation study to support this choice?

3. How does the varying proportion of local tokens across different datasets and between training and inference affect the method's generalizability?

4. Is there any other local token selection strategy that might yield better results?

5. Would incorporating image-associated text in the diffusion model conditions improve the performance observed in Table 6?
soundness: 3
strengths: --The paper presents an innovative approach by being the first to leverage generative feedback from text-to-image diffusion models to optimize the visual encoder of CLIP, aiming to addressing its limitation in fine-grained visual perception.

--The experiment part is extensive, validating their approach using the MMVP-VLM benchmark and demonstrating robust improvement. The integration with MLLM shows its impact, while experiments across 29 image classification and retrieval benchmarks confirm that the original capabilities of CLIP are preserved.

--The paper is clearly written, with well-explained methods and detailed descriptions of the proposed framework and experimental results.
summary: The paper introduces DIVA, a self-supervised framework designed to enhance CLIP's ability to perceive fine-grained visual details by leveraging generative feedback from text-to-image diffusion models. The framework incorporates a novel visual dense recap scheme, conditioning the diffusion models with CLIP's dense visual features and using image reconstruction loss for optimization. The results show that DIVA significantly improves CLIP's visual perception and its performance on multimodal and visual understanding tasks, as demonstrated on the MMVP-VLM benchmark. Additionally, DIVA maintains CLIP’s strong zero-shot capabilities across 29 image classification and retrieval benchmarks, proving its effectiveness in optimizing CLIP's discriminative representations.
weaknesses: --Disruption of Visual-Language Alignment: Optimizing CLIP’s visual encoder may disrupt the inherent visual-language alignment of CLIP. Although the authors validate that CLIP’s original capabilities are retained through experiments on 29 datasets, these tests are somewhat limited, as they only cover image classification and unimodal retrieval (text or image retrieval). Because the text encoder is not fine-tuned, the retrieval results for text remain unchanged, making text-retrieval experiments redundant to some degree. Moreover, the paper lacks experiments on tasks that rely heavily on cross-modal alignment, an area where CLIP excels, such as cross-modal retrieval. If such disruption occurs, it would be valuable to explore methods for preserving cross-modal alignment, such as applying a reconstruction loss to the visual class token to maintain the original feature.

--Part Token Selection Strategy: The method involves inputting tokens from CLIP’s visual encoder, including the global (class) token and a subset of local tokens, as conditions for the diffusion model generation. Although the authors state that using only the global token or a combination of the global token with all local tokens yields suboptimal results than global + part local tokens, they do not clarify how the proportion of local tokens is determined. Additionally, this proportion appears to vary across different datasets and between training and inference, raising concerns about the method’s generalizability. Furthermore, the strategy for selecting part tokens could impact the results, and using a simple random selection might not be optimal.

--Empty Text Conditions: In Table 6, the conditions for the diffusion model include an empty text condition. It would be interesting to explore whether incorporating text descriptions corresponding to the images could further enhance the results.

--Minor typo: There is an extra word "for" at line 350.

Meta-review: This paper introduces DIVA, a post-training technique for CLIP-like models to overcome some of their shortcomings using a self-supervised diffusion approach that is trained from images only, using the CLIP feature to condition the diffusion model. This approach improves performance on fine-grained visual recognition, multimodal understanding and segmentation tasks. 
The reviews mention several strengths, including that the paper is well written, significant improvements over prior work, well executed experiments, interesting novel direction, multiple tasks are considered, method compatible with various existing VLMs. 
Weaknesses: effect of finetuning on CC3M not ablated, lack of detail on the cost of the sampling process, lack of clarity in places (method description and evaluation protocol).

Decision: Accept (Poster)