un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP
Authors: Yinqi Li
Jiahe Zhao
Hong Chang
RuiBing Hou
Shiguang Shan
Xilin Chen
Venue: NeurIPS 2025 poster ()
Keywords: CLIP model
unCLIP generative model
model inversion
visual detail capture
Primary area: deep_learning
TL;DR: This work improves CLIP’s visual detail capturing ability by inverting the unCLIP generative model, which we find suitable for achieving this goal.

Abstract: Contrastive Language-Image Pre-training (CLIP) has become a foundation model and has been applied to various vision and multimodal tasks. However, recent works indicate that CLIP falls short in distinguishing detailed differences in images and shows suboptimal performance on dense-prediction and vision-centric multimodal tasks. Therefore, this work focuses on improving existing CLIP models, aiming to capture as many visual details in images as possible. We find that a specific type of generative models, unCLIP, provides a suitable framework for achieving our goal. Specifically, unCLIP trains an image generator conditioned on the CLIP image embedding. In other words, it inverts the CLIP image encoder. Compared to discriminative models like CLIP, generative models are better at capturing image details because they are trained to learn the data distribution of images. Additionally, the conditional input space of unCLIP aligns with CLIP's original image-text embedding space. Therefore, we propose to invert unCLIP (dubbed un$^2$CLIP) to improve the CLIP model. In this way, the improved image encoder can gain unCLIP's visual detail capturing ability while preserving its alignment with the original text encoder simultaneously. We evaluate our improved CLIP across various tasks to which CLIP has been applied, including the challenging MMVP-VLM benchmark, the dense-prediction open-vocabulary segmentation task, and multimodal large language model tasks. Experiments show that un$^2$CLIP significantly improves the original CLIP and previous CLIP improvement methods. Code and models are available at https://github.com/LiYinqi/un2CLIP.

un2CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP
Yinqi Lil2, Jiahe Zhao! Hong Changl Ruibing Hou', Shiguang Shan Xilin Chen State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, China Universityy of Chinese Academy of Sciences (CAS), China yinqi. wwi.ipplip...... het.wipp.......... ac...a.cc, (zhaojiahe22s, changhong, houruibing, sgahan, xlchen)@ict..aco
Abstract
Contrastive Language-Image Pre-training (CLIP) has become foundation model and has been applied to various vision and multimodal tasks. However, recent works indicate that CLIP falls short in distinguishing detailed differences in images and shows suboptimal performance on cense-prediction and vision-centric multimodal tasks. Therefore, this work focuses on improving existing CLIP models, aiming to capture many visual details in images as possible. We find that a specific type of generative models, unCLIP, provides a suitable framework for achieving our goal. Specifically, unCLIP trains an image generator conditioned on the CLIIP image embedding. other words, inverts the CLIIP image encoder. Compared to discriminative models like CLIP, generative models are better at capturing image details because they are trained to learn the data distribution of images. Additionally, the conditional input space of unCLIP aligns with CLIP's original image-text embedding space. Therefore, we propose to invert unCLIP (dubbed un'CLIP) to improve the CLIP model. In this way, the improved image encoder can gain unCLIP's visual detail capturing ability while preserving its alignment with the original text encoder simultaneously. We evaluate improved CLIP across various tasks to which CLIP has been applied, including the challenging MMVP-VLMM L.M benchmark, the dense-predietion open- vecenvocabulrry segmentation task, and multimodal large language model tasks. Experiments show that CILII significantly improves the original CLIP and previous CLIP improvement methods. Code and models are available at https:// github. com/ comm /unICIIP
Introduction
Contrastive Language-Image Pre-training (CLIP) models trained on web-scale datasets have shown great success in learning transferable representations for image classification [1]. Since their release, they have been widely adopted for many vision and multimodal tasks, such as open-vocabulary segmentation [2,3,4] 4] and multimodal large language model (MLLM) tasks [5, 6, 7]. These extensions can be attributed to the learned web-scale knowledge and the vision-language alignment property of CLIP models. However, CLIP cannot always perform well on these extended, finer-grained tasks that require more detailed image understanding, as observed in dense vision tasks [2, 8] and MILM literature [9]. This phenomenon may be due to CLIP's training data and objective, where the global-level image-text contrastive learning makes the image encoder poor at capturing visual details.
To address this problem, some works modify the network architecture inference time to make the encoder gather less global information [2, 3, 4], some other works deploy additional visual self- supervised encoders in dense-vision tasks [10, 11] and MLLM frameworks [9, 12, 13, 14]. Although effective, these methods do not fundamentally address the problem that CLIP falls short of capturing visual details. To build finer-graine CLIP models, existing works mainly focus on training new
39th Conference on Neural Information Processing Systems (NeurTPS 2025).

---

CLIP Image Encoder
unCLIP Image Generator
CLIP Image Encoder
unCLIP Image Generator
CLIP Image Encoder
unCLIP Image Generator
Input Image
CLIP Image Embedding
Input Image
CUP Image Embedding
Generated Image
CLIP Image
Input Image
Image
Image
Embedding
(a) unCLIP Training
(b) unCLIP Inference (Sampling)
(c) un Training (CLIP Finetuning)
Figure I: Comparison of unCLIP [20] and CLPP pipelines. (a) (b) unCLIP provides an encoding-
decoding tool for observing which features of the image are disregarded by CLIP (more examples shown in Figure 3 of [20]). (c) Our un CLIP further leverages this framework to improve CLIP, aiming to recapture the disregarded features. The unCLIP model contains a text to image-embedding "prior" module for supporting the text-to-image pipeline, which is not used in our work.
CLIP variants with more detailed vision-text supervisions such as region-text alignment [8, 15, 16, 17, 18, 19]. However, collecting high-quality region- paired datasets in real world is more difficult than acquiring original CLIP's image-text pairs, because the former is limited in amount at web and typically requires human annotation. Besides, re-training CLIP models is costly that would like to avoid. Therefore, in this work, we focus on improving existing CLIP models from the vision-centric perspective using image data only.
Specifically, we find that a specific type of image generation models, unCLIP [20], provides a suitable framework for achieving our goal. To start with, firstly, unCLIP offers way to qualitatively visualize which features are disregarded by the pretrained CLIP image encoder. As illustrated in Figure (a), unCLIP trains image generator that inverts the CLIP image encoder, taking the CLIP image embedding as input to generate images. After training, the encoding-decoding pipeline provides us a tool observe which features of the image are not captured by the encoder, as shown in Figure (bb).
Furthermore, this encoding-decoding pipeline not only serves as a tool to observe CLIP's failures, but also offers a proper framework to improve CLIP in vision-centric manner. To be specific, building finer-gained CLIP requires enhancing the visual detail capturing ability of its image encoder while preserving the image-text alignment property at the same time. We note that the unCLIP framework suitable for achieving this goal because (1) unCLIP a generative model learning the underlying distribution of image data, which enhances its capability in capturing the complexities and variations in images; (2) unCLIP takes the image embeddings from the output layer of the pretrained CLIP as its conditional input, which are aligned with their corresponding CLIP text embeddings.
Based on these two properties, we propose to finetune the CLIP encoder an unCLIP generator inversion way, as illustrated in Figure (()), thereby transferring the generator's rich visual knowledge into the encoder and leveraging the remarkable language-alignment property the unCLIP generato's input space. We name our method unLCLIP since inverts the unCLIP image generator. Compared to prior work [2]], which improves the CLIP image encoder using a pretrained text-to-image generative model that has a mismatched input space with CLIP image embeddings, our un CLIIP is designed within the wtr---mbdddnngalliiieeeee framework, thereby facilitating more effective finetuning.
Our contributions can be summarized as follows:
We find that unCLIP provides suitable framework for improving the CLIP image encoder's ability to capture visual details while maintaining its alignment with the language encoding space.
Based the above finding, we propose pure image-based method unLCLIP improve CLIP, which finetunes the CLIP image encoder by inverting unCLIP.
Our experiments show that CnLC significantly improves CLIP and outperforms prior methods on various tasks and benchmarks, including the MMVP-VLM benchmark consisting of CLIP-blind pairs [9], the dense vision task of open-vocabulary segmentation, and vision-centric MLLM tasks.
2 Related Works
CLIP and Downstream Applications. CLIP trains a pair image and text encoders using contrastive loss on web-collected image-text pairs, which learns transferable representations and shows remarkable zero-shot classification performance [1]. Due to large-scale pretraining and the vision- language alignment property, CLIP has become foundation model, widely applied across a variety of

---

vision and multimodal tasks, including open-vocabulary semantic segmentation [2. 3, 4, 22, 23, 24], objet detection [8. 15, 17, 25, 26]. text-to-image generation [20, 27, 28, 29], and large-language- model-based visual question answering [30, 5, 6, 7]. However, recent studies have shown out that CLIP underperforms on tasks that require fine- ane-grained visual understanding. Tong et al. [9] find that CLIP struggles to distinguish certain visual patterns, which also impairs the performance of MLLMs built on top of CLIP. Additionally, [2, 8J claim that CLIP, being trained with the image-level objective, tends to capture global semantics while neglecting local details, limiting its effectiveness as a backbone for dense prediction tasks.
Vision-centric CLIP Improvements. To improve CLIP's ability to perform fine- tine-grained visual tasks, existing efforts can be broadly categorized into two groups. The first group targets specific downstream applications. For instance, some works [2, 3, modify CLIP's image encoder architecture at inference time to make gather less global-Ievel information, thereby improving performance on the pixel-level dense prediction task. Targeted at vision-centrie MILM tasks, works [9, 12, 13, 14] incorporate additional visual self-supervised encoders, and [31] finetunes the CLIP encoder with visual supervision within the MLLM framework. to address CLIP's shortcoming in learning detailed image representations. Although effective, these approaches are downstream--takk-pp -aee-seec-is and do not fundamentally resolve CLIP's intrinsic limitation in modeling detailed visual representations.
On the other hand, another line of work seeks to improve CLIP the upstream stage by modify- ing its training data and objectives to produce finer-graine representations. Specifically, several approaches [8. 15, 16, 17. 18, 19] perform CLIP-like pretraining but align image regions with textual descriptions, enabling the model to learn more detailed, region-level visual representations. However, collecting high-quality, large-scale region-text pairs more challenging than acquiring image-text pairs, as the former typically requires human annotation, while the latter can be easily sourced via web scraping. Our work also aims to fundamentally address CLIP's limitation in capturing visual details at the upstream level. In contrast to region-level approaches, and leverage generative models to enhance visual detail capturing. The most relevant prior work is DIVA [21] J. which uses pretrained text-to-image generative model whose input mismatches CLIP's output to improve CLIP. In comparison, our framework uses a generative model that operates in the same representation space as CLIP, enabling more effective and seamless enbancement process.
rely solely image data
Generative Models for Representation Learning. Generative models such as diffusion mod- els [32, 33] trained on large-scale datasets have shown remarkable progress in generating photore- alistic images [34, 27, 28, 29]. The remarkable generation ability implies that they have accurately modeled the underlying data distribution and learned effective representations of images. Based on this motivation, [35, 36, 37, 38, 39] leverage the pretrained generative models as the backbone for visual perception, and show impressive performance on solving dense vision tasks such as semantic segmentation and depth estimation. This work also leverages the rich representations in generative models for visual perception, but transfer them into an existing discriminative model CLIP.
Method
We first introduce some preliminaries of generative models and unCLIP in Section 3.1, and then describe our CCLI method in Section 3.2.
3.1 Preliminaries
Generative Models. Generative models, or image generation models in specific, are trained to learn the data distribution of images usually by maximizing the log-likelihood IEg log (x),,, or Eg log (x|c) for conditional generative models, where G denotes the generator, x stands for images, and e is the conditional input such class and text.
Diffusion models [32, 33] are one of the popular types of generative models recently. They are trained to reverse forward diffusion process, where noises e ~ N' (0. I) are gradually added to the clean image x over a number of timesteps 1 = I..,.T. For the reverse process, a denoising network Eg f) is trained to estimate the added noise at each timestep, where x, is the noisy image at t. Finally, the log-like likelhhodd maximization objective is implemented by optimizing a variational bound of it:
max Ex (xx)) & min Ex,es [lle Cx, 1)13]
(1)
3

---

unCLIP: beeembedding-cooddiione Generative Model. Given pretrained CLIP [1] image encoder E. unCLIP [20] aims train decoder that can invert the encoder E. The decoder designed to be non-deterministic, meaning it can produce multiple images for given CLIP image embedding. This allows for a more comprehensive examination of CLIP's disregarded features. Specifically, the decoder is implemented as probabilistic image generator G conditioned on the image embeddings from the output layer of CLIP image encoder, as illustrated in Figure I(a). In [20], the authors employ diffusion model as the generator, using the following training objective:
max Ex
R min min Ex.Exx.ellll Eg E(x)]1
3.2 Cn2CLIP: Improving CLIP via Inverting unCLIP
CLIP models have been widely adopted in many vision and multimodal understanding tasks [2, 5, 6]. However, recent studies have revealed that CLIP falls short in distinguishing certain visual-detail related patterns [9] and exhibits suboptimal performance solving dense vision tasks [2, 8]. This section aims to develop an approach to alleviate this problem.
Since original CLIP models are trained on global-Ievel image-text pairs, developing new CLIP variant that focuses on visual details from scratch poses challenges both in designing suitable training objectives and in collecting training data pairs, where high-quality. open-world region-text or pixel- text pairs would be difficult to collect. Moreover, re- re-training CLIP is costly. Therefore, we would like improve existing pretrained CLIP models from the vision-centric perspective. To be specific, given pretrained CLIP model, we aim to improve its image encoder to capture as many visual details in images as possible while maintaining its language- language-alignmen property simultaneously.
Goal of Our Problem. Formally, the goal is to maximize the mutual information hetween an input image and its embedding E(x) produced by the CLIP image encoder, subject a language-alignmen constraint:
max (x; E(x), s.t. d(E()).) E
where y denotes the semantic aligned text embedding of x, produced by the CLIP text encoder, and stands for distance.
Backend of Our Framework. Note that the mutual information in Eq. (3) can be expressed as
[(x; E(x)) = Hx)) - Ex)) in terms of entropy H. Therefore, max, (x; E(x)) in Eq. (3)
equals
min (x E()) = min Ex |- log p(x) E(X)] = max E [log p(x)E(())]
(4)
E
where the first equation is according to the definition of the conditional entropy.
By comparing Eq. (4) with Eq. (2), we observe that the pretrained unCLIP model, which takes the CLIP image embedding E(x) as input to generate images x, provides a suitable probability model
(x] E()) for estimating Eq. (4). This motivates to adopt the pretrained unCLIP model G(E()) as the backend of our framework, enabling us to improve the front-end CLIP image encoder E(+), as illustrated in Figure (cc).
Reducing Language-shift During Finetuning. The other factor remaining in our objective (Eq. (3)) is to make embeddings produced by the finetuned image encoder have correct semantics that can be interpreted by the original CLIP text encoder. Since our framework relies on image data only, maintaining this language- alignmmet property poses nontrivial challenge. For addressing this challenge, we note that the input space of the unCLIP generator G lies within the CLIP image-text embedding space. Therefore, for reducing the potential language- shift during the image- encoder finetuning, we freeze the parameters of G, thereby encouraging the optimized embedding E(x), when fed G, staying close G's original input domain.
un CILI.. Taking these together, our CLIP finetuning objective is
max Ex (x) E(x) v min Exen [lle 1x, E(X)) E
(5)
The training objective can be interpreted as inverting the unCLIP image generator G. which possesses strong visual detail representations and has an input space aligned with the embedding space of CLIP. By optimizing the CL.IP image encoder E in this unCL.IP-inversoon (dubbed un framework, we effectively enhance E's is ability capture fine- fine-grained visual information. In practice, when finetuning

---

E using Eq. (5), we inherit the training configuration of the unCLIP diffusion model, such as the timestep and noise schedule. This is like a replay of the training procedure of unCLIP G, but here we freeze the trained G and update E.
Comparison to the Prior Work DIVA [21]. DIVA is related approach that also leverages pretrained generative models to enhance CLIP. However, its architecture and optimization differ fundamentally from ours. DIVA deploys a text-to-image generative model [28] behind the CLIP image encoder. Notably, the input space of such generators differs from the output space of the CLIP image encoder, both in embedding dimensionality and, more importantly, in semantic representation. To bridge this gap. DIVA inserts a trainable projection layer P between the encoder and the generator. Its resulting training objective can be written as maxg, Ex [log Pa ( ((E()))) which deviates the derived goal in Eq. (4), where P would take away part of the knowledge learned from G. Therefore, using generator that misaligned with the CLIP embedding space may be suboptimal for improving the CLIP image encoder.
4 Experiments
4.1 Experimental Setup
Pretrained CLIP and unCLIP Models. Since Ramesh et al. [20] do not provide official unCLIP code and models, we use open-sourced implementation, Stable unCLIP! in our experiments. Stable unCL.P provides two pretrained models conditioned on different CLIP image embeddings OpenA CLIP V-T-L.-14 @ 224 [1] and OpenCLIP ViT-H-14 224 [40], respectively. For evaluating the generality of our method within other CLIP backbones, we train another unCLIP model, conditioned on SigLIP VIT-So-14@t 384 [41], based on the above open-sourced implementation. Besides, we find in preliminary toy experiment (in Section I. that the encoders of OpenAI CLIP WTT-L-14@224 have a similar embedding space to OpenAI CLIP ViT-L-144 336 [1], therefore, for saving the computational cost of training new unCLIP models, we use the same existing Stable unCLIP for finetuning both of them. More details of these pretrained Stable unCLIP models are provided in Section B.2.
un CLIP Training Details. CnICIIP is trained 8 Nvidia-A 1100-400B GPUs with global batch size of 32, learning rate of 3e-7, using Adam W optimizer. For a fair comparison, we train IIL on the CC3M dataset [42] over 1 epoch following [21]. taking around 15~32 hours for different model types. The remaining syper-parameters are kept the same as the training configuration of Stable unCLIP in the codebase.
Compared Methods. In addition to the original CLIP models, we mainly compare to the pioneering work DIVA [21] in this field that uses generative models improve pretrained CLIP models. Different from ours, DIVA uses a pretrained text-to-image generative model as the backend, whose input space misaligned to CLIP image encoders' output space. We also note that, most recently, there exists a contemporaneous work to ours named GenHancer [43]. GenHancer does not leverage existing well-trained generative models but trains imperfect generative models themselves for improving CLIP, which is different from DIVA and our unLCLIP. More detailed discussions of the relationships and differences DIVA and GenHancer are given in separate section at Section A. In this section, for a more comprehensive comparison, we also present the results of GenHancer for reference but are dimmed in gray due to contemporaneity.
Evaluated Tasks and Benchmarks. We evaluate on several tasks to which CLIP has been applied and require more detailed image understanding ability, including the image-level MMVP-VLM benchmark [9] evaluating whether some detailed visual patterns are successfully captured (Section 4.2), the pixel-level dense vision-language inference task [2, 3, 4] (Section 4.3), and multimodal large language model related tasks [6, 7] (Section 4.4). Detailed introductions to these tasks and benchmarks are provided in corresponding subsections. For completeness, we also present the results on the classical evaluation tasks of CLIP, i.c., zero-shot classification and retrieval, in Section 4.5.
Url: https://github. . comp/Stailiit-AAsssaeelitttiiiss text-to- to-image model Stable Diffusion [28]. Stable unCLIP has an additional conditional text encoder. We feed it the empty string during un 'CLI training.
Due to built upon the
5

---

4.2 CLIP-Blind Pair (MMVP-VLM Benchmark) Evaluation
We first evaluate our finetuned CLIP models on the MMVP-VLM benchmark [9]. The benchmark covers 9 visual patterns, each comprising 15 image pairs (30 images) accompanied by textual descrip- tions. The image pairs are collected in an adversarial manner to the original CLIP model, which are proximate in CLIP feature space but distant in the feature space of visual self-supervised model (DINOv2 [44]. Only if both of the images are assigned to the accurate captions can a pair be deemed correct
The results are presented in Table 1. Our method achieves the best average performance across different CLIP models. Notably, un'CLIP significantly outperforms the original CLIP models and the previous DIVA method. This indicates that un CLIP is general and effective method in improving CLIP to distinguish more detailed visual differences in images. Our performance also matches and slightly outperforms the contemporaneous work GenHancer.
Table 1: MMVP-VLM benchmark evaluation. The benchmark contains 9 visual patterns that original CLIP models often misinterpret: Orientation and Direction, Q: Presence of Specific Features, Ø State and Condition, t:: Quantity and Count, f: Positional and Relational Context, Color and Appearance, * Structural and Physical Characteristics, A: Texts, O Viewpoint and Perspective. t denotes our reproduced results using official codes correspondingly.
CLIP Model
Resol.
#Params
Method
Q
ti
$
A
Avg
Original
13.3 13.3
20.0
20.0
13.3
53.3
20.0
6.7
13.3
19.3
DIVA
13.3
20.0
40.0
6.7
20.0
53.3
46.7
20.0
13.3
25.9
OpenAl VIT-L-14
224
427.6M
GenHancer
13.3
33.3
33.3
20.0
6.7
73.3
46.7
20.0
40.0
31.9
CLIP
0.0 33.3
46.7
26.7
13.3
80.0
40.0
20.0
33.3
32.6
Original
0.0 20.0
40.0
20.0
6.7
20.0
33.3
6.7
33.3
20.0
DIVA
26.7 20.0
33.3
13.3
13.3
46.7
26.7
6.7
40.0
25.2
OpenAl ViT-L-14
3362
427.9M
GenHancer
6.7
20.0
33.3
20.0
6.7
73.3
53.3
26.7
26.7
29.6
CLPP
6.7 33.3
46.7
13.3
13.3
80.0
40.0
20.0
20.0
30.4
Original
6.7
13.3
53.3
26.7
6.7
73.3
40.0
13.3
26.7
28.9
OpenCLIP ViT-H-14
224
986.1M
DIVA
13.3
13.3
53.3
26.7
6.7
73.3
46.7
13.3
26.7
30.4
GenHancer!
13.3
6.7
46.7
20.0
33.3
80.0
26.7
40.0
33.3
33.3
unLI
26.7
13.3
53.3
20.0
33.3
86.7
46.7
13.3
33.3
36.3
Original
20.0 26.7
60.0
33.3
13.3
66.7
33.3
26.7
53.3
37.0
DIVA
26.7
33.3
53.3
26.7
13.3
80.0
40.0
26.7
46.7
38.5
SigLIP VIT-SO-14
3842 878.0M
GenHancer
26.7
20.0
66.7
33.3
13.3
86.7
40.0
26.7
46.7
40,0
CI.P
20.0
20.0
60.0
46.7
26.7
73.3
40.0
26.7
60.0
41.5
4.3
Dense Vision-Language Inference Evaluation
Next, we evaluate on the dense-prediction semantic segmentation task, which is pixel-level task. thus evaluating more the capttr-n ability of CLIP models. The segmentation task also acts as helpful tool for qualitatively visualizing the behavior of the improved CLIP models.
Evaluation Setup. We follow training-free open- open-vocabulary semantic segmentation works [2, 3, 4] to evaluate our method. By keeping the CLIP model frozen, this setting provides good way for diagnosing the model's 's pixel-level understanding capabilities. The work [2] first proposes to apply the pretrained CLIP model [1] to zero-shot semantic segmentation, by comparing the local patches of image embeddings to the candidate text embeddings. The following works, including MaskCLIP [2]. SCLIP [3], and ClearCLIP [4], modify the inference-time network architecture to improve the perfor- mance. We employ these different methods, which have different initial performances, to evaluate the generality of our improved CLIP by substituting the CLIP model with our finetuned one.
Datasets and Metric. Following [2, 3, 4], we employ the mean Intersection over Union (mloU) metric and evaluate on eight datasets widely used for open-vocabulary semantic segmentation. These datasets can categorized into two groups: (1) Without background category: PASCAL VOC20
(VOC20) [45], PASCAL Context59 (Ctx59) [46], COCO-Stur (Stuff) [47], Cityscapes (City) [48], and ADE20K (ADE) [49]; (2) With a background category: PASCAL VOC (VOC21) [45], PASCAL Context (Ctx60) [46], and COCO Object (Object) [47].

---

Table Open-vocabulary semantic segmentation quantitative comparison. Results of DIVA and GenHancer are obtained using official checkpoints. The CLIP backbone is OpenAI ViT-L-14@3300
Segmentation Method
CLIP- Improve
Without background class
With background class
Average
Method
VOC20
Ctx59
Stuff
City
ADE
VOC21
Ctx60
Object
Original
11.7
3.4
1.7
2.5
0.9
7.7
2.9
3.3
4.3
DIVA
12.0
3,4
1.7
2.5
1.0
2.9
3.3
4.3
CLIP
GenHancer
8.4
2.9
2.7
0.7
3.1
un
17.3
5.1
2.6
3.8
1.3
9.3
4.3
4.3
6.0
Original
24.7
10.1
10.3
6.1
9.2
12.7
DIVA
25.7
10.4
7.6
10.4
6.3
22.4
9.5
12,6
13.1
MaskCLIP
GenHancer
13.5
6.4
3.4
9.2
3.7
12.3
5.9
4.9
7.4
CLIP
30.0
12.9
8.9
13.1
7.5
25.2
11.6
13.5
15.3
Original
37.3
12.7
8.5
10.2
4.6
28.7
11.9
14.9
16.1
DIVA
37.7
12.8
8.5
10.3
4.6
28.9
11.9
15.0
16.2
SCLIP
GenHancer
21.0
7.7
3.6
6.8
2.2
15.1
7.0
5.3
8.6
un CLIP
53.8
19.5
12.0
16.1
6.9
38.6
17.9
19.3
23.0
Original
72.4
26.0
18.1
22.8
14.2
42.6
23.2
27.1
30.8
DIVA
72.3
25.9
18.1
22.7
14.0
42.6
23.2
27.1
30.7
ClearCLIP
GenHancer
52.1
22.9
11.8
17.1
10.3
20.0
21.1
Cn'C
76.5
30.5
20.6
26.4
16.0
47.6
27.3
29.6
34.3
Quantitative Results. Table summarizes the performance of different CLIP improvement methods using different apen-vocabulary segmentation methods. We observe that our method CCIP achieves the best results across different datasets and segmentation methods, significantly improving performance of using the original CLIP model. Moreover, it worth noting that switching to our finetuned model can further achieve an average 3.5 mloU improvement for the ttate-of-the-arr ClearCLIP method. also note that the previous method DIVA achieves smaller improvements, and the contemporaneous work GenHancer achieves performance drops on this task. The following visualization results may provide some reasons.
Qualitative Results. In Figure 2, we present qualitative comparison of the original CLIP model and its im- provements, using the ClearCLIP seg- mentation method. can be seen that: (1) Although the overall semantics of the image is correctly predicted using the original model, there are many local noises (false predictions) in the segmentation results. This is because the original CLIP model is trained towards a global image un- derstanding objective, neglecting im- age details. (2) DIVA's segmentation maps very close the original CLIP model, indicating that DIVA has relatively conservative finetun- ing stride, which cannot significantly improve the pixel-level task's perfor- mance. (3) GenHancer helps clim- inate some noise compared the original, but it makes some original- correctly predicted pixels wrong (e.g.. the first and third columns), resulting the overall weak quantitative re-
Contevt59
COCO-Stuff
DE20K
VOC21
Contect50 COCO Object
Tawege
Original
DIVA
GeiForo
Groadd Truth LECLLI
Figure 2: Open-vocabulary semantic segmentation qualita- tive comparison.
sults. (4) After our -oriented improvement, part of the false predictions are eliminated, and the results become smoother, indicating that CILIIIP an effective upstream finetuning approach for making CLIP models better at performing dense prediction tasks.

---

4.4 Multimodal Large Language Model Evaluation
Lastly, we evaluate whether our finetuned CLIP model can enhance the performance of MLLMs where CLIP serves as the vision encoder, with a particular focus on vision-centrie benchmarks. To ensure fair comparisons, we adopt the evaluation setup in [21, 43] which test the improved CLIP using LLaVA-11 [7] without modifying LLaVA's default training configuration. We report results on the same set of benchmarks [9, 12, 50, 51, 52, 53] used in GenHancer [43] to simplify efforts in baseline reproduction. Due the memory constrain of our computing resources, we conduct experiments based on the 7B-version Vicuna [54]. The results are presented in Table 3. We observe that the improved visual detail capturing ability of our finetuned CLIP also benefits MLLMs, leading to improved performance, particularly on vision-centric benchmarks.
Table 3: MILM benchmark evaluation. Best and second best results are highlighted in bold and underline. Results on NaturalBench follow the official evaluation protocol [50], which differs from that in GenHancer [43], resulting in some missing entries. Baseline numbers are taken from [43].
Vision-centric Renchmarks
General Benchmarks
LLM
CLIP
MMVP
NaturalBench [50]
CV-Bench 2D [12]
CV-Bench
POPE [51]
SciQA- Hallusion IMG[S2] Avg. [53]
[9]
Acc Q-Ace L-Ace G-Ace ADE20K COCO
3D [12]
rand pop adv
Original
24.7
67.3
37.7
43.8
12.7
49.6
60.9
58.7
87.3 86.1 84,2
66.8
27.6
Vicuna-7B
DIVA
31.3
51.3
63.4
60.2
87.9 87.0 84.6
66.3
28.6
GenHancer
30.7
52.9
63.6
63.2
88.1
86.7 84.6
66.5
28.4
CLPP
31.3
68.7
40.0
45.9
15.1
53,9
65.1
61.2
88.0
87.4
85.4
68.4
28.4
4.5 Zero-shot Classification and Retrieval Evaluation
For completeness, we evaluate the improved CLIP models on the two classical tasks: zero-shot image classification and zero-shot text-image retrieval. It is worth noting, however, that these tasks and their standard benchmarks (ImageNet-1KK K [55], CIFAR- 10 [56], CIFAR- 100 [56], Caltech- -01 [57], SUN397 [58], FGVC Aircraft [59]. Stanford Cars [60], Flickr30K [61], and COCO [62]), particularly those for image classification, are not designed to assess fine ine-grained visual understanding. Rather, they tend to favor models that capture high-level, category-discrminnatv features while ignoring subtle
censificationniireeeaaa details. This evaluation setup therefore contrasts with the main objective of our work, which is to enhance CLIP's ability to capture visual details as much as possible.
Table 4: Zero-shot classification and retrieval evaluation. Results of DIVA and GenHancer are obtained using official checkpoints. The CL.IP model is OpenAl VIT-L-114 @224.
Image-tn-Text Retrieval
Text-to-lmage Retrieval@5
Zero-shot Image Classification
Method
IN-IK
C-10
C-100
Cal-101
SUN397
Aircralt Cars
Plickr30K
COCO Flickr30K COCO
Original
75.5
95.6
75.9
86.7
67.6
31.7
77.9
97.3
79.4
87.3
61.0
DIVA
75.5
95.5
76.3
87.1
67.5
31.6
78.0
97.3
79.7
86.9
61.0
GenHancer
40.2
77.5
44.2
79.3
42.4
7.2
21.0
87.2
61.7
81.6
51.0
un U
62.4
89.0
65.6
86.8
59.2
22.0
63.3
96.4
77.6
90.1
65.5
As shown in Table 4. our finetuned model, which exhibits significantly enhanced performance on dense and vision-centric tasks in Section 4.2-4.4, exhibits noticeable performance drop on zero-shot classification and comparable results on retrieval. Importantly, this drop in classification accuracy does not necessarily reflect degraded image-texl alignment: Even on relatively simple datasets such as CIFAR-10, which contains only 10 coarse categories, the performance drop remains evident, In contrast, on segmentation benchmarks involving much larger number of classes (c.g., 150-class ADE20K, 60-class Context), which also rely on the text encoder for label prediction, our fine-tuned model consistently outperforms the original CLIP (Table 2).
This discrepancy indicates that the observed drop arises from task differences rather than image-text misalignment. Classification tasks generally favor representations that emphasize dominant foreground semantics. However, somewhat "unfortunatell" from the standpoint of classification, our finetuned model corrects many cases where the original CLIP mistakenly attends to background regions as salient foreground objects (see Figure 2). These corrections enhance visual detail understanding but lead to lower accuracy on conventional classification benchmarks.
8

---

34
Table 5: Ablation studies of CLIP.
Diffusion Loss (1)
0.3393
31
Diffusion
MMVP-
0.3388
28
Method
Loss (I)
VLM (1)
25
0.3383
Original
0.3396
19.3
22
0.3378
un²CI.IP
0.3378
32.6
19
Proj. layer (random init.)
0.3441
16.3
0.0
0.5
1.0
2.0
Proj. layer (identity init.)
0.3378
30.4
Epoch
Proj. layer (two stage)
0.3403
30.4
Figure 3: Diffusion loss and MMVP-VIM
Updating generator G
0.3376
27.4
performance with respect to training epochs.
4.6 Ablation Studies
We conduct ablation studies of our method on the MMVP-VLM benchmark, using OpenAl ViT-L- 44@224 as the baseline model. Before analyzing our method, we first introduce diagnostic tool that measures the reconstruction ability of the CL.PPencoding generator-decoding pipeline.
Diffusion Loss Diagnostic Tool. We adopt the diffusion loss (Eq. (2), (5)) a diagnostic metric to evaluate the reconstruction ability of different CLIP encoders E (c.g., before and after finetuning). A lower loss indicates that the encoder captures more details from the input image, leading to a better reconstruction. To compute the expectation term Ex,e, in practice, we random sample noises e and timesteps (with x from the test set). However, since random sampling can affect result comparability, to make the results comparable, we sample and store two sets of (20) random (e, f) pairs for each x, and use the same sets across all evaluation trials (e-g., before and after finetuning). This pre-storing strategy makes the metric analogous to the "test loss" used in [63, 64]: although it shares the same formulation the training objective (Eq. (5)), it is computed with fixed randomness and test images. In contrast, the training loss is unsuitable for comparison because and f vary across iterations, leading non-comparable results.
Does Better Reconstruction Lead to Better Recognition? To answer this question, we first train our default un CLLP over different epochs and plot their losses and MMVP-VLM performances. As presented in Figure 3, we observe that models with smaller losses achieve better recognition performances. The reason behind this is that the diffusion loss is lower bound of the generative models' likelihood Ex |log (x E(x)).. which directly relates to our finetuning goal of capturing more visual details, as introduced in Section 3.2. Notably, since calculating the diffusion loss does not require task labels, it can be used as a tool for predicting the tendency of task performance when using our default design?.
Introducing Projection Layers. In the previous work DIVA [21], a linear projection layer is inserted between the CLIP image encoder and the generative model, because DIVA uses pretrained text- to-image model whose input space mismatches the CLIP image embedding in terms of embedding dimension and semantics. The projection layer is trained together with CLIP during finetuning. We investigate the impact of introducing a projection layer into our framework. Since the input space of unCLIP has already been aligned with CLIP image embeddings, inserting a projection layer is actually not needed in our framework and may break the alignment. Our experiments verified this. As shown in Table 5, we first conduct experiment using randomly initialized weights for the linear projector. Since the inserted projector alters the data flow in the pretrained CLIP-unCLIP model, achieves a higher loss than the original, causing downgraded performance. We further modify the initialization to be an identity weight matrix with zero bias, making it as if the projector does not exist at the beginning of finetuning. This modification enables the framework to make progress, as shown in Table 5. However, once the projector is updated, the alignment between the encoder's output and the generator's input does not hold, and may taking away part of the knowledge learned from the generator, leading to suboptimal performance compared to our default method. We also experimented with a two-stage training strategy for the projector, following [6], where the projector is trained first and then frozen during the image encoder finetuning. This approach yields slightly worse performance than our default setting. We attribute this to the fact that projection layers are unnecessary in our framework, as the image encoder and generator are already aligned in unCLIP. Even when the projector is pretrained separately, it remains challenging for it to fully reproduce the naturally aligned encoder-generatee environment established in unCLIP.
TTh conclusion would not hold if not following our default design, shown in the following ablations.

---

Updating the Generator G. As mentioned in Section 3.2, we set the unCLIP generator to be frozen during the finetuning process, encourage the finetuned encoder's outputs stay close the original unCLIPP's input space, i.e., the original CLIP image-text embedding space. Here, we examine the impact of updating the generator together with the encoder. The result presented in the last row of Table 5. It can be seen that the full finetuning achieves the best reconstruction, there are more parameters that can be tuned. However, similar to the issue the above-introduced learnable projector, updating the generator causes the finetuned encoder to move away from the original embedding space, resulting in a performance drop. In such cases, we cannot expect better recognition performances when observing better reconstructions.
Visualization Analysis. We give a visual- ization analysis of our default method and some representative ablations. Specifically, we use the finetuned encoder to perform the pipeline on an input im- age, thereby visualizing which features of the image are successfully captured. When sam- pling from different models, we use the same initial random noise to make the visualization results comparable. The results are shown in Figure 4. First, by comparing generated im- ages of the original CLIP and un'CLIP, it can be seen that after CLIP finetuning, the main patterns of the images success- fully captured, such as orientation of the first example and spatial position of the second example. Longer finetuning achieves better qualitative reconstructions, which aligns with the quantitative result in Figure 3. On the other hand, updating G achieves a visually comparable or slightly better reconstruction than our default (c.g., row 2, a better shape of the snowman), but in this case better reconstruction does not means a better recognition performance, as analyzed in the previous paragraph.
Sumpled Images Using the Finetuned Models Original CLIP un un'CLIP (le't 0.5, right: Lepoch) Updating G
Input Emage
GINA DAGIUS
CLOSED-
SOMAGING
FAPUKS
FARNICES
Figure 4: unCLIP generated images using original and finetuned CLIP models.
5 Conclusion
In this paper, we propose an image-based CLIP finetuning method un?CLIP to address the problem that pretrained CLIP models fall short in capturing visual details. By inverting a generative model that takes the CLIP image embedding as input, our method enables the finetuned CLIP to gain knowledge from the powerful generative model while preserving the alignment to its original embedding space simultaneously. Our method is simple yet effective, based on the key finding that the existing unCLIP generative model fits exactly to our goal. Extensive experiments across image and pixel level tasks demonstrate that by changing the original CLIP to our finetuned one, the performance of tasks to which CLIP has been applied and require visual detail capturing be significantly improved, such as open- vocabulayyylary semantic segmentation and vision-centric multimodal understanding.
Limitations. A potential limitation of this work is that finetuning CLIP requires pretrained unCLIP model first. Thankfully, the community has provided some pretrained unCLIP models, which are built upon widely used CLIP backbones. But if we take the computational cost of training the unCLIP into consideration, such an additional cost may be tolerable, given that improving the foundation model CLIP is an upstream work that could benefit many vision-centric downstream tasks where CLIP has been applied.
Acknowledgments
This work is partially supported by National Science and Technology Major Project 20217D01 11901 and National Natural Science Foundation of China 62376259.
10

---

References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763, 2021.
[2] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense lahels from CLIP. In European Conference on Computer Vision, pages 696-712, 2022.
[3] Feng Wang, Jieru Mei, and Alan Yuille. SCLIP: Rethinking self-attention for dense -sion-language inference. In European Conference on Computer Vision, pages 315 332, 2024.
[4] Mengeheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. ClearCLIP: Decomposing CLIP representations for dense vision lasion-language inference. In European Conference on Computer Vision, pages 143-160, 2024.
[5] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Rootstrapping amguage-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, pages 19730-19742, 2023.
[6] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36:34892- -6.34692-34916,1 2023.
[7] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved basclines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[8] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel C. F. Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. RegionCLIP. Region- based language-ing -mmgge pretraining. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 16793-16803, 2022.
[9] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal LLMs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568-9578, 2024.
[10] Monika Wysoczańska, Oriane Siméoni, Michaël Ramamonjisoa, Andrei Bursuc, Tomasz Trzcińskii, and Patrick Pérez. CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmen- tation. In European Conference on Computer Vision, pages 320-337, 2024.
[11] Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. ProXyCLIP: Proxy attention improves CLIP for open-vocabulary segmentation. In European Conference on Computer Vision, pages 70-88, 2024.
[12] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 0077310--7356, 2024.
[13] Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and LuLiu. MoVA: Adapting mixture of vision experts multimodal context. Advances Neural Information Processing Systems, 0011113355-033333 2024.
[14] Oguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. BRAVE: Broadening the visual encoding of rision-language models, European Conference on Computer Vision, pages 113-132. 2024.
[15] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang. Jianwei Yang, Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang. Jenq-Neng Hwang, Kai-We- Chang, and Jianfeng Gao. Grounded language-image pre-training. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Puttern Recognition, pages 10965-10975, June 2022.
[16] Yan Zeng, Xinsong Zhang, and Hang Li. hulti-grained vision language pre-training: Aligning texts with visual concepts. In International Conference on Machine Learning, pages 25994-26009 2022.
[17] Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-awart pretraining for open- vocabulary object detection with vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11 11154, 2023.

---

[18] Dong Jing, Xiaolong He. Yutian Luo, Nanyi Fei, Guoxing Yang. Wei Wei, Huiwen Zhao, and Zhiwu Lu. FineCLIP: Self-distilled region-based CLIP for better fine-grained understanding. Advances in Neural Information Processing Systems, 37:27896-27918, 2024.
[19] Bowen Shi, Peisen Zhao, Zichen Wang, Yuhang Zhang. Yaoming Wang, Jin Li, Wenrui Dai, Junni Zou, Hongkai Xiong. Qi Tian, and Xiaopeng Zhang. UMG-CLIP A unified -ulltl-grnullrity vision generalist for open world understanding, In European Conference on Computer Vision, pages 259-277, 2024.
[20] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchiaa text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06 22::2064440222 25, 2022.
[21] Wenxuan Wang. Quan Sun, Fan Zhang, Yepeng Tang, Jing Liu, and Xinlong Wang. Diffusion feedback helps CLIP see better. International Conference on Learning Representations, 2025.
[22] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision, pages 736-753, 2022.
[23] Mengde Xu, Zheng Zhang. Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open vocauuaccy semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2954,, 2023.
[24] Feng Liang. Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang. Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with maaskeddapted CLIP. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061-7070, 2023.
[25] Yu Du. Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14084-14093, 2022.
[26] Hanoona Bangalath, Muhammad Maaz, Muhammad Uzair Khattak, Salman H Khan, and Fahad Shah- baz Khan. Bridging the gap between object and image-level representations for open- nocn-vocabulary detection. Advances in Neural Information Processing Systems, 35:33781-33794 2022.
[27] Alexander Nichol, Prafulla Dhariwal, Adiyya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pages 16784-16804, 2022.
[28] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent difflusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[29] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SIXL: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2024.
[30] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikol aj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:237 1232237122336666 23736, 2022.
[31] Haochen Wang, Anlin Zheng, Yucheng Zhao, Tiancai Wang, Zheng Ge, Xiangyu Zhang, and Zhaoxiang Zhang. Reconstructive visual instruction tuning. In International Conference on Learning Representations, 2025.
[32] Jascha Sohl-Doe Dickstein, Eric Weiss, Niru Muheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265, 2015.
[33] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-683 6851, 2020.
[34] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8799 87900087742008 2021.
12

---

[35] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 5729-5739, 2023.
[36] Neehar Kondapaneni, Markus Marks, Manuel Knott, Rogerio Guimaraes, and Pietro Perona. Text-image alignment for diffusion based perception. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 13883- 13893, 2024.
[37] Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, and Chunhua Shen. Diffusion models trained with large data are transferable visual models. arXiv preprint arXiv: 2403,06090, 2024.
[38] Suraj Patni, Aradhye Agarwal, and Chetan Arora. ECoDepth: Effective conditioning of diffusion models monocular depth estimation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 28285-28295, 2024.
[39] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9492-9502, 2024.
[40] Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[41] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975 11986, 2023.
[42] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Annual Meeting of the Association for Computational Linguistics, 2018.
[43] Shijie Ma. Yuying Ge, Teng Wang, Yuxin Guo, Yixiao Ge, and Ying Shan. GenHancer: Imperfect generative models are secretly strong vision-centric enhancers, arXiv preprint arXiv:2503. 19480, 2025.
[44] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang-Was Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Trunsactions on Machine Learning Research, 2024.
[45] Mark Everingham and John Winn. The pascal visual object classes challenge 2012 (VOC2012) development kit. Pattern Anal. Stat. Model. Comput. Learn.. Tech. Rep, (107((1455):5 2012.
[46] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lec, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 991-898, 2014.
[47] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 1209-1218, 2018.
Thing and stuff classes in context. In
[48] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfekd, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 32 3313-32,,, 2016.
[49] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. International Journal of Computer Vision, 127:302-321, 2019.
[50] Baiqi Li. Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang. Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. NaturalBench: Evaluating vision- languae models on natural advessaiial samples. Advances in Neural Information Processing Systems, 37: 2024.
13

---

[51] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Prucessing, pages 292-305, 2023.
[52] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Wei- Chang. Song- Chug Zhu, Oyvind Tafjord. Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains science question answering, Advances in Neural Information Processing Systems, 35:2507-2222 22017,22211,22 2022.
[53] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang. Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. HallusionBemch: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14375- 14385, 2024.
[54] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing. Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM- as-a-judge with MT-bench and chatbot arena. In Neural Information Processing Systems Datasets and Benchmarks Track, pages 46595-46623, 2023.
[55] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hicrarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2009.
[56] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[57] Li Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Computer Vision and Pattern Recognition Workshop, pages 178-178, 2004.
[58] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In IEEE Conference Computer Vision and Pattern Recognition, pages 3485-3492. 2010.
[59] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 1306.5151, 2013.
[60] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In IEEE International Conference on Computer Vision Workshops, pages 554-561, 2013.
[61] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[62] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects context. European Conference on Computer Vision, pages 740-755, 2014.
[63] Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your difftusion model is secretly a zero-shot classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2206-2217, 2023. 202..
[64] Yinqi Li, Hong Chang, Ruibing Hou, Shiguang Shan, and Xilin Chen. DIVE: Inverting conditional diffusion models for discriminative tasks. arXiv preprint arXiv: 2504. 1 17253, 2025.
[65] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference- free evaluation metric for image captioning. In Proceedingy of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 7514-7528. 2021.
[66] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
[67] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. In Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: A summary of the paper's contributions is provided at the end of the introduction Section I.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings,
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the limitations in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If apppliaa,,,, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't I acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

The development of our method is theoretical driven, which bridges the training objective of generative model to our CLIP finetuning goal, as introduced in Section 3.2.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the implementation details of our method in Section 4.1 and Section B.2, and benchmark evaluation setups in Section 4.2-Section 4.5. We have also released our code and models at https: / //gi wtt.a/u.ccoomiiiii/////n/aaa
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but should be possible for other researchers to have some path reproducing verifying the results.
5. Open access to data and code
16

---

Question:
Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We have released our code and models at https://giite http://itthuu...mm//..ai.i/ com/LiYingi/
un2CLIP.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Betpi///guudde//ooeeuue www.ammmssinoooiccii.o.o. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips wttpcccpppppcccccccccc..ccccclct heabbiicgguidss.... /Codeeeodeeeeeeeeeeeeeee /ttaa//guiieeeeeeuuniiieeeeeee cnrionPollin onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
reproducible, they
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide training details such as dataset, pretrained models, hyperparameters in Section 4.1 1 and Section B.2, and test details such as benchmark and dataset, evaluation protocol in Section 4.2-Section 4.5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Because it would be computationally expensive for us to run all baselines and our methods multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We introduced computer resource and overall computational cost in Sec- tion 4.1. The detailed costs of each experiment are listed in Section B.3.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https /ttp:://nneu..eeeeee cttppppp..pppppeeeeeee cctppblic//aaaecccccccccccccctct Ctabblcc/thicsteee BEt//ehhiissiae csuuideliness
Answer: [Yes]
Justification: We have read the NeurlPS Code of Ethics and the research conducted in the paper conform it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consider- ation due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[NA]
Justification:
We conduct foundational research based on datasets that are public for research
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The dataset we use in this paper (CC3M) a filtered dataset, and the safeguards of the generative models we use (Stable unCLIP, which is built upon Stable Diffusion) are also considered by their creator.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guide lines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the papers of the datasets (CC3M), codes and models (Stable unCLIP) this paper built upon in Section 4.1. The name of the license is mentioned in Section B.2.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g,, CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thoode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We have not yet released our assets. We will release our code and models, together with usage instruction files, the paper is accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of L.LMs if an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: hatpp://nnurra //nnururpp.... wct./ooofeeencce/22555555 for what should or should not be described.
21

---

A Relationships and Differences to Prior and Contemporaneous Works
This paper leverages generative models to improve CLIP's visual detail capturing ability. There are two related works in this direction one prior [21] and one contemporaneous [43]. We discuss the relationships and differences with them below. An overview is summarized in Table 6.
Table 6: Comparison with related works. E stands for the CLIP image encoder and G stands for the generative model. *: with the generative model frozen.
Architecture
Work
Findings
Aligned with CLIP?
Generative Model
Projector
Large-scale pretrained text- conditioned model
Trainable with E
DIVA [21]
x
Small-scale projector-incond incorporated model trained from scratch
Pretrained with G
Reconstruction
Ganler [43]
/
# RRecognitinn
Large-scale pretrained CLIP-image- umbeedding-coniiiie model
Reconstruction t
un'CLIP
Free
+ Recognition 1
Comparison with Prior Work DIVA [21]. DIVA is a pioneering method in this line of research. employs pretrained text-to-image generative model, Stable Diffusion [28], as the backend to improve CLIP. Since the output space of the CLIP image encoder is not aligned with the conditional input space of Stable Diffusion, both in terms of embedding dimensionality and semantics, DIVA inserts a projector between the CLIP encoder and the generative model. The projector is trainable during CLIP finetuning, which may take away part of the knowledge learned from the generator. Different from DIVA, we use the unCLIP generative model, which takes the CLIP image embedding as input, thereby achieving projector-free framework, enabling seamless and effective process for CLIP encoder enhancement.
Comparison with Contemporaneous Work GenHancer [43]. GenHancer is contemporaneous work that appeared online on March 25th, 2025. Different from DIVA and our un²CLiP that use existing well-trained generative models, GenHancer trains generative models from scratch specifically to improve the CLIP, and draws a main conclusion that "perfect generation (reconstruction) does not always yield desirable visual representations". We analyze the key relationships and differences below:
(1) Relationship in Architecture: Projector-incorpraate vs. Projector-free. Similar to DIVA but different from ours, GenHancer introduces a projector between the CLIP encoder and the generative model. Unlike DIVA, the projector in GenHancer is pretrained together with the generative model in a pretraining stage, prior to CLIP finetuning. This design can be interpreted as the projector is incorporated into the generative model, allowing the generative model's 's input to align with CLIP's output at the CLIP finetuning stage. Viewed this way, both GenHancer and our CnL achieve an aligned, seamless encoder-generrtor pipeline, which facilitates more effective enhancement of the CLIP image encoder.
(2) Difference in Architecture: Capability of the Generative Model. GenHancer does not utilize existing large-scale pretrained generative models. Instead, it trains lightweight generative models from scratch, using the CC3M dataset [42] for single epoch. In contrast, our method leverages the pretrained Stable unCLIP model, which is built upon Stable Diffusion and trained with approximately 200,000 A100 hours This pretrained backbone offers substantially greater model capacity. Given that our optimization objective (Eq. (4)) is accurately estimating the conditional distribution p(x Ex)), a more capable generator provides stronger training signal for improving the CLIP encoder. Moreover, using a well-trained generator enables visualization of how much the encoder has improved, and could provide intuitive explanations for the prediction behaviors of the models, through the generated images via the encoding-decodiin pipeline, as demonstrated in Figure 4 of the main paper and Figure 5 of this appendix, compared with Figure 4 in GenHancer.
(3) Divergent Findings: Relationship between Reconstruction and Recognition Performance. GenHancer reports that "perfect generation (reconstruction) does not always yield desirable visual
2- wun.iiptennvirommenaa-immm
22

---

representations" in its Figure 1. Our conclusion differs, as shown in the ablation studies in Section 4.6 of the main paper. The key distinction lies in how reconstruction quality is measured. We use the diffusion loss, which is the lower bound of the generative model's s likelihood, as the measurement for reconstruction as well as our finetuning objective. A lower diffusion loss indicates that more process. Therefore, as for our framework with
information is preserved in the encoding-decodde freezing the generator during the encoder's s finetuning, better reconstruction leads to improved recog- nition performance in visual detail tasks, as analyzed in Section 4.6. On the other hand, GenHancer uses CLIP score [65] as the measurement for both generation and reconstruction. Since the CLIP score primarily measures image-text alignment rather than pixel-level fidelity, it is more effective for evaluating text- faitfuiitsssssess of generated images of text-to-iman models, but suboptimal for evaluating visual reconstruction quality of image-to-ing -omeggg ancoing-ddecodingg pipelines.
B
Additional Experimental Details
B.1 A Toy Experiment Validating Alignments between OpenAl ViT-L-14@224 and ViT-L--143336 336
As described in Section 4.1 of the main paper. we find that the image and text encoders of OpenAI CLIP CLIP ViT-L- @422 and OpenAI CLIP VTT-L-14G @ 336 have similar embedding space. Therefore, use the pretrained Stable unCLIP for OpenAT CLIP ViT-L-14@222 to improve both of therey avoiding the cost of training a separate unCLIP model for OpenAl CLIP ViT-L-144@33.
This observation based on the following toy experiment, which we swap the image and text encoders between the two CLIP models and evaluate the resulting combinations. We use the zero- shot classification task for this experiment. The results are summarized in Table 7. We find that (I) the two original models have similar performance, and (2) after swapping their image and text encoders, the hybrid models retain comparable performance to the originals. These results suggest that the embedding spaces of the two models are closely aligned. Therefore, for efficiency, use the pretrained unCLIP model for OpenAI CLIP ViT-L- 224 to improve both of them during our un-CI CLLIP training.
Table 7: Zero-shot classification performance of image-text encoder swapped CLIP models. The two CLIP models are OpenAl CLIP ViT-L-14@224 and OpenAl CLIP ViT-L-14@336
Image Encoder
Text Encoder
IN-IK
C-10
C-100
Cal-101
SUN397
Aircraft [59]
Cars
[55]
[56]
[56]
[57]
[58]
[60]
@224
@224
75.5
95.6
75.9
86.7
67.6
31.7
77.9
@336
@336
76.6
94.9
74.4
87.2
68.7
33.4
79.3
222
@336
75.4
95.5
76.0
86.7
67.7
31.5
78.0
@336
@224
76.5
95.0
74.5
87.2
68.5
33.4
79.4
B.2 Pretrained Stable unCLIP models
We use the code (MIT License)* and models (CreativeML Open RAIL++-M License) of Stable unCLIP to implement our method. This release includes two pretrained unCLIP models, conditioned on OpenAl CLIP VIT-L-14@22) 224 [1] and OpenCLIP ViT-H-14@224
[40] image embeddings. These two models are finetuned versions of the stable-difffsion-22 model adapted to accept CLIP image embeddings conditional inputs. Stable Diffusion is type of latent diffusion model [28] that performs denoising in the latent space of a pretrained KL-VVAR [66] model. The latent size of and therefore of the two unCLIP models, 96x96x4. The details of these models are summarized in Table 8. As explained in Section 4.1 and Section B. we use the same unCLIP model to improve both the OpenAl CLIP ViT-L- 14 224 and OpenAI CLIP VIT-L-14@333 336 image encoders during our un CLLPP training stage.
saale-diffusionn
As introduced in Section 4. to evaluate the generality of our approach across different CLIP back- bones, we additionally train a new Stable unCLIP model conditioned on SigLIP ViT-S0-14@384 [41]
23

---

Table 8: Model hyper-parameters of Stable unCLIP and training costs.
Conditional Branch
Denoising Backbone
unCLIP
un CLIP
CLIP Image Encoder
#Params
Input Size
#Params
Training Cost
Training Cost
OpenAl CLIP ViT-L-14 224
303 M
96x96x4
869 M
Pretrained
30h
OpenAl CLIP
304 M
96x96x4
869 M
N/A
30h
OpenCLIP VTT #424
632 M
96x96x4
870 M
Pretrained
32h
SigLIP ViT-SO-14@384
428
64x64x4
869 M
~dd
15h
image embeddings, based on the open-source implementation mentioned above. To reduce training cost, this model is built upon space of 64x64x4. We train this unCLIP model on the CC3M dataset [42], using global batch size of 2048 following the configuration of stable-diffusion catesiionn1---- 2-1-base. The model is trained for 15K iterations (about 10 epochs over CC3M), taking about 5 days with Nvidia-A100-40GB GPUs, as summarized in the last row of Table 8.
which operates in a smaller latent
B.3 Computational Costs
The training cost of un'CLIP in each experiment is reported in the rightmost column of Table 8, where the SigLIP experiment, due to the smaller input size to the main denoising network, has relatively faster training speed. The full research project, including some preliminary, failed, ablative, and downstream task experiments, takes about 3 4 times the sum of the reported training costs.
C Additional Ablation Studies
C.1 CLIP Training (CLIP Finetuning) Dataset
By default, we follow DIVA [21] and use CC3M [42] as the training dataset for fair comparison. We also note that the concurrent work GenHancer [43] adopts the same setting. However, none of these works investigate the effect of the training dataset choice, leaving this this section, we study how the choice of training dataset within our Un framework affects the generalizability of the finetuned model. For this ablation, evaluate not only the MMVP-VIM benchmark but also on open-vocabulary segmentation tasks to comprehensively assess generalization.
open question. In
Training ImageNet-IK. We first replace CC3M with ImageNet-IK [55]. class-balanced dataset containing ~1.3 images across 1000 categories. We keep the total number of training iterations the same as with CC3M to isolate the effect of dataset content. As shown in Table 9 (segmentation results averaged over datasets), the ImageNet-finee finetunedddd model achieves competitive results with the CC3M- finetunid one MMVP-VLM. This may be because the MMVP-VLL LLM dataset is relatively small, and its image patterns can well covered by both CC3M and ImageNet training sets. In fact, part of MMVP-VI.M's images originate from ImageNet, introduced in [9].
However, on segmentation benchmarks, the ImageNet-finetie finetuned CLIP does not yield consistent improvements. Notably, ImageNet- K is highly structured dataset (c.g., class-balanced, lacking the "person" class), which differs from web-collected datasets such as CC3M, CLIP's pretraining dataset WebImageTen, Text [1], and Stable unCLIP's training dataset LAION-5B [67]. This raises the question of whether is the dataset content (distribution) or datasel scale that results in poor generalizability.
Training on 10% CC3M images. To further investigate this question, we train another model using randomly sampled 10% subset of CC3M (~0.3M images), even smaller than ImageNet- 1K in scale. As shown in the last row of Table 9, this model exhibits better generalization than the maageNet-finetunel one, suggesting that dataset distribution (e.g., source diversity and similarity to CLIP's pretraining data) plays more crucial role than dataset scale in achieving better generalized finetuned CLIP models.
http:://wwggiigghhcc.oosaiiiiist 1-base
24

---

Table Ablation study on CLIP training datasets.
CLLIP Training Dataset Scale
MMVP
CLIP
MaskCLIP
SCLIP
ClearCLIP
Method
VV.M
Seg.
Seg.
Seg.
Seg.
Original CLIP
N/A
19.3
14.7
34.4
un'CLIP CC3M (default)
~3 M
32.6
5,8
15.6
25,9
34,8
CI ImageNet-IK
11.3 M
32.6
13.9
24.0
33.0
CI.II 10% CC3M
-0.3 M
31.1
5.8
15.3
25.9
34.5
C.2 Training with Image-Text Data
In addition to the generator-frowe frozen strategy, we further explore using image-text supervision to mitigate the potential language-shift problem during image encoder finetuning. Specifically, incorporate an additional image-text loss into our training objective. updating both the image encoder E and the generator G. This loss is implemented as the negative cosine similarity between image and text features from the CLIP encoders, using paired image-text data from CC3M. Jointly optimizing the two objectives introduces a balancing hyperparameter between the default unCL.P--inversonn loss and the introduced image-text loss, for which we test two values (1 and 0.1).
Before conducting experiments, note that the added image-text objective may reintroduce the very issue this paper aims to address: captions often describe coarser-grained semantics than images themselves, potentially making the finetuned model less sensitive visual details. Results summarized Table 10. We observe that: (1) Simply combining the two losses with equal weight (#4) causes performance drop compared
introducing the image-text loss (#3), suggesting that the added supervision indeed hampers the model's ability to capture visual details, as hypothesized; and (2) Using a smaller weight for the image-text loss (#5) yields better results, indicating a more balanced trade-off between visual-detail capturing and image-text alignment during optimization. However, #5 still falls short of #2. These results demonstrate that our default strategy, leveraging the mmage-text aligned space of unCLIP and updating only the encoder, remains simple (no additional losses hyperparameter tuning). effective (achieving the best performance), and efficient (fewer trainable parameters).
Table 10: Effect of incorporating image-text supervision during un CLIP training.
#
Method
un 2 CLIP Training Loss
MMVP-VLM
1
Original CLIP
N/A
19.3
2
Default (Training only)
unCLIP-inversee loss
32.6
Training E and G
unCLIP-inversion
27.4
4 Training E and G
unCL.PPiivvessio loss + 1 x image-text loss
25.9
5 Training E and G
unCLIP-inversion loss + 0.1 x image-text loss
28.9
D Additional Qualitative Results
D.1 MMVP-VLM Visualization Results
Figure 5 presents qualitative examples of the MMVP-VLM benchmark. For each case, also apply the CLIP-
pipeline to both the original and our improved CLIP models, as done in the visualization analysis paragraph in Section 4.6 of the main paper. These generated images help provide intuitive explanations for the prediction behaviors of the models illustrating why the original (our improved) CLIP model makes incorrect (correct) predictions for cases.
25

---

Candidate Images
Sampled Images Original CLIP
Candidate Images
Sampled Images
Query Text
unFCLIP
Query Text
Original CLIP
unACLIP
Ribif Oude
snowman with clothes
sign with black background
Closed
snowman without clothes
a sign with red background
CLOSED
SOIGGING
FAPUKS
butterfly with wings open
glass drink
butterfly with wings closed
of drink
Figure 5: MMVP-VLM visualization results. Predictions of the original CLIP and our improved CLIP shown with red and blue arrows, respectively. Generated images using the CLIP-UnCLIP encoding-decoddig pipeline are shown at right, providing visual insight into each model's 's predictions.
26

---

D.2 MLLM Visualization Results
Figure presents qualitative examples of MLLM tasks, focusing on vision-centric benchmarks.
Question:
Original
untCLIP
Are some fruits cut open or are all fruits uncut?
#1: (a)
#1: (a)
(a) Some cut open
(b) All are uncut
#2: (a)
#2: (b)
Question:
Original
un'CLIP
In the picture, how is the snake primarlly positioned?
(b)
(a)
(b)
(b)
(a) Coiled
(b) Head extended
#1
#2
MMVP
Question:
Original
un'CLIP
the dog wearing number clearly the lead?
Yes.
there ladder background?
Yes.
No.
Question:
Original
un²CLIP
Is the dog wearing number 2 clearly in the lead?
No.
No.
Is there ladder in the background?
Yes.
Yes.
NaturalBench
Question:
Question:
How many clocks
Considering relative positions the bowl (annotated by the box) and the broccoli in the image provided, where is the bowl locat-ed with respect to broccoli?
the image?
(A) (B)
(C) (D) 0
(A) Left
(B) Right
Original (A] 2
x
Original
(B) Right
X
un2CUP
(B)
un²CUP
(A) Left
CV-Bench 2D
Question:
Question:
Which object closer the camera taking this photo, the table (high- lighted by box) the television (highlighted by blue box)?
Estimate the real-world distances between objects this image. Which object closer the lamp (highlighted by red box),
nightstand (highlighted blue box) or dresser
(A) table
(B) television
(highlighted by green box)?
(A) nightstand
(B) dresser
Original
television
Original
(B) dresser
unPCLIP
(A) table
unPCLIP
(A) nightstand
CV-Bench 3D
Figure 6: Vision-centric MILM visualization results.
27


Reviews:
[{"review_id": "RIYJo441uI", "reviewer": "Reviewer_rnBc", "summary": "This paper applies the unCLIP-inversion framework, which finetunes the image encoder of CLIP with a frozen generative module of unCLIP. Benefiting from the generator conditioned on the CLIP image embedding, the image encoder is able to capturing more rich visual details and improve the performance of MMVP-VLM, open-vocabulary segmentation and vision-centric MLLM tasks.", "questions": "please see weaknesses.\nThe method in this paper is reasonably designed and has superior performance compared with other CLIP methods that introduce generators. However, it is limited to comparisons with such methods and cannot strongly prove the promotion of the generative model to the image encoder of CLIP. Moreover, this paper also needs to prove that the generator will not undermine the initial image-text alignment ability of the CLIP model.", "limitations": "see weaknesses", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strength\n1. This work makes full use of the pre-alignment of CLIP image features and generative models in unCLIP, allowing the image encoder to benefit from the generative model in a relatively unified feature space.\n2. The method in this paper only relies on pure image data and has a relatively low fine-tuning cost.\n\nWeakness\n1. Although this method freezes the text encoder of CLIP and claims to retain the initial image-text alignment capability of CLIP, the lack of comparison with the base CLIP model (such as SigLIP ViT-SO-14@384) in the experiments raises doubts about whether generative fine-tuning would undermine the initial discriminative ability of CLIP.\n2. The comparative methods of the experiment only selected models that introduced the generator. Although this method outperforms them, to prove that the generator indeed brings more abundant visual details and knowledge to the image encoder, comparisons with other CLIP methods that do not use the generator are necessary.", "was_revised": true, "final_justification": "Thanks for the author's clarification. My concerns have been addressed.", "rebuttal": "We thank the reviewer for the detailed review and for recognizing the reasonableness of our method and its superior performance compared to other generator-based approaches.\n\nWe have carefully taken every comment and our responses are as below. We are more than willing to provide additional clarification should any questions remain. If you find that your concerns have been resolved, we would be grateful if you could consider updating your score.\n \n> **W1. Lack of comparison with the base CLIP model (such as SigLIP ViT-SO-14@384) in the experiments, which raises doubts about whether generative fine-tuning would undermine the initial discriminative ability of CLIP.**\n\nActually, *in all of our experiments (Tables 1-3 in Section 4.2-4.4), our method have been compared with the base CLIP models.* The **\"Original\"** rows in Tables 1-3 represent the performance the base, original pretrained CLIP models. And it can be seen that our un$^2$CLIP finetuned models outperform the base CLIP models on these vision-detail-related tasks. For example, in the last group of Table 1, un$^2$CLIP outperforms the base CLIP model SigLIP ViT-SO-14@384 on the MMVP-VLM benchmark (41.5 vs 37.0). These results suggest that our generative finetuning does not undermine the discriminative ability of CLIP, but rather improves its capacity to capture fine-grained visual information.\n\nFurthermore, we checked our manuscript and find that this confusion may stem from that we did not highlight in the \"Compared Methods\" paragraph of Section 4.1 that the \"Original\" CLIP model is also one of the compared baselines in our experiments. We will add this clarification in that paragraph of the revised paper to make this point clearer.\n\n---\n> **W2. & Q.(1). The comparative methods of the experiment only selected models that introduced the generator. Comparisons with other CLIP methods that do not use the generator are necessary.**\n\nWe speculate that this concern may be related to the reviewer's previous **W1** question. As mentioned in our response to the previous question, our experiments include comparisons with the original base CLIP models, which do not involve the generator.\n\n*Regarding other methods,* to the best of our knowledge, existing works that focus on improving pretrained CLIP's visual detail capturing ability, at the upstream task-agnostic level, are all generative model based approaches, including the previous DIVA and the concurrent GenHancer. *These are directly compared with in our experiments, and our choice of baselines is consistent with these works.* If the reviewer could point out the some specific works that should be also compared with, we would be more than willing to discuss with and try our best to conduct further experiments to give empirical comparisons during the following discussion period.\n\nP.S. Taking W1 & W2 together, considering that the reviewer may also be referring to a comparison between our un$^2$CLIP (with the CLIP backbone) and the SigLIP model, below is our response from this aspect: \nOur goal is not to develop a stronger CLIP backbone like SigLIP, but to enrich existing pretrained CLIP's visual detail capture ability. SigLIP involves a different training objective and configuration compared to the CLIP, making it not directly comparable in this context. Such comparisons are also not adopted in prior or concurrent works in this field (DIVA and GenHancer). That said, our method (and related work DIVA and GenHancer) are orthogonal to and can be combined with different CLIP backbone improvement works such as SigLIP. As shown in Table 1 of our paper and mentioned in our response before, our method can improve such backbones on vision-detail-related tasks.\n\n---\n> **Q.(2). The paper needs to prove that the generator will not undermine the initial image-text alignment ability of the CLIP model.**\n\nWhen developing our approach, we have taken the image-text alignment ability into consideration. Specifically, since the input space of the unCLIP generator lies within the CLIP initial image-text embedding space, for reducing the potential language-shift when finetuning the image encoder, we freeze the parameters of the unCLIP generator to encourage the optimized image embeddings staying close to unCLIP's original input space. These are introduced in Section 3.2.\n\nMoreover, *the experimental results in Section 4.2 (MMVP-VLM evaluation) and 4.3 (open-vocabulary segmentation) implicitly validated the preserved image-text alignment property*: These two tasks both use the original CLIP text encoder together with the finetuned image encoder to obtain the prediction results, meaning that the alignment should be maintained for successful performance. \n\nFor instance, the segmentation results in Section 4.3 are obtained by comparing the local patches of image features to the candidate text features. The results on various datasets (COCO, ADE, etc.), containing various classes including both indoor and outdoor scenarios, show the improved performance when using our finetuned model. On the contrary, if the finetuned model breaks the image-text alignment ability, it would be difficult to achieve consistent improvement over different datasets."}, {"review_id": "SkugN77TKM", "reviewer": "Reviewer_j8dh", "summary": "This paper proposes un²CLIP, a method to enhance CLIP’s ability to capture fine-grained visual details without requiring additional text-annotated data or retraining from scratch. By leveraging the unCLIP generative framework—where a diffusion-based image generator is conditioned on CLIP image embeddings—the authors invert the process: they finetune the CLIP image encoder using the fixed unCLIP generator so that the encoder’s embeddings retain more image information while remaining aligned with the original text encoder. The proposed approach is shown to yield significant improvements over both baseline CLIP and prior generative model-based enhancements (notably DIVA) on benchmarks such as MMVP-VLM, open-vocabulary semantic segmentation (dense prediction), and vision-centric multimodal LLM tasks.", "questions": "See the weaknesses. I generally think the method is simple and elegant, but the performance on some benchmarks hinders me from giving high scores.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Though the idea of inverting unclip is simple, the method is well-motivated, and the experimental results are quite good. It can improve the performance of CLIP on multiple tasks and benchmarks. Compared to other competitors, the improvement is significant.\n- The paper provides comprehensive experiments, including multiple baselines, detailed ablation studies, and both quantitative and qualitative results that strongly support its claims.\n- The paper is generally well-structured and readable, with logical development from motivation, through theoretical grounding, to empirical validation.\n\nWeaknesses:\n- My main concern is that the improvement of un^2CLIP on MLLM evaluations is quite marginal. As shown in table 3, it performs similar to DIVA on MMVP and general benchmarks, which indicates the general image understanding ability of un^2CLIP remains the same. While MLLM related tasks are important, this is one of the most important metrics to evaluate the image encoder. Also, on traditional tasks like zero-shot classification and retrieval, its performance drops compared to original CLIP, showing that generative training focuses more on details while ignoring the global semantics.", "was_revised": true, "final_justification": "My concerns have been addressed. Overall, the idea of this work is interesting, but the novelty seems a little limited. I lean to accept the paper, but I will not debate if it is rejected.", "rebuttal": "We thank the reviewer for the through review and the positive comments on the paper's motivation, writing, the elegance of the method, and the comprehensive experiments.\n\nBelow, we provide point-by-point responses to the reviewer's concerns. We are more than willing to provide additional clarification should any questions remain.\n\n> **Table 3: Performances of un$^2$CLIP and DIVA on MMVP benchmark are similar.**\n\nThis may be partially attributed to the relatively small scale of the MMVP benchmark, which includes only 150 sample pairs. A relatively smaller sample size may reduce the statistical significance of performance differences, especially in some competitive settings. In contrast, on other vision-centric benchmarks that contain more evaluation samples (e.g., CV-Bench 2D COCO that contains 800+ samples), un$^2$CLIP shows more consistent and significant improvements over prior methods.\n\n---\n> **Table 3: Performances on general benchmarks are similar, which indicates the general image understanding ability of un$^2$CLIP remains the same.**\n\n**We agree with the reviewer’s understanding.** General MLLM benchmarks typically do not involve questions that require visual detail understanding ability. For example, in the POPE benchmark that evaluates object hallucination, questions are typically of the form \"Is there a [something] in the image?\". Such questions usually rely more on high-level semantics than visual details. And it can be seen in Table 3 that most CLIP visual detail improvement methods, including ours, perform similarly to the original CLIP on these benchmarks. **These general benchmarks are not the main focus of our work (and other related CLIP improvement methods) since we aim to improve CLIP's visual detail capturing ability.**\n\nOn the other hand, the maintained performance to the original model on these general benchmarks *suggests that our finetuned CLIP model preserves the general, global semantics on MLLM tasks*, which is different to the conclusion drawn from the results of the traditional zero-shot classification task. We give some analysis in our response to the next comment below.\n\n---\n> **On traditional tasks (zero-shot classification and retrieval), un$^2$CLIP's performance drops compared to original CLIP, showing that generative training focuses more on details while *ignoring* the global semantics.**\n\nWe would like to point out that the global semantics are not **\"ignored\"** by the un$^2$CLIP-finetuned CLIP model. A more exact conclusion is that the global semantics are **not highlighted** in the un$^2$CLIP-finetuned CLIP model (i.e., global semantics **still exist**). Below are our analysis.\n\nFirst, as noted in our response to the previous comment, the results on MLLM general benchmarks suggest the finetuned CLIP have similar general image understanding ability with the original CLIP model. However, it is different from the observation of the results on the zero-shot classification task where the performance drops are relatively larger.\n\nThe key difference of these two tasks is that the CLIP features fed to MLLM are processed by later learnable heads (projector and LLM in the system), while in zero-shot classification, CLIP features are directly used for the final prediction. The learnable heads in the former architecture offers a re-extracting opportunity for the required (global) semantics. While the learnable heads do no appear in the zero-shot classification framework. **This means that the frozen CLIP model should have highly aggregated global features to achieve better results on the training-free classification task, which does not exactly measure how the global semantics exist in the model.*\n\nTo further illustrate, let us take an example from the visualization result of the open-vocabulary semantic segmentation experiment. In the rightest column of Fig. 2 in our paper (the giraffe example), it can be seen that the original model extracts more foreground semantic (giraffe) on the final output, while our finetuned model assigns more accurate semantics to the image. The behaviour of the original model, although misclassified many background areas, is obviously more easier to achieve success in the global-semantic classification task (the extreme case would be that it offers the global semantic label to every pixel in the image). On the other hand, it can be also seen that our finetuned model **does not \"ignore\"** the global semantic of the image. It just assigns the semantic to correct areas. Such a behavior makes it fall behind on the traditional classification tasks."}, {"review_id": "JbP61wHVKI", "reviewer": "Reviewer_dYip", "summary": "This paper proposes an enhancement to CLIP's ability to capture visual details by utilizing gradient feedback from unCLIP. Unlike its closely related work, DIVA, this approach specifically addresses the issue of feature space misalignment between the input of the generator and the output of CLIP's image encoder. To implement this, the main technical modifications include replacing the text-to-image generator with unCLIP and removing the MLP projector. The effectiveness of the proposed method is demonstrated through various benchmarks that assess visual detail accuracy.", "questions": "please refer to the weakness part.", "limitations": "The limitations are properly discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n\n- The proposed method demonstrates strong effectiveness, which is thoroughly validated through comprehensive experimental evaluations.\n\n- The paper's focus on addressing the misalignment between the input space of generators and the output space of the CLIP image encoder is reasonable.\n\n- The qualitative improvements in visual detail capture, as clearly illustrated in Figure 4, provide compelling evidence for the benefits of the fine-tuned CLIP approach.\n\n### Limitations:\n- Novelty:\nThe technical contribution in comparison to DIVA is somewhat limited.\n\n- Performance Trade-off:\nThe reported degradation in zero-shot classification performance (Tab.8, Appendix C) raises concerns. While the authors attribute this to the model's preference for fine-grained details, the superior performance of DIVA (a comparable approach) undermines this explanation. Further investigation into this performance discrepancy would strengthen the paper.\n\n- Implementation:\nTraining a separate unCLIP generator for each CLIP image encoder creates significant computational overhead and limits implementation flexibility. Moreover, the effect of the scale of the training dataset on the generalizability of the CLIP image encoder during the unCLIP training process remains an unresolved question that deserves further discussion.\n\n- Training Procedure:\nA two-stage ablation—first training the MLP projector and then fine-tuning the image encoder while keeping the projector's weights fixed—could provide clearer insights into the role of the projector head. The current joint training strategies in the ablation may introduce unnecessary instability to the optimization process.", "was_revised": true, "final_justification": "Most of my concerns have been addressed. I will keep my original rating.", "rebuttal": "We thank the reviewer for the thorough review and constructive comments. We are glad that the reviewer finds the paper's focus reasonable, the experimental evaluation comprehensive and effective, and the qualitative results compelling.\n\nBelow we provide point-by-point responses to the reviewer's concerns. We are more than willing to provide additional clarification should any questions remain.\n\n> **1. Novelty: The technical contribution in comparison to DIVA is somewhat limited.**\n\nWe acknowledge that our approach shares the high-level similarity with DIVA in using pretrained generative models to improve CLIP. The key contribution of our work lies in *dentifying and addressing the encoder-generator feature space mismatch issue* which is not well-addressed by prior work. Specifically, we *find* that an existing type of generative models (*unCLIP*) *offers a more seamless and effective CLIP enhancement framework*, and achieve better performance on various benchmarks.\n\n---\n> **2. Further explanation of the performance degradation in zero-shot classification task.**\n\nAs discussed in our response to Reviewer j8dh's last comment (where we use the visualization result of the open-vocabulary semantic segmentation task, i.e. Figure 2, to help analyze models' preferences), **our finetuned model corrects most of the foreground or other false predictions in background areas that the original CLIP model tends to rely on for classification**, resulting in performance degradation in the classification task. This is consistent with our previous attribution wroten in Appendix C (model's preference for fine-grained details).\n\nRegarding DIVA's performance, it can be seen in Figure 2 that **DIVA behaves similarly to the original CLIP in segmentation** (quantitative results in Table 2 also verify this), such as showing many foreground semantics incorrectly predicted in background areas (the rightmost giraffe example) or showing some other false predictions in background areas (the leftmost cow example), making the foreground semantics dominate the prediction results. This phenomenon can explain DIVA's maintained zero-shot classification performance compared to the original CLIP model shown in Table 8 in Appendix C.\n\nP.S. We may also observe that although DIVA shows marginal improvements on the pixel-level segmentation task in Table 2 and Figure 2, it achieves notable improvements on another visual-detail-related task MMVP-VLM in Table 1 (although falls behind our method). This may be because the MMVP-VLM benchmark emphasizes recognition of *visual patterns* (e.g., object orientation, state, relation, etc.), which are less related to the vision *class labels* used in segmentation and classification benchmarks. Therefore, the visualization of segmentation results can offer useful insight into classification performance of different models but are relatively *less* indicative for explaining performances on the MMVP-VLM benchmark.\n\n---\n> **3.(1). Training a separate unCLIP generator for each CLIP image encoder creates significant computational overhead and limits implementation flexibility.**\n\nThis is indeed a limitation of our approach which has been summarized in the last paragraph of Section 5. And we have found that (in Appendix B.1, and mentioned in Section 4.1) some existing pretrained CLIP models share similar embedding spaces thus they can share a same unCLIP model for reducing computational overheads, which is a trick solution for some models.\n\nIn addition, training unCLIP generators actually does not take too many resources because there exist many open-sourced well-trained powerful generative models in the community such as Stable Diffusion. And in the codebase used in our experiments the unCLIP models are finetuned versions of Stable Diffusion, making the training overhead relatively affordable. For example, the unCLIP model conditioned on pretrained SigLIP embeddings used in the experiment is trained by ourselves, which took about 5 days on 8 A100-40G gpus. Such an experimental scale is much smaller than training large generative models from scratch, and can be implemented in a simple single-node experimental environment. These details are summarized in Appendix B.2.\n\n---\n> **3.(2). Effect of the scale of the training dataset during ~~unCLIP~~ un$^2$CLIP training process.** \n\n(We assume the reviewer intended to refer to un$^2$CLIP rather than unCLIP, as the generative models like unCLIP or Stable Diffusion are usually adopted from existing checkpoints whose pretraining dataset scales are typically in large-scale such as LAION-5B to ensure general generation capability.)\n\nWe thank the reviewer for raising this important question. The effect of the training dataset scale during the CLIP's generative finetuning process is indeed an unresolved question in this line of research. By default, we follow DIVA and use CC3M as the training dataset for fair comparison. We observe that the concurrent work GenHancer also adopts this setting. And there are no related ablation studies about the training dataset in all of these papers, making this issue an open question.\n\n**Experimental Setup.** Following the reviewer's suggestion, we study the effect of the training dataset scale within our un$^2$CLIP framework on the generalizability of the finetuned model. For this ablation experiment, we evaluate not only on the MMVP-VLM benchmark but also on open-vocabulary segmentation tasks to comprehensively assess generalization.\n\n**Training on ImageNet-1K.** We first switch from CC3M to another widely-used dataset ImageNet-1K, a class-balanced dataset with ~1.3M images across 1000 categories. We keep the number of training iterations the same as with CC3M to isolate the effect of dataset content. As shown in the table below (segmentation results are averaged over 8 datasets), the ImageNet-finetuned model achieves competitive results with the CC3M-finetuned one on MMVP-VLM. This may be because the MMVP-VLM dataset is on a small scale, whose image patterns can be well covered by both CC3M and ImageNet training sets. (Actually, part of the images in MMVP-VLM are taken from ImageNet, as introduced in MMVP-VLM's paper [9].)\n\nHowever, on segmentation benchmarks, interestingly, the ImageNet-finetuned CLIP cannot bring consistent performance improvement. Note that ImageNet-1K is a highly structured dataset (e.g., class-balanced, not containing the \"person\" class), which is different from web-collected datasets, such as CC3M, CLIP's training dataset WebImageText, and unCLIP's training dataset LAION-5B, in terms of data distribution. This raises doubts about **whether dataset content (distribution) or dataset scale results in poor generalizability**.\n\n**Training on 10% CC3M images.** To further investigate the above-mentioned question, we conducted another experiment training on a randomly sampled subset of CC3M, with the training sample numbers (~0.3M) even smaller than ImageNet-1K. As shown in the last row of the table, this model achieves better generalizability than the ImageNet-finetuned one, suggesting that **finetuning dataset distribution** (e.g., source diversity, similarity with CLIP's pretraining dataset) **plays a more crucial role** than finetuning dataset scale in achieving better generalized finetuned CLIP models.\n\n|Method|un$^2$CLIP Training Dataset Scale|MMVP-VLM|CLIP Seg.|MaskCLIP Seg.|SCLIP Seg.|ClearCLIP Seg.|\n|-|-|-|-|-|-|-|\n|Original CLIP|N/A|19.3|5.3|14.7|24.0|34.4|\n|un$^2$CLIP - CC3M (default)|~3M|**32.6**|**5.8**|**15.6**|**25.9**|**34.8**|\n|un$^2$CLIP - ImageNet-1K|~1.3M|**32.6**|5.3|13.9|24.0|33.0|\n|un$^2$CLIP - 10% CC3M|~0.3M|31.1|**5.8**|15.3|**25.9**|34.5|\n\n---\n> **4. Two-stage training of the ablation experiment about the projector.**\n\nThank you for this constructive suggestion. We agree that training the projector first then freezing it during the image encoder finetuning leads to a more stable optimization process compared to the joint training strategy used in our initial ablation. Following the reviewer’s advice, we have conducted this two-stage training experiment:\n\nAn implementation detail is that the first stage's configuration adopts a larger learning rate (1e-5) than the second stage, because the number of trainable parameters is relatively smaller (only the projector). Similar configurations are also adopted in other multi-stage training frameworks such as LLaVA.\n\nThe last row of the following table reports the result. We observe that it achieves a slightly suboptimal performance compared to our default setting. This is because projection layers are actually not needed in our framework, where the image encoder and generator are already aligned in unCLIP. In this 2-stage experiment, even though the projector was separately trained in advance, it may still be difficult for the projector to learn to achieve the previous encoder-generator aligned environment perfectly.\n\n|Method|MMVP-VLM|\n|:-|:-:|\n|Original CLIP|19.3|\n|un$^2$CLIP default (no projector)|**32.6**|\n|Jointly trained projector (random init.)|16.3|\n|Jointly trained projector (identity init.)|30.4|\n|2-stage training strategy|30.4|\n\nWe will incorporate these added experiments into the revised paper to make the ablation studies more comprehensive."}, {"review_id": "TiL0ayfOuV", "reviewer": "Reviewer_dkG9", "summary": "This paper proposes to improve the fine-grained detail representation capability of a CLIP with a reconstruction loss. The proposed method, un^2CLIP, achieves this by introducing the image generator of unCLIP (a previous work that learns a reconstruction model upon the clip feature). The original image encoder is fine-tuned with image data only, using the reconstruction loss as guidance.  The authors conducted evaluation on MMVP-VLM, open-vocabulary segmentation and MLLMs. The evaluations and ablation studies justified the design of the proposed method.", "questions": "* The authors freeze the image generator and only fine-tune the image encoder, so as to reduce language-shift when training the model with only image data. How about jointly tuning the encoder and generator with text-image data (since the training was conducted on CC-\n3M with image-text pairs)? I see the ablation in Table 4, but it seems the ablation was still conducted with image data only?\n\n* This work claims the freeze-generator-finetune-encoder training method reduces the language-shift problem, making the embedding space of the finetuned model still lie in the original space. How is the performance of the the fine-tuned model on zero-shot recognition/text-image retrieval? More evaluation on this would be helpful to justify this claim. \n\n* Some properties of the un^2CLIP are not evaluated. For example, in the concurrent work GenHancer, the performance of generation and understanding with respect to training iterations are different: better reconstruction does not always lead to better understanding. Does this also apply to the proposed method? More experiments are expected to better showcase the property of the proposed method.\n\n\n\n**Minor:**\n\nIt is not very clear to me how the diffusion loss is used as the indicator of the training difficulty. It would be better if the authors could elaborate on this at somewhere in the paper or the appendix.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n* The problem studied in this work is interesting. The poor fine-grained visual representation capability of CLIP's visual encoder has long been a problem in the MLLM community.\n\n* The proposed method is straightforward, easy to understand and implement. \n\n\n**Weaknesses:**\n\n* The proposed method relies on a pretrained image generator from unCLIP, which limits its generalization to other CLIP encoders since not every CLIP model has a corresponding image generator.\n\n* Some evaluations are missing, such as training with image-text pair, evaluation on CLIP related benchmarks (see questions below). \n\n* Utilizing generation models to improve the representation capability of an image encoder is not a very new paradigm, the novelty of the proposed method is relatively limited.\n\n\nOverall, I think this paper is good and interesting, but the novelty seems to be relatively limited. Adding more experiments can be helpful to justify the claims in the paper and help the readers better understand the property of the proposed method.", "was_revised": true, "final_justification": "Most of my concerns are addressed during the rebuttal period. I thus keep my original rating that the paper is slightly above the accepetance threshold.", "rebuttal": "We thank the reviewer for the detailed review and helpful feedback. We are encouraged that the reviewer finds the paper overall good and interesting, and the method easy to understand and implement.\n\nBelow, we provide detailed point-by-point responses. We are more than willing to provide additional clarification should any questions remain.\n\n> **W1. The proposed method relies on a pretrained image generator from unCLIP, which limits its generalization to other CLIP encoders since not every CLIP model has a corresponding image generator.**\n \nRelying on a pretrained unCLIP image generator is indeed a limitation of our approach and we have summarized it in the last paragraph of Section 5. There are two solutions to address this issue, as detailed in Appendix B and mentioned in Section 4.1:\n\n(1) A trick partial solution using shared generators. We find that some existing CLIP models (e.g., OpenAI CLIP ViT-L-14@224 and ViT-L-14@336) have highly similar embedding spaces thus they can share a same unCLIP image generator.\n\n(2) A general solution to train corresponding unCLIP image generators. Training an unCLIP generator for a specific CLIP encoder is feasible and not overly expensive because there are some pretrained powerful generative models in the community such as Stable Diffusion that unCLIP can built upon. In the codebase used in our experiments, the unCLIP models are finetuned versions of Stable Diffusion. And the unCLIP model for the SigLIP used in the experiment is trained by ourselves, taking about 5 days on 8 A100-40G gpus. Such costs may be tolerable, given that improving the foundation model CLIP is an upstream work that could benefit many vision-centric downstream tasks where CLIP is applied.\n\n---\n> **W3. Utilizing generation models to improve the representation capability of an image encoder is not a very new paradigm.**\n\nWe agree with the reviewer's comment that using generative models to improve CLIP image encoders is not an entirely new paradigm, as we share a high-level similarity with the prior work DIVA in this point. Our key contribution lies in *identifying and addressing the encoder-generator mismatch issue* of the previous work, through *finding* that an existing type of generative models (*unCLIP*) *offers a more seamless and effective* CLIP enhancement framework, and demonstrating better performance on various benchmarks.\n\n---\n> **Q1 (W2). Jointly tuning the encoder and generator with text-image data.**\n\nThank you for this suggestion which helps broaden the scope of our ablation study.\n\n**Implementation.** Following the reviewer's advice, we incorporate an additional image-text loss into our training objective, updating both the image encoder $E$ and the generator $G$. This loss is implemented as the negative cosine similarity between the image and text features from the CLIP encoders, using paired training data from CC3M. Jointly optimizing two losses introduces a hyperparameter $\\lambda$ that balances the previous unCLIP-inversion loss and the image-text loss. We tried two values for $\\lambda$ (1 and 0.1) during this tight rebuttal period.\n\n**Pre-analysis.** *We note that the added image-text objective may re-introduce the issue that this paper aims to address*: captions often reflect coarser-grained semantics than images themselves, making the finetuned model capture less visual details.\n\n**Results.** Results are reported in the last two rows of the table below. We observe that:\n- Simply adding the two losses with equal weight (#4) causes a performance drop compared to not introducing the image-text loss (#3), suggesting that the introduced image-text loss negatively impacts the model’s ability to capture visual details, as hypothesized.\n- Using a smaller weight for the image-text loss (#5) yields a better result, indicating a more balanced trade-off between visual-detail capturing and image-text-alignment during optimization.\n\nHowever, #5 still falls short of #2. These results demonstrate that our default strategy, leveraging the image-text aligned space of unCLIP and updating only the encoder, remains *simple* (no other loss terms or tuning trade-offs), *effective* (current the best performance), and *efficiency* (fewer trainable parameters).\n\n|#|Method|un$^2$CLIP Training Loss|MMVP-VLM|\n|-|:-|-|:-:|\n|1|Original CLIP|N/A|19.3|\n|2|un$^2$CLIP default (training $E$ only)|unCLIP-inversion loss|**32.6**|\n|3|Training $E$ and $G$|unCLIP-inversion loss|27.4|\n|4|Training $E$ and $G$|unCLIP-inversion loss + 1 × image-text loss|25.9|\n|5|Training $E$ and $G$|unCLIP-inversion loss + 0.1 × image-text loss|28.9|\n\n---\n> **Q2 (W2). How is the performance on zero-shot recognition/text-image retrieval? Evaluation on this would be helpful to justify the image-text-alignment claim.**\n\nThe zero-shot classification and text-image retrieval results are in Table 8 in Appendix C, where we observe notable performance drop on most classification datasets, and maintained text-image retrieval performance compared to the original CLIP model.\n\nHowever, we argue that **the drop in classification accuracy does not necessarily indicate a degradation in image-text alignment**. Because even on relatively *\"easy\" classification dataset* like CIFAR-10 that contains only 10 coarse classes, the performance *drop is evident*. In contrast, on *various segmentation benchmarks that contain more classes* (e.g. 150-class ADE, 60-class Context) which also rely on the text encoder for prediction, our finetuned model *consistently outperforms* the original CLIP.\n\nThis discrepancy suggests that **the classification drop stems from task differences, not broken alignment**. As discussed in our responses to Reviewer j8dh's last comment and Reviewer dYip's comment 2, classification tasks tend to favor models that focus on foreground semantics. And \"unfortunately\" our finetuned model corrects many mispredictions where CLIP mistakenly classifies background regions as salient foreground objects (see Fig. 2). These corrections improve visual detail understanding but degrade performance on the conventional classification task. \n\nImportantly, if the image-text alignment is truly broken, the finetuned model would also fail on the segmentation tasks, which it does not. This supports the claim that the alignment remains intact. We will move the classification/retrieval experiments and analysis from the appendix to the maintext in the revised paper to highlight this point.\n\n---\n> **Q3. Does better reconstruction lead to better understanding?**\n\nWe also noticed this question and it is studied in Paragraph 3 of Section 4.5 in our paper, where we draw a **different conclusion** - better reconstruction in our *default*\\* framework always lead to better understanding (recognition) - to the concurrent work GenHancer. \n\nThe key distinction causing the different conclusion lies in *how reconstruction is measured*. In GenHaner paper's Fig. 1, \"reconstruction\" is treated as \"generation\", and evaluated using the CLIP score which is often used to evaluate text faithfulness of generated images of text-to-image models. However, this metric is arguably unsuitable for evaluating \"reconstruction\" in the image-only encoding-decoding frameworks. In contrast, we use a more task-relevant metric - the estimation of Eq. (4) to measure reconstruction (the introduced diffusion loss in Section 4.5, further clarified in our response to the next question). Since Eq. (4) directly connects to the goal of our problem, it is reasonable that better reconstruction (lower loss) leads to better recognition. These are summarized in the last paragraph of Appendix A.\n\nP.S. For a more figurative verification of our conclusion, we can see the visualization result in Fig. 4 (Page 9) of our paper, where the middle columns of images (using un$^2$CLIP finetuned CLIP encoder) achieve better reconstructions of the input images, compared to the original CLIP encoder. And un$^2$CLIP also achieves better quantitative results shown in Table 1 or 4.\n\n\\* \"default\" here refers to not using other training configurations such as also updating the generator introduced in our ablation studies. This point is also clarified in Footnote 2 of our paper. \n\n---\n> **Minor. How the diffusion loss is used as the indicator of the training difficulty.**\n\nWe would like to clarify that the diffusion loss defined in Section 4.5 is *not used to measure training difficulty*. Instead, it is *designed to measure how well the image is reconstructed through the encoder-generator pipeline*. Specifically, we use it as an indicator of the encoder's vision detail information extraction capability, measured **before and after** the encoder's finetuning.\n\nFormally, this metric is intended to compute the value of Eq. (4), i.e., how much information in image $\\mathbf{x}$ is preserved after encoder $E$'s extraction. In practice, we estimate it using the diffusion loss introduced in Eq. (2), which is a lower bound of Eq. (4). For calculating the expectation term $\\mathbb{E}_{\\mathbf{x}, \\mathbf{\\epsilon}, t}$ in Eq. (2), random sampling noises $\\epsilon$ and timesteps $t$ is needed ($\\mathbf{x}$ are test set images) in practice. Since random sampling involves result **comparableness**, to make the two results (before and after finetuning) comparable, we pre-sample and save two sets of random noises and timesteps, and re-use the same sets for both of the two evaluations. \n\nThe pre-saving strategy makes the metric analogous to a \"validation loss\": while it is computed using the same loss function as training (Eq. (2)), it uses fixed randomness and test images. In contrast, the training loss cannot be used as the indicator, as the sampled $\\epsilon$ and $t$ vary during different training iterations, making the results not comparable.\n\nSome of the above explanation is presented in Section 4.5. In the revised paper, we will add a section in appendix to give a more detailed introduction about how the metric is calculated and its role."}]

Author rebuttal:

Meta-review: The paper proposes a method that improves the fine-grained recognition abilities of CLIP via an  encoding-decoding training approach.

The paper received four reviews with mixed, mainly borderline ratings: BR (no answer after rebuttal) - A - BA - BA

The reviewers mainly highlighted that the proposed inversion module to enhance the compositional understanding capabilities of CLIP-like models is an interesting and intuitive approach, and that the proposed algorithm should be easy to integrate into CLIP architectures.
Most weaknesses raised by reviewers have been addressed in the rebuttal. 

The AC follows the majority of the reviewer voting and recommends accepting the paper.  
The AC would encourage the authors to integrate the findings of the rebuttal in the CR version of the paper.
Decision: Accept (poster)