VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions
Authors: Ziteng Wang
Siqi Yang
Limeng Qiao
Lin Ma
Venue: NeurIPS 2025 poster ()
Keywords: CLIP
Fine-Grained
Multimodal-Model
Primary area: deep_learning

Abstract: Despite the success of Vision-Language Models (VLMs) like CLIP in aligning vision and language, their proficiency in detailed, fine-grained visual comprehension remains a key challenge. We present CLIP-IN, a novel framework that bolsters CLIP's fine-grained perception through two core innovations. Firstly, we leverage instruction-editing datasets, originally designed for image manipulation, as a unique source of hard negative image-text pairs. Coupled with a symmetric hard negative contrastive loss, this enables the model to effectively distinguish subtle visual-semantic differences. Secondly, CLIP-IN incorporates long descriptive captions, utilizing rotary positional encodings to capture rich semantic context often missed by standard CLIP. Our experiments demonstrate that CLIP-IN achieves substantial gains on the MMVP benchmark and various fine-grained visual recognition tasks, without compromising robust zero-shot performance on broader classification and retrieval tasks. Critically, integrating CLIP-IN's visual representations into Multimodal Large Language Models significantly reduces visual hallucinations and enhances reasoning abilities. This work underscores the considerable potential of synergizing targeted, instruction-based contrastive learning with comprehensive descriptive information to elevate the fine-grained understanding of VLMs.

VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions
Ziteng Wang*
Siqi Yang"!
The Chinese University of Hong Kong, Shenzhen tommnnyuangzt0gmaill com
Meituan
yangdug aeannuuq.net... net
Limeng Qiao
Lin Ma
Meituan
Meituan
qiaolm@pku edu.
forest www.amgmail....mm com
Abstract
Despite the success of Vision-Language Models (VLMs) like CLIP in aligning vi- sion and language, their proficiency in detailed, ane-grained visual comprehension remains key challenge. We present CLIP-IN, novel framework that bolsters CLIP's fine-grained perception through two core innovations. Firstly, we lever- age instruction-editing datasets, originally designed for image manipulation, unique source hard negative image-text pairs. Coupled with a symmetric hard negative contrastive loss, this enables the model to effectively distinguish subtle visual-semantic differences. Secondly, CLIP-IN incorporates long descriptive cap- tions, utilizing rotary positional encodings to capture rich semantic context often missed by standard CLIP. Our experiments demonstrate that CLIP-IN achieves sub- stantial gains the MMVP benchmark and various fine-grained visual recognition tasks, without compromising robust zero-shot performance on broader classifica- tion and retrieval tasks. Critically, integrating CLIP-IN's visual representations into Multimodal Large Language Models significantly reduces visual hallucinations and enhances reasoning abilities. This work underscores the considerable potential of synergizing targeted, instruction-bad based contrastive learning with comprehensive descriptive information elevate the fine- -reigga understanding of VLMs. Project is available here!
Introduction
The Contrastive Language-Image Pre-training (CLIP) model [30] has revolutionized vision-language representation learning by aligning visual and textual concepts within unified embedding space. Trained on extensive web-scraped image-text pairs (34). CLIP exhibits remarkable zero-shot general- ization across diverse tasks, serving foundational model for applications like image classification, cross-modal retrieval, object detection and segmention, and multimodal large language models
(MLLMs) BI 20 3]
Despite success in high-level semantic understanding, CLIP demonstrates limitations in capturing fine-grained visual details such color, quantity, and spatial relationships [16] [59 55 [5] 23 [2] 49] For instance, distinguishing between a "black cat with a yellow bow tie" and one with a "red bow tie"
"Equal contribution. Work done when Ziteng Wang worked as an intern with Meituan. 'Coresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Instruct: Change the bow !"
(b) Hard Negative Image-Text Text Pairs
tie color to red
Leup
INeecLP t (75, r,
Leggurr
LNegCuIP
1.
M,7) (F, T.T) (T) P,F) (T,, (,7)
(1. T)
CLIP
NegCLIP
(P, r3, r (F, r*)
TripletCUPP
L'In
Lix
Source Caption yos black cat
Target Caption T black cat wearing red bow tie.
INegcLIP
INegcuP
Exegeur
wearing
yellow bow tle.
T (1, T,TT))
(T', P,P) (T', R,P)
CUP-IN
(a) Instruction Editing Data
(c) Hard Negative Training
Figure I: Instruction Editing Data as Hard Negatives. (a) We illustrate how instruction editing data provides challenging negative examples for CLIP. Given a source image and caption, an editing instruction leads a target image and caption with subtle, fine-graine -reegraiied changes. The (source (source_image. sarget_cattion and (target sarget_image, source source_caption) pairs serve as hard negatives in (b), requiring the model to distinguish these nuanced visual-semantic differences. (c) We propose a symmetric hard negative contrastive loss to explicitly train the model to discern these subtle visual-semantic differences from both image-to-text and text-to-image perspectives.
can be challenging for CLIP when the overall scene context is similar (Figure LT. These limitations are inherited by MLLMs built upon CLIP's visual encoders, impacting their perceptual accuracy and contributing to issues like object hallucination [39, 46]. Benchmarks like Multimodal Visual Patterns (MMVP) 39 specifically highlight these "CLIP-blind spots" by presenting perceptually distinct image pairs that CLIP often confuses.
These shortcomings primarily arise from two aspects of CLIP's pre-training. First, the global image- text alignment objective can overlook subtle visual details and intricate inter-object relationships [16] Second, CLIP's text encoder, typically employing absolute positional embeddings with fixed 77-token limit, restricts the effective utilization of longer, more descriptive captions that could
59
23 50
provide richer supervision for fine-grained details [ 55 24 58).
Existing approaches to enhance fine- eneegrained understanding in CLIP have explored region-level contrastive learning [16] 59 50), which often requires complex region proposals additional
supervision. Self-distillation methods [4 23 22 aim to improve local-to- tlogbbaall consistency
without explicit regional annotations but may suffer from weakened semantic grounding be limited by the teacher model's capabilities. Hard negative mining strategies [53] 56 27 focus on challenging the model with difficult negative examples, often generated by perturbing captions using text-to- image synthesis, which may lack the necessary visual similarity or fine-grained control. Long-caption extend CLIP's text processing capacity, but simply increasing the token limit does not guarantee improved fine-grained alignment or prevent performance degradation on short text inputs. This motivates the central question: How can we effectively and scalably enrich vision- language models like CLIP with robust fine-grained visual understanding by harnessing data sources that offer explicit, targeted supervision for subtle lesual-semantic distinctions, while simultaneously leveraging the contextual richness of descriptive language?
methods (T 55 24)
In this work, introduce CLIP-IN (CLIP with INstruction edit data and INformative long data), novel framework that addresses this challenge by synergistically integrating instruction editing data and long descriptive captions. Our core innovation lies in repurposing instruction editing datasets, originally designed for image manipulation, as a valuable source of hard negative image-text pairs for contrastive learning. Datasets like UltraEdit [57] provide tuples of (source image, source carptin, editing instruction, target_ image, target_ capptonn,,, inherently offering hard negative examples with fine-grained differences in objects, attributes, and spatial relationships. This contrasts with synthetic hard negatives generated by eext-to-iage models, as seen in TripletCLIP (27). which often lack precise control and visual similarity to the source image. We propose a symmetric hard negative contrastive loss to explicitly train the model to discern these subtle visual-semantic differences from both image-to-text and text-to-image perspectives, implicitly learning fine-grained visual distinctions and their linguistic descriptions, unlike explicit regional contrastive learning (50] 15
Additionally, We propose that long caption data is complementary the instruction editing data. We leverage long descriptive captions provide broader contextual richness. To enable existing CLIP

---

Stage 1: Text Encoder Distillation
Stage 2: Contrastive Learning
Long Caption:
Teacher Text Encoder with absolute PE
Training Set
stack smooth, polished stores metic meticcoously balanced on top of each other, symbolizing balance and tranquiiiyy. The stones, varying in vice and cofor, create a hermonious and meditative focal point The pebbly shore
senena beach scane
(r", r, r
(F. r. r
(P, Tshort plong )
gt
(T', P.F)
(T', F)
mult colored diepth the sea stretches to moet horizon, where sun setting meeiing a warm. go goiden glow acrass the dky and reflecting on the water's s surfhoo.
scatsered
Latstiu
Long Short Caption Triplet
Hard Negative Triplet
ariiing foreground background,
gs
Vision Encoder
Student Text Encoder with RoPE
MLLM
shortt +
Liong
LuN
Short Caption: Balance stones on the beach. Peece of mind.
Equilibnium life Ca stock footage
Figure 2: CLIP-IN Framework Overview, Stagel. We adapt the CLIP text encoder to process long captions using Rotary Positional Embeddings (RoPE) via knowledge distillation. Stage2. Our proposed framework, CLIP-IN, leverages two complementary data sources: instruction editing data and long descriptive captions. Instruction editing data excels teaching the "where" and "how" of subtle visual details, while long captions provide the broader "what" and "why" of the scene, capturing complex relationships and contextual information.
models to effectively process long text, we adapt the text encoder by incorporating Rotary Positional Embeddings (RoPE) (36] through knowledge distillation from the pretrained clip text encoder model extended context.
By training on both instruction editing data and long captions, CLIP-IN aims to learn representations that are simultaneously semantically rich and visually grounded fine-graine -rainer level. Instruction editing data excels at teaching the "where" and "how" of subtle visual details, while long captions provide the broader "what" and "why" of the scene, capturing complex relationships and contextual information. This synergy enables CLIP-IN to outperform models trained on either data source alone.
Our primary contributions are: (1) A novel approach to utilize instruction-cditing datasets rich source of hard negative image-text pairs for enhancing CLIP's fine- ane-ggriined visual perception, extending the utility of these datasets beyond image generation tasks. (2) A synergistic training framework for CLIP that combines instruction-barn based image editing data and long descriptive captions to significantly improve its fine-grained visual-linguistic understanding by leveraging explicit and implicit supervision signals. (3) A methodology to adapt CLIP's text encoder for processing long captions by integrating Rotary Positional Embeddings (RoPE) via knowledge distillation, overcoming the inherent context length limitations. (4) Extensive experiments demonstrating consistent improve- ments on zero-shot visual benchmarks, fine-grained visual recognition tasks, and evaluations on MMVP and MLLM benchmarks.
2 Related Works
Contrastive Language-Image Pre-training. Contrastive Language-Image Pre-training (CLIP) 30 has demonstrated remarkable success in learning aligned visual and textual representations by con- trasting positive image-text pairs against negative ones on a massive scale. Subsequent works like EVACLIP (37,, MetaCLIP [51), SigLIP [54], and SigLIP2 [40] have further improved the perfor- mance and robustness of CLIP models through architectural modifications and training strategies.
Fine-grained Understanding in CLIP. Despite its effectiveness, standard CLIP often struggles with fine- grained visual understanding, including discerning subtle attributes, complex inter-object relationships, and precise object counts. This limitation stems, in part, from the typically concise nature of image captions used in pre-training, which may lack the detailed descriptions necessary for capturing such granular distinctions. To address this, several approaches have been proposed. GLIP 16), RegionCI.IP (59], and FG-CLIP [50] leverage grounding data to explicitly align image regions with corresponding textual phrases, thereby enhancing the model's 's ability to understand local-
3

---

ized visual details. Complementary to this, methods like CLIPSefl 47 SILC (2), MaskEmb A and TIPS [22] focus on improving the consistency between global and local representations through self- supervised learning objectives, such as local-to-ge to-global correspondence learning via self-distillation and masked patch embeddings. Furthermore, DIVA 44 explores the use of generative feedback from text-to-image diffusion models to refine CLIP representations using only image data. However, these methods are primarily designed and evaluated on datasets with short captions, potentially limiting their effectivvness in scenarios requiring the processing of more detailed textual descriptions.
Hard Negative Mining and Generation for CLIP. Hard negatives, semantically distinct samples close in the embedding space, are crucial for learning discriminative features in contrastive learning Prior works explored identifying hard negatives within datasets or generating synthetic negative
captions [53
56
52
35
7. More recently, generating synthetic hard negative images has gained For example, Peng et al. (28 synthesized images by rearranging segmented
attention 28
33
45
objects, potentially lacking real-world complexity and scalability. Sahin ef al. [33 used inpainting for limited synthetic image generation, while Wang et al. [45] manipulated discrete visual tokens without generating corresponding negative texts. Triple(CLIP (27], the most related work, generates hard negative image-text pairs using LLMs and text-to-image models for triplet loss. However, its generated images often lack control and deviate significantly, primarily enabling image-to-text hard negative learning. Our approach uniquely leverages large-scale instruction editing datasets like UltraEdit [57]. which provide real images and localized visual changes, mitigating domain biases of purely generated data and yielding challenging hard negatives. This explicit grounding allows for symmetric hard negative losses addressing both image and text mismatches.
Extending CLIP to Long Captions. Standard CLIP models are typically limited in their text input length (e.g.. 77 tokens), which can hinder their ability to process detailed and descriptive long captions. DCI [41] highlighted this limitation and, along with DOCCI [25), introduced datasets with dense, long captions for benchmarking. DreamLIP (58) proposed leveraging MLLM -Leeerggeerated long captions and dynamically sampling sub-captions to create multiple positive pairs, although it does not directly process the full long captions. Recent efforts (55 15 50 have extended CLIP's text capacity to 248 tokens by employing positional embedding interpolation and fine-tuning existing CLIP models. LoTLIP [48) incorporates long captions during the pre-tirining stage and introduces corner tokens to aggregate diverse textual information. TULIP (24) adopts a distillation approach, transferring the knowledge of a CLIP text encoder enhanced with relative positional embeddings (RoPE (36).. While these methods demonstrate progress in handling long captions, they often rely on large-scale datasets and 1.6 billion in FG-CLIP (50) Direct
of long caption-image image pairs (e.g., millions in (58 55 15 24
fine-tuning on such data can sometimes lead to performance degradation on tasks involving short text inputs, such as image classification. In this work, we aim to exploit the synergistic benefits of instraction-editing hard image-text pairs and long caption data to achieve improved fine-grained understanding without compromising performance on standard CLIP benchmarks.
3 Methodology
3.1
Preliminaries
Language-Image Pre-training (CLIP). Given batch of N image-text pairs (X, {(x,, y:) y CLIP aims to align semantically similar image-text pairs in a shared embedding space while separating dissimilar ones. The architecture consists of image encoder f(-) and text encoder g(-), both mapping their respective inputs to this common space. CLIP employs the InfoNCE loss (26) to achieve this alignment. For an image x; the anchor, the image-to-text contrastive loss, Lizr, is defined as:
Contrastive
1 Σ log exp(([(z) N
= ()
1=1 Σx exp((( (f)))))))))))))) )(
where (.-)) denotes the cosine similarity and is a temperature parameter. The overall CLIP loss, LCUP, is a symmetric combination of the image-to-text and text-to-image contrastive losses:
LAP = La. (x + (),
Overview of CLIP-IN. As depicted in Figure our primary objective to significantly enhance CLIP's capacity for discerning fine-grained visual details. To this end, we introduce CLIP-IN,

---

two-stage training framework that leverages two complementary data modalities, First, Instruction Editing Data provides explicit supervision for learning localized visual transformations guided by precise textual instructions, thereby improving the model's sensitivity to subtle visual attributes and intricate relationships. Second, Long Captions offer rich semantic context and detailed descriptions of complex visual scenes, enabling a more comprehensive understanding beyond localized features. In the first stage of our framework, we adapt the text encoder to effectively process long captions by replacing absolute positional embeddings with RoPE and performing knowledge distillation. Subsequently, the second stage involves a joint contrastive learning process utilizing both instruction editing data and long captions to achieve the desired enhancement in fine- ane-grained visual perception.
3.2 Instruction Editing Data as Novel Source for Hard Negative Training
Instruction editing data offers a unique form of supervision by explicitly linking textual instructions to precise visual modifications in real images, as exemplified in Figure I This structure is particularly valuable for generating challenging negative samples that target fine- mne-grained visual distinctions.
Formulating Hard Negatives from Instruction Editing Data. Each instance in an instruction editing dataset is a tuple (1',1', T T ) where I is the source image, I' is the target (edited) image, 7 is the editing instruction, and T and T are the corresponding source and target captions. From this data, we construct positive pairs (Ie T*) and (7', T"), and critically, negative pairs as (I*TT) TT) and (T*TT)) These hard negatives are challenging because the image pairs (I", (I",1% I') and text pairs (T TTt) are minimally but semantically distinct due to the fine- graingg edits described by T This structured approach to generating hard negatives from real image edits provides strong learning signal for enhancing fine- greeggraindd visual understanding. We utilize the recent UltraEdit dataset [57], comprising approximately 4 million high-qualiny instruction based editing samples across diverse editing types and instructions, as our primary source of instruction editing data.
define hard
Symmetric Hard Negative Contrastive Loss. Inspired by prior work on hard negative mining in contrastive learning [53] (56,, we formulate a symmetric hard negative contrastive loss. Given a hard triplet (X; where X' is set of anchor images, Y is the of corresponding positive captions, and y" is the set of hard negative captions, the hard negative contrastive loss for image-to-text alignment is:
N log
exp((((a).
(2:3)))) =
N 1=1
E (())(((()) + Em
From our instruction editing data, derive two image-to-text hard triplets: (X* "",,"" and (X', y", )²), where X === (1"}, y (T x' {T'), and y [T*). The image-to-text hard negative loss is then defined as:
C = (N y², "²))" + (x' ,,*)""
Unlike methods like TriplerCI.IP (27] which struggled with text-to-image hard negative loss due to challenges in generating controlled negative images, instruction editing data provides dell-defined hard negatives in both modalities. Therefore, we propose a symmetric hard negative loss by also considering the text-to-image direction:
CTN LNegCL (V", xx)) + ENogeL()* (²))
(5)
The final symmetric hard negative loss is the sum of both directional components:
CHN CAN L
This symmetric loss encourages the model to learn fine- fine-grained visual-semantic distinctions from both image and text perspectives, leveraging the inherent structure of instruction editing data to create semantically meaningful and challenging negative examples. The illustration of the above hard negative training losses can be seen in Figure (b).
3.3 Synergizing Instructional Data with Semantically Rich Long Captions
Generating Semantically Rich Long Captions. To provide comprehensive semantic context that complements the fine-grained details learned from instruction editing data, generate long,
5

---

descriptive captions for the images in our training set. We employ ttate-of-the the-hhe-rt vision- -aagun-a model, Intern VL 3 to generate these detailed captions. which average around 300 tokens in length.
Integrating Rotary Positional Encodings (RoPE). Standard CLIP models with fixed-length absolute positional embeddings are limited in their ability to process long text sequences. To address this, we replace the absolute positional embeddings in the text encoder with Rotary Positional Encodings (RoPE) (36.. RoPE encodes absolute position through rotation matrices, enabling the self-attention mechanism to inherently consider relative positional information. This modification allows for greater flexibility in handling variable sequence lengths and improves extrapolation to longer inputs, which is crucial for processing our generated long captions.
Knowledge Distillation for Relative Position Encoding. To adapt the CLIP text encoder to utilize RoPE and effectively process both short and long captions while preserving its pre-trained knowledge, employ knowledge distillation strategy. This approach avoids the need for full retraining from scratch. Following insights from CLIP-KD [9] and TULIP (24), we initialize a student text encoder gs(+) with the RoPE configuration and distill knowledge from frozen teacher text encoder gr(() (the original CLIP text encoder). During this distillation phase, we exclusively use the generated long captions. For long caption ylong exceeding the teacher's context length cT, we truncate it yrmene of length cr. The distillation loss, Lussil, aims to align the embeddings produced by the student and teacher models for the truncated caption:
(0r (Vrrnn)) ) 9s (Yuune)
Ldistill
(gr II - [gs (Yeune))
where (+-)) is the dot product and ||- Il denotes the L2 norm. After distillation, the student text encoder gs(()) retains the capabilities of the teacher model within its original context length while gaining the ability to process longer sequences due to the integration of RoPE.
3.4 CLIP-IN Training Pipeline
Our CLIP-IN IN framework integrates the aforementioned components into cohesive two-stage training pipeline designed to enhance fine-grained visual perception while maintaining general vision-language alignment capabilities. In the first stage, we adapt the text encoder by replacing absolute positional embeddings with RoPE and performing knowledge distillation using long captions, as described by B 7 This step ensures the text encoder can handle variable- sequences while retaining pre- tre-tra knowledge. In the second stage, we jointly train the entire model using both instruction editing data and long caption data. We utilize the RoPE-enabled text encoder from the first stage. For the instruction editing data, we apply the symmetric hard negative contrastive loss LIN (Eq. ()) to specifically target fine-grained visual understanding. For the long caption data, we employ the standard contrastive loss LCL (Eq. [T on the long captions paired with their corresponding images. Additionally, we also include the standard contrastive loss on the original short alt-text captions associated with the images to maintain performance on standard CLIP benchmarks. The overall training objective L is weighted sum of these losses:
>stan CcL Voton) + AcongCaL (X', Mongg) + AuN
(8)
where Vshoot represents the original short alt-text captions, Viong represents the generated long captions, and Aehoons Along and λHN are hyperparameters that balance the contribution of each loss term. The overview of this training pipeline is illustrated in Figure ?
Experiments
4.1
Implementation Details
For hard negative training, we utilize the UltraEdit dataset [57], consisting of 4 million instruction- based image editing samples. For the long caption data, we employ InternVL.2 []] to generate detailed captions for approximately 18 million image-text pairs randomly sampled from a diverse set of datasets including CC3M, CC12M, COYO, and LAION [34]. All models are trained on 16 NVIDIA 1100 GPUs with 80GB memory. We use an Adam W optimizer with a learning rate of le-4 and a weight decay of 0.05. The loss weights are set to Xahen = 1.0, Atong = 0.1, and AHN = 0.1. We evaluate our framework by fine-tuning pre-trained state-of-the-an CLIP models. The global batch size for training with the ViT-L backbone is 16,384, and it is 4096 for the SigLIP2 models.

---

4.2 Zero-Shot Classification and Retrieval
Datasets, We assess the zero-shot image classification performance on the widely used ImageNet- IK dataset [5]. For zero-shot short-text retrieval, we the Flickr30K dataset [29] and the MS-COCO dataset [8]. To evaluate zero-shot long-text retrieval capabilities, we utilize the DCI benchmark
and IK subset of the ShareGPT4V dataset 2], following previous works [50] 55
Results. The results presented in Table demonstrate the effectiveness of our CLIP-IN framework across various zero-shot evaluation tasks. Notably, our method consistently achieves competitive and often superior performance compared to strong baselines, including SigLIP2. For the SigLIP2 backbone, our approach shows clear improvements. With the ViT-SO/14 backbone at 224 resolution, our method achieves a slightly higher ImageNet-IK Top-1 accuracy (83.4%) compared to SigLIP2 (83.2%). More significantly, accuracy (78.8% vs. 76.4%) and average long caption retrieval accuracy (67.3% vs. 62.0%). A similar trend is observed with the larger ViT-SO/16 backbone at 384 resolution. While our method exhibits strong performance in long caption retrieval, it trails slightly behind FG-CLIP, which achieves an average of 81.8% compared to our 76.4% (with ViT-L/14 at 336 resolution). This discrepancy be primarily attributed to the substantially larger long caption training dataset utilized by FG-CLIP (1 billion pairs) contrast
method outperforms SigLIP2 in average short caption retrieval
18 million pairs. However, our ImageNet- IK classification accuracy of 77.0% significantly exceeds that of FG-CLIP (76.1 1%), indicating that our approach achieves a better balance between fine-grained retrieval and general image understanding despite using considerably less long caption data.
Table 1:
Evaluation of zero-shot performance on various image benchmarks.
CLS
Short Caption Retrieval
Long Caption Retrieval ShareGPT4V
Method
Backbone
Res
IN-IK
Flickr
COCO
DCI
Top-1
Avg
1-T
T-1
I-T
T-+1
Avg
I-T
T-I
1-T
T-xl
OpenAl CLIP
30
VIT-L/14
224
75.5
60.7
85.2
64.9
56.3
36.5
64.3
84.2
83.7
45.3
44.0
Ours
VIT-L/14
224
76.3
72.9
92.9
79.4
68.9
50.5
76.8
92.3
91.9
61.7
62.0
OpenAl CLIP
VIT-L/14
76.6
62.5
87.4
67.3
58.0
37.1
61.0
86.5
83.6
37.2
36.4
EVA-CLIP 37
VIT-L/14
336
80.4
69.8
89.2
77.9
64.2
47.9
69.0
91.5
89.4
47.2
47.8
Long-CI.IP
VIT-1/144
336
73.5
68.8
90.0
76.2
62.8
46.3
72.0
95.8
95.6
44,2
52.5
FineCLIP 15
VIT-L/14
336
60.8
60.6
73.4
82.7
40.1
46.2
FG-CLIP 50
VIT-L/14
336
76.1
73.8
93.7
81.5
68.9
50.9
81.8
97.4
96.8
66.7
66.1
Ours
VIT-L/14
336
77.0
73.1
93.8
79.3
68.2
51.1
76.4
93.5
91.6
58.4
61.9
DFN-H
ViT-H/14
74.8
92.8
80.1
72.3
53.9
79.8
92.5
90.3
68.7
67.5
Ours
VIT-H/14
224
83.4
75.6
93.0
80.8
73.6
54.8
81.5
93.8
92.4
70.5
69.1
DFN-H 8
VIT-H/14
378
84.4
75.9
94.0
82.0
71.9
55.6
82.3
93.9
92.5
71.6
71.0
Ours
ViT-H/14
378
84.1
76.8
94.6
82.2
74.0
56.4
83.5
95.4
93.9
72.7
71.9
SigLIP2
40
VIT-SO/14
76.4
94.6
84.3
71.5
55.1
62.0
76.4
76.2
45.4
50.0
Ours
ViT-S0/14
224
83.4
78.8
94.9
85.1
76.2
58.9
67.3
81.5
80.7
52.5
54.4
SigLIP2 40
ViT-SO/16
84.1
77.1
95.9
85.3
71.2
56.0
59.1
70.7
72.8
43.4
49.6
Ours
VIT-SO/16
384
83.7
79.6
96.4
85.6
76.5
59.8
64.0
77.7
76.4
50.0
51.7
4.3 Fine-Grained Visual Perception Evaluation
Datasets. We first evaluate fine-grained visual perception on the Multimodal Visual Patterns (MMVP) benchmark, which is specifically designed to probe the weaknesses of vision-language models in perceiving subtle visual differences. To evaluate the ability to understand the composition of images,
we further evaluate the Winnoground [38] SugarCrepe SPEC (28 and ARO [53]
Results. As detailed in Table 2 our CLIP-IN model demonstrates substantial improvement in fine-grained visual perception on the MMVP benchmark. With a ViT-L/14 backbone, our approach elevates the average accuracy from 18.5% to 30.4% (+11.9pp), with particularly striking gains in recognizing object State/Condition (+33.3pp) and in Feature Detection (+20.0pp). Furthermore, our SigI.IP2-basee variant not only surpasses the original's average accuracy (36.3% vs. 35.6%)
but also shows marked improvements in complex categories such as Positional Context (+13.3pp)
7

---

Table 2: Performance of CLIP based models on various visual patterns of MMVP-VLM benchmark.
Symbols for visual patterns as ([9)) are inherited: Orientation and Direction, Q: Presence of Specific Features, C State and Condition, f:: Quantity and Count, f: Positional and Relational Context, P Color and Appearance, c Structural and Physical Characteristics, A: Texts, Viewpoint and Perspective.
Method
Backbone
Res
Q
11
A
Avg
OpenAl CLIP
30
ViT-L/14
13.3
20.0
13.3
53.3
20.0
13.3
18.5
Ours
ViT-L/14
224
6.7
33.3
53.3
20.0
13.3
60.0
33.3
26.7
26.7
30.4
OpenAI CLIP
VIT-L/14
20.0
40.0
20.0
20.0
33.3
40.0
20.0
DIVA 44
ViT-L/14
336
26.7
20.0
33.3
13.3
13.3
46.7
26.7
6.7
40.0
25.2
Ours
ViT-L/14
336
13.3
13.3
46.7
13.3
13.3
53.3
33.3
20.0
28.3
26.1
DEN g
VIT-H/14
224
20.0
26.7
73.3
26.7
26.7
66.7
46.7
20.0
53.3
39.9
Ours
VIT-H/14
224
20.0
26.7
73.3
26.7
33.3
66.7
46.7
26.7
53.3
41.5
DEN
g
VIT-H/14
378
13.3
20.0
53.3
33.3
26.7
66.7
40.0
20.0
40.0
34.8
Ours
ViT-H/14
378
13.3
20.0
60.0
33.3
26.7
66.7
40.0
20.0
46.7
36.3
SigLIP2
40
ViT-S0/14
224
13.3
20.0
60.0
26.7
6.7
80.0
53.3
20.0
40.0
35.6
Ours
VIT-SO/14
224
13.3
13.3
60.0
26.7
20.0
80.0
46.7
13.3
53.3
36.3
SigLIP2
40
VIT-SO/16
384
13.3
20.0
46.7
40.0
20.0
73.3
53.3
6.7
46.7
35.6
Ours
VIT-S0/16
384
13.3
20.0
60.0
26.7
66.7
40.0
46.7
36.3
Table 3: Evaluation on compositional reasoning benchmarks.
Method
Backbone
Res
ARO
MMVP
Winoground
SugarCrepe
SPEC
Avg
relation attribute
Avg
image
group
Avg
T:sl
1>T
OpenAI CLIP
30
VIT-L/14
224
58.9
59.3
58.5
18.5
15.9
28.3
10.5
8.8
75.6
32.3
33.2
31.3
Ours
VIT-L/14
224
64.4
64.3
64.4
30.4
17.1
28.0
138
9.5
71.5
36.3
37.6
35.0
OpenAl CLIP
30
VIT-L/14
336
61.0
60.1
61.9
20.0
15.4
28.3
10.5
7,5
74.8
32.1
32.8
31.1
Ours
VIT-L/14
336
60.7
58.1
63,2
26.1
18.1
33,0
11.8
9.5
77.2
35.2
35.1
35.2
SigLIP2 40
VTT S0/14
224
49.7
49.0
50.4
35.6
6.9
9.0
9.3
2.5
49.5
27.3
27.4
27.2
VIT-SO/14
224
49.5
36.3
7.5
50.5
30.5
30.4
SigLIP2
VIT-SO/16
384
47.3
35.6
8.5
2.3
30.9
VIT-S0/16
and Viewpoint (+13.3pp). These results affirm that our training strategy effectively addresses the well-known limitations standard CLIP models in discerning subtle visual patterns.
Building upon this enhanced perceptual acuity, CLIP-IN also exhibits superior compositional reason- ing, consistently outperforming strong baselines across suite of challenging benchmarks, as shown in Table On ARO, which evaluates attribute, relation, and compositional understanding, our model achieves 58.9 avg vs. OpenAl CLIP's 58.4 (+0.5pp) 224 resolution and 60.5 vs. 59.7 (+0.8pp) 336 resolution, with notable gains in relation (64.3 vs. 64.3) and attribute (58.5 vs. 58.4). indicating improved sensitivity subtle visual details. On Winoground, challenging minimal-pair benchmark, CLIP-IN significantly improves text-to-image accuracy from 28.3% to 33.0% (+4.7pp). image-to-text from 10.5% 1.8% (+1.3pp), and group accuracy from 7.5% 9.5% (+2.0pp). highlighting its enhanced capability to ground small linguistic changes in distinct images. On SugarCrepe, designed evaluate mitigative reasoning through hard negatives, our model achieves 79.4% average accuracy vs. CLIP's 3.8% (+5.6pp), reflecting robustness in attribute binding and relational reasoning. Finally, on SPEC, which focuses on precise spatial and compositional understanding, CLIP-IN surpasses CLIP by 34.8% vs. 32.0% (+2.8pp) and Siglip2 by 30.5% vs. 27.5% (+3.0pp), with gains in both T-I (35.1% vs. 32.8%) and I-TT (35.2% vs. 31.1%) directions, confirming stronger cross-modal alignment under synthetic but precise conditions. These consistent improvements across diverse architectures (ViT-I. and s0400m) and tasks validate that our method-coo combinnnn instruction editing and long caption
enhances compositional reasoning in vision-languadl models.

---

4.4 Evaluation on MLLM
We adopt LLaV5-1.5 19 as the baseline framework explore the potential of our proposed visual encoders in MILM.
Datasets. We train our model with the setting in LLLVA--13 and evaluate model performance on various multimodal understanding henchmarks (i.e., MMVP 391 POPE 17). MMEE-Prrcpptoo [TO]
MMBench 21,, MMBBnch-CN 21. LLaV---eenh-in--hhe Wild
20
Results. Table presents the performance of LLaVA-1.5 when equipped with different visual backbones. We compare the performance of our CLIP-IN enhanced visual encoder against the original OpenAl CLIP backbone and DIVA 44 a recent method aimed at improving CLIP's visual representations. The results demonstrate that using our CLIP-IN visual backbone leads to significant improvements across several benchmarks, particularly on MMVP, MME, and both English and Chinese versions of MMBench, indicating enhanced fine-grained perception and overall multimodal understanding capabilities.
Table 4: Performance gains achieved by our enhanced CLIP visual backbone for MLLM. All methods use OpenAI VIT-L/14 at 336x336 resolution pretrained backbone.
POPE
MMBench
Method
VIT
LLM
MMVP
MME
LLaVA Aild
rand
pop
adv
OpenAl CLIP 50
24.7
87.3
86.1
84.2
1510.7
64.3
58.3
65.4
LLaVA- 1.5
19
DIVA 44
Vicuna-78
31.3
87.9
87.0
84.6
1500.6
66.4
60.6
66.3
Ours
28.0
88.5
87.2
85.2
1709.0
72.9
70.3
68.5
Table 5: Comparisons on synthetic data and data scale.
Retrieval Avg
Flickr
COCO
Method
Training Data
IN-IK
T-II
I-+T T-x1 I-T
Ours
TripletData 1.4M
76.3
70.58
77.6 92.0 47.9 64.8
UltraFdit 1.4M
75.8
72.9
78.3
92.9
51.1 69.2
UltraEdit 4M
76.3
72.9
79.4
92.9
50.5 68.9
4.5
Ablation Studies
Data Scale and Source Analysis. To demonstrate the effectiveness of instruction editing data, compare our approach with training using TripletData [27]. a method that generates synthetic hard negative image-text pairs based text-to-image generation model. Since TripletData provides 1.4 million samples, we also train our model using a randomly sampled subset of 1.4 million examples from UltraEdit for fair comparison. Using balanced subset of 1.4 million samples from UltraEdit and TripletData, UltraEdit showed a 2.32% increase in average retrieval accuracy (72.9% vs. 70.58%). On ImageNet-1K. UltraEdit's accuracy was 75.8%. slightly less than TripletData' 76.3%. Expanding UltraEdit to 4 million samples improved ImageNet- 1K accuracy to match TripletData at 76.3%, while retrieval accuracy stayed at 72.9%. This indicates high-quality instruction editing data significantly boosts fine-grained retrieval and improves image classification with more data.
Impact of Instruction Editing Data and Long Caption Data. Pables/Jande reveal the comple- mentary nature of our two data sources. Instruction editing data alone improves retrieval accuracy from 62.5% to 72.0% (+9.5pp), confirming that hard negatives enhance fine-grained discrimination. Long caption data similarly boosts retrieval to 7 71.0% (+8.5pp) and ImageNet- accuracy to 76.8% (+0.2pp). Most importantly, their combination yields the highest retrieval performance (73.1 1)), best classification accuracy (77.0%), and significant MLLM improvements. While each data source alone decreases MMVP performance in LLaVA-1.5 1.5, their combination substantially improves it (+3.3pp), demonstrating how these complementary signals effectively address CLIP's fine-grained perception limitations.
Impact of Rotary Positional Embeddings (RoPE). To isolate the benefit of our RoPE-based text encoder distillation, we compare it against strong baseline that uses absolute positional encoding

---

Table 6: Ablation studies on the contribution of different components in CLIP-IN. We evaluate the impact of instruction editing data and long captions, both individually and in combination, using OpenAl ViT-L/14 at 336x336 resolution as pretrained model.
Retrieval
Flickr
COCO
Method
InstructData LangData
IN- IK
Avg
T-1 [-T
T-1
1-T
OpenAl ViT-L/14 30
76.6
62.5
67.3
87.4
37.1 58.0
Ours
76.6
72.0
78.5
93.0
49.7 66.6
76.8
71.0
78.3
92.6
48.4
64.8
77.0
73.1
79.3
93.8
51.1
68.2
Table Ablation studies on the contribution of different components in CLIP-IN on the MLLM benchmarks.
POPE
MMBench
Method
InstruciData
LongData
MMVP
MME
LLaVA-Wild
rand
pop
adv
LLa
24.7
87.3
86.1 84.2
1510
64.3
58.3
65.4
Ours
22.1
85.4
84.4 83.8
1679
68.4
72.1
67.0
V
20.9
85.6
84.8
82.5
1646
67.1
71.6
65.9
28.0
88.5
87.2
85.2
1709
72.9
70.3
68.5
interpolation, as popularized by LongCLIP [55]. We initialize our second stage of training with LongCLIP, a text encoder pre-trained using positional interpolation and compare it to our proposed RoPE-based model. All other settings are identical. We use the ViT-L/114@2 /2/1//42222 224 model for this ablation. The results in Table show that our RoPF-based approach is a more effective method for extending the text encoder's context length. significantly outperforms the interpolation baseline on general classification (IN-IK: +6.2%) short-text retrieval (+3.5%).
Table 8: Ablation studies the impact of Rotary Positional Embeddings (RoPE).
CLS
Short Caption Retrieval
Long Caption Retrieval
Method
IN-IK
Avg
Flickr
COCO
Avg
ShareGPT4V
DCI
Top-1
Avg
I-T
T-I
I-T
T-11
Avg
I-T
T-II
I-T
T-I
Ours(w LongCLIP)
70.1
69.4
89.1
76.8
64.7
47.1
78.0
95.9
96.1
60.1
59.7
Ours(w RoPE)
76.3
72.9
92.9
79.4
68.9
50.5
76.8
92.3
91.9
61.7
62.0
Conclusions
We introduced CLIP-IN, a novel two-stage framework designed to enhance CLIP's fine-grained visual perception capabilities. Our approach leverages the unique properties of instruction editing data as a rich source of hard negative image-text pairs, enabling the model to learn subtle visual-semantic distinctions. Furthermore, we integrated semantically rich long captions and adapted the CLIP text encoder with RoPE via knowledge distillation to capture comprehensive contextual information. Our extensive experiments across various zero-shot classification and retrieval benchmarks, fine- grained visual perception tasks, and evaluations on multimodal large language models consistently demonstrate the effectiveness of CLIP-IN in significantly improving performance, particularly in discerning fine-grained visual details. Ablation studies further validated the complementary nature of instruction editing data and long captions in achieving these gains. By effectively harnessing these two distinct data sources, CLIP-IN advances the state-of-the-an in vision- language representation learning and offers promising direction for building more perceptually accurate and semantically aware multimodal models. Future work could explore the application of CLIP-IN to other vision- language tasks, investigate more sophisticated methods for generating and utilizing long captions, and extend the framework to incorporate even larger and more diverse instruction editing datasets.
10

---

References
[1] Mothilal Asokan, Kebin Wu, and Fatima Albreiki. Finelip: Extending clip's reach via fine-grained aligmment with longer text inputs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[2] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang. Feng Zhao, and Dahua Lin. Sharegpidv: Improving large multi-modal models with hetter captions. In European Conference on Computer Vision, pages 370-387. Springer, 2024.
[3] Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- -inguiiin tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185 24198, 2024.
[4] lan Connick Covert, Tony Sun, James Zou, and Tatsunori Hashimoto. Locality alignment improves vision-language models. In International Conference on Learning Representations (ICLR), 2025.
[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pages 248-255, leee, 2009.
[6] Sivan Doveh, Assaf Arbelle, Sivan Harary, Roci Herzig, Donghyun Kim, Paola Rescante--onillee Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, et al. Dense and aligned captions (dac) promote compositional reasoning in vi models. Advances in Neural Information Processing Systems, 36:76137- 76150, 2023.
[7] Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision language concepts to vision & language models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2657- 2668,2668, 2023.
[8] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. ICLR, 2024.
[9] Zhiyuan Fang and Yezhou Yang. Knowledge distillation across vision and language. In Advancements in Knowledge Distillation: Towards New Horizons of Intellignnt Systems, pages 65-94, Springer, 2023.
[10] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
[11] Xiuye Gu, Tsung-Yi Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104. 13921,,11311.1.. 2021.
[12] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, Dawn Xiandong Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distributiio generalization. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8320-8329 (3200-83,,,.00 2020.
[13] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Xiaodong Song. Natural adversarial examples. 202/ IEEF/CV Conference on Computer Vision and Pattern Recognition (CVPR), pages 15257-15266. 2019.
[14] Cheng-Yu Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision- languuge compositionality. Advances in neural information processing systems, 36:31096-1 11331096311166 2023.
[15] Dong Jing, Xiaolong Hc, Yutian Luo, Nanyi Fei, Wei Wei, Huiwen Zhao, Zhiwu Lu, al. Fineclip: Self-distilled region- clip for better fine-grained undcrstanding. Advances in Neural Information Processing Systems, 17:27896-27918. 2024.
[16] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 10965-10975 2022.
[17] Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji- Rong Wen. Evaluating object hallucination in large vision-languse models. arXiv preprint 205:22005.10355, 2023.
11

---

[18] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13. pages 740-755 Springer, 2014.
[19] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[20] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information proessing systems, 36:34892-34916 2023.
[21] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216-233. Springer, 2024.
[22] Dovis-Kokitsi Maninis, Kaifeng Chen, Soham Ghosh, Arjun Karpur, Koert Chen, Ye Xia, Bingyi Cao. Daniel Salz, Guangxing Han, Jan Dlabal, Dan Gnanapragasam, Mojtaba Seyedhosseini, Howard Zhou, and André Araujo. TIPS: Text-Image Pretraining with Spatial Awareness. In International Conference on Learning Representations (ICLR), 2025.
[23] Muhammad Forjad Nacem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Lue Van Gool, and Federico Tombari. Sile: Improving vision language pretraining with self-distillation. In European Conference on Computer Vision, pages 38-55. Springer, 2024.
[24] Ivona Najdenkoska, Mohammad Mahdi Derakhshani, Yuki M Asano, Nanne van Noord, Marcel Worring, and Cees GM Snoek. Tulip: Token-length upgraded clip. ICLR, 2024.
[25] Yasumasa Once. Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho. Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garreut Tanzer, et al. Docci: Descriptions of connected and contrasting images. In European Conference on Computer Vision, pages 291-309. Springer, 2024.
[26] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 18X7vv8807.03748 2018
[27] Maitreya Patel, Naga Sai Abhiram Kusumba, Sheng Cheng. Changhoon Kim, Tejas Gokhale, Chitta Baral, et al. Tripletclip: Improving compositional reasoning of clip via synthetic vision-languee negatives. Advances in neural information processing systems, 37:32731-32760. 2024.
[28] Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. Synthesize diagnose and optimize: Towards fine- grained Dision-language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13279-13288, 2024.
[29] Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2644 2015.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-888 1763-876.. PmLR, 2021.
[31] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint ar 2204.22004.61222 1(2):3, 2022.
[32] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, 2019.
[33] Ugur Sahin, Hang Li, Qadeer Khan, Daniel Cremers, and Volker Tresp. Enhancing multimodal com- positional reasoning of visual language models with generative negative mining. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pages 5563-5573, 2024.
[34] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation amage-text models. Advances in newral information processing systems. 35:25278 25294, 2022.
12

---

[35] Harman Singh, Pengchuan Zhang. Qifan Wang. Mengjiao Wang, Wenhan Xiong, Jingfei Du and Yu Chen. Coarse-to-fim contrastive learning in mmage-text-graph space for improved vision-language compositional- ity. arXiv preprint arXiv:2305.13812 2023.
[36] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063 2024.
[37] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint 2rXiv:2303.1338) 2023.
[38] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5238-5248, 2022.
[39] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xic. Eyes wide shut? exploring the visual shortcomings of multimodal lls.. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9568-9578, 2024.
[40] Michael Tschannen, Alexey Gritsenko, Xiao Wang. Muhammad Ferjad Nacem, Tbrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual languae encoders with improved semantic understanding, localization, and dense features. arXiv preprint 2002 14786, 2025.
[41] Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero- Soriano. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26700- 26709, 2024.
[42] Haohan Wang. Songwei Ge, Eric P. Xing, and Zachary Chase Lipton. Learning robust global representations by penalizing local predictive power. In Neural Information Processing Systems, 2019.
[43] Tan Wang, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang. Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Equivariant similarity for vision-language foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11998-12008, 2023.
[44] Wenxuan Wang, Quan Sun, Fan Zhang, Yepeng Tang, Jing Liu, and Xinlong Wang. Diffusion feedback helps clip see better. ICLR, 2025.
[45] Yeyuan Wang, Dehong Gao, Lei Yi, Linbo Jin, Jinxia Zhang, Libin Yang, and Xiaoyan Cai. Enhancing fine-grained vision-language pretraining with negative augmented samples. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 8060-8068, 2025.
[46] Shengguang Wu, Fan-Yun Sun, Kaiyue Wen, and Nick Haber. Symmetrical visual contrastive optimization: Aligning vision-language models with minimal contrastive images. arXiv preprint arXiv.2502.13322, 2025.
[47] Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision transformer distills itself for open-vocabulary dense prediction. ICLR, 2024.
[48] Wei Wu, Kecheng Zheng, Shuailei Ma, Fan Lu, Yuxin Guo, Yifei Zhang, Wei Chen, Qingpei Guo, Yujun Shen, and Zue 7ha. Lotlip: Improving language-image pre-training for long text understanding. In Neural Information Processing Systems, 2024.
[49] Rui Xiao, Sanghwan Kim, Mariana-Iuliann Georgeseu, Zeynep Akata, and Stephan Alaniz, Flair: Vlm with fine-graines annuuageinformed image representations. arXiv preprint arXiv:2412.03561, 2024.
[50] Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li. Dawei Liang, Gengshen Zhang, Dawei Leng, and Yuhui Yin. Fg-elip: Fine-grained visual and textual alignment. In ICMI., 2025.
[51] Hu Xu, Saining Xie. Xiaoqing Ellen Tan, Ya-Yoo Huang. Russell Howes. Vasu Sharma, Shang- Meen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309.1660 2023.
[52] Kaicheng Yang. Jiankang Deng. Xiang An, Jiawei Li, Ziyong Feng. Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adaptive angguage-image pre-training with synthetic caption. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2922-2931, 2023.
13

---

[53] Mert Yuksekgonul. Federico Bianchi. Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what do about it? ICLR. 2022.
[54] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings the IEEE/CVF international conference compuler vision, pages 11975-11986, 2023.
[55] Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. European Conference on Computer Vision, pages 310 325. Springer, 2024.
[56] Le Zhang, Rabiul Awal, and Aishwarya Agrawal. Contrasting intra-modal and ranking cross-modal hard negatives to enhance visio-linguistic compositional understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13774-13784. 2024.
[57] Haozhe Zhao, Xiaojian Shawn Ma. Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-ball line-grained image editing at scale. Advances in Neural Information Processing Systems, 37:3058-3088 2024.
[58] Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language-iman image pre-training with long captions. In European Conference on Computer Vision, pages 73 90. Springer, 2024.
[59] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-basse basond language-image pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16793-16803, 2022.
[60] Chong Zhou, Chen Change Loy, Bo Dai. Extract free dense labels from clip. European Conference on Computer Vision, pages 696-712. Springer, 2022.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide a short (1-2 sentence) justification right after your answer (even for NA).
integral part of your paper submission. They visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The checklist answers
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd a proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" " or "we were unable to find the license for the dataset we used"). "[No] " or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope. The paper clearly identifies CLIP's s limitations in fine-grained visual perception and proposes CLIP-IN
solution that combines instruction editing data and long descriptive captions. The methodology section thoroughly details how instruction editing datasets repurposed as hard negatives and how the text encoder is adapted for long captions using RoPE. The experimental results across zero-shot classification, retrieval, fine-grained perception benchmarks (MMVP, FG-OVD), and MLLM performance support the claimed improvements. The paper also includes comprehensive ablation studies that demonstrate the complementary benefits of both data sources.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
15

---

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
The paper has discussed the limitations of the proposed work in the quality of our long caption datasets depends on the capabilities of the MILMM models used for generation. Similarly, our approach relies heavily on the quality and diversity of instruction editing datasets like Ultraddi.. These datasets may contain biases terms of the types of edits represented, potentially limiting the model's s ability to recognize certain fine-grained distinctions not well-represented in the training data.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" " section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper appropriately addresses all theoretical components with clear assumptions and complete proofs. For the contrastive learning formulations, the paper builds upon the established InfoNCE loss framework (Equation 1) and clearly extends it to the symmetric hard negative contrastive loss (Equations 3-5), with explicit assumptions about the structure of instruction editing data tuples.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
16

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides comprehensive information necessary to reproduce the main experimental results supporting the CLIP-IN approach. The methodology section clearly outlines the two-stage training pipeline, detailing how instruction editing data from UltraEdit (4 million samples) is repurposed as hard negatives and how long captions (averag- ing 300 tokens) generated using Intern VL. The paper explicitly defines all mathematical formulations, including the symmetric hard negative contrastive loss (Equations 3-5) and the overall training objective with specified weights. Implementation details are thoroughly documented, including hardware specifications, learning rate, and weight decay.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either be a way to access this model for reproducing the results or a way reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
We will release our data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cc) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.co
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Implementation details are thoroughly documented, including hardware speci- fications, learning rate, weight decay, and data splits.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper appropriately addresses statistical significance in its experimental evaluation. For the key benchmarks (MMVP, fine-graing recognition tasks, and MLLM evaluations), the authors report results with the same evaluation scripts and datasets with the state-of-thed art methods, ensuring the reliability of the performance improvements.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main claims the paper.
The factors of variability that the emor bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors). It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It OK to report --sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We include hardware specifications (16 NVIDIA A100 GPUs with 80G memory) in the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppppss/eerrps....cpppppe////p http://ss..c/puicccctccccccuceea
Answer: [Yes]
Justification: The research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: This work underscores the considerable potential of synergizing targeted, instruction-batt contrastive learning with comprehensive descriptive information to clevate the fine- grained understanding of VLMs.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Code, data and models, used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, parerssipppssiiiiicc. thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The submision has included details about training, license, limitations, etc.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy httpp:///ourrpp..c////oeeeeeeeee (025/LLM) for what should or should not be described.
22

---

A Comprehensive Evaluation with Diverse Classification Variants
Dataset. The IN-Ik (ImageNet-IKg benchmark serves as the foundational standard for evaluating image classification models, typically measuring top-1 accuracy on its large-scale validation set. IN-A 13 is dataset of real-world adversarially filtered images that fool current ImageNet classifiers. IN-V2 [32 introduces more challenging and modern test set to better reflect current model generalization capabilittis. while IN-R [22 Imngeeet-RRenitionn assesses robustness artistic and stylized depictions of object classes, and IN-S [42) (ImageNet-Sketch) evaluates performance on hand-drawn sketches, testing abstraction and shape understanding.
Results. Table preeentessetss comprehensive evaluation of our model (Ours) against the baseline models (OpenAI CLIP and Siglip2) across multiple ImageNet variants, including standard classification (IN-Ik), adversarial robustness (IN-A), artistic renditions (IN-R), sketch recognition (IN-S), and generalization new data (IN-V2). Across backbones, Ours consistently outperforms the corresponding baselines, demonstrating improved generalization and robustness. On the ViT-L backbone, our model achieves higher average accuracy than OpenAl CLIP (+0.2pp on 224 and +0.6pp on 336), with notable gains in IN-R (89.4 vs. 87.8) and IN-S (61.0 vs. 59.6), indicating stronger performance on stylized and abstract visual inputs. When compared to Siglip2 on S0400m architecture, Ours shows comparable or slightly better results, particularly on IN-R (95.7 vs. 95.3) and IN-$ (68.9 vs. 68. 1), while maintaining competitive performance on IN-A and IN-V2. The consistent improvements across diverse visual domains suggest that our method enhances the model's ability to generalize beyond standard photographic data, especially challenging conditions involving artistic variation and abstraction.
Table 9: Performance on ImageNet variants
Method
Backbone
Res
IN1k
IN-A
IN-R
IN-S
IN-V2
Avg
OpenAl CLIP 30
ViT-L/14
224
75.5
70.8
87.8
59.6
69.8
72.7
Ours
ViT-L/14
224
76.3
67.7
89.4
61.0
70.1
72.9
OpenAl CLIP
30
ViT-L/14
336
76.6
77.5
89.1
61.0
70.9
75.0
Ours
ViT-L/144
336
77.0
76.0
90.6
62.9
71.3
75.6
SigLIP2
40
ViT-S0/14
224
83.2
81.5
95.3
68.1
74.3
80.5
Ours
ViT-SO/14
224
83.4
81.3
95.7
68.9
74.6
80.8
SigLIP2
440
ViT-SO/16
384
84.1
83.2
96.7
70.2
76.1
82.1
Ours
ViT-SO/16
384
83.7
83.2
96.4
70.7
75.7
82.0
B Additional Ablation Studies
B.1 Impact of the Symmetric Hard Negative Loss (LHN)
To validate our proposed symmetric hard negative contrastive loss (LHN), we compare it against two other hard negative mining strategies. NegCLIP (53) loss which uses only the image-to-ter hard negative term and triplet loss 27 uses separate image-to-text and text-to-image but without the symmetric formulation.
use our main ViT-L/14@336 model this ablation. The results Table 10 confirm the superiority of our symmetric loss design. Our approach consistently and substantially outperforms both alternative strategies across all evaluated benchmarks. Compared to the Triplet loss, our symmetric loss achieves remarkable +2.8% improvement ImageNet- IK accuracy and boosts the average short-text retrieval score by 2.0%. The performance gap even more pronounced when compared to the one-directional NegCLIP loss, where our method shows gains of +3.1 on ImageNet- K and +2.4% on average retrieval.
B.2 Impact of Long Caption Data
To ensure the effectiveness of long caption data, we did the following ablation experiment on ViT- 1/14@336. Table TT show that the simpler model does not enhance fine-grained perception; in fact, it achieves an MMVP score of only 20.7. Our full framework, with the long-text component, reaches
23

---

Table 10: Ablation studies on Symmetric Hard Negative Loss (N)
CLS
Image-Texx Text Retrieval
Method
IN-IK
Flickr30k
MSCOCO
Top-1
Avg
T-1
1-T
T-+1
1-T
Ours (w/ Le loss)
73.9
70.7
77.9
90.5
48.6
65.9
Ours (w/ Triplet loss)
74.2
71.1
78.1
89.8
49.4
66.9
Ours proposed symmetric (_H
77.0
73.1
79.3
93.8
51.1
68.2
substantially higher score of 26.1. What's more, our full model also demonstrates significantly stronger performance across all retrieval tasks, outperforming the simpler model on both short-text retrieval (73.1 65.5) and long-text retrieval (76.4 vs. 69.0). In addition, the full framework also maintains a stronger ImageNet- classification score (77.0 vs. 76.3). Overall, This experiment confirms that the long-text capability is necessary component. The synergistic combination of long- caption context and hard-negative training is essential to achieve the reported gains in fine-grained visual perception and overall model performance.
Table 11: Ablation studies different training data
CLS
Short Caption Retrieval
Long Caption Retrieval ShareGPT4V
Method
IN-IK
MMVP
Flickr
COCO
DCI
Top-1
Avg
1-T
T-I
1-T
T-I
Avg
1-T
T-1
I-T
T-I
Ours (w/o long captions)
76.3
20.7
65.5
89.2
70.7
61.2
40.7
69.0
85.2
84.3
52.9
$3.6
Ours (full model)
77.0
26.1
73.1
93.8
79.3
68.2
51.1
76.4
93.5
91.6
58.4
61.9
B.3 Impact of I. Long Caption Data Scale
The identified gap in long caption retrieval performance appears to stem from the disparity training data scale (18M pairs vs. FG-CLIP's 1B pairs). To address this, we expanded our training dataset from 18M to 30M pairs. The results demonstrate our framework's scalability, as shown Table I2
Table 12: Ablation studies on long caption data scale
CLS
Short Caption Retrieval
Long Caption Retrieval ShareGPT4V
Method
IN-1K
Flickr
COCO
DCI
Top-1
Avg
I-I
T~
I-+T
T-I
Avg
I-:T
T-+I
I-+T
T-+I
FG-CLIP
76.1
93.7
81.5
68.9
50.9
81.8
97.4
96.8
66.7
66.1
Ours (18M long caption data)
77.0
73.1
93.8
79.3
68.2
51.1
76.4
93.5
91.6
58.4
61.9
Ours (30M long caption data)
76.2
74.9
94.2
81.5
70.3
53.5
85.1
97.9
96.0
72.7
73.8
Our updated model, trained on 30M pairs, achieves long caption retrieval accuracy of 85. 85,1%, surpassing FG-CLIP's performance of 81.8% by +3.3%. This highlights our framework' 's capability to excel with significantly less training data (over 30 times less data). Despite the improvements in retrieval performance, our classification results remain similar those of FG-CLIP, ensuring balanced overall performance across various metrics.
C Details of Compositional Benchmarks
Winoground [38] is a challenging benchmark designed to assess visio-linguistic compositional reasoning. It consists of 400 image-text sets, where each contains two images and two captions. The captions are minimally different (e.g., differing by a single word related to an object, attribute, or relation) and correspond to distinct images. SugarCrepe [14 is benchmark specifically curated to evaluate compositional reasoning in vision-language models by generating hard negative examples. aims to mitigate biases found in other datasets by presenting scenarios that require understanding attribute binding, relations, and verb-centric compositions. It contains over 96,000 image-text pairs across six different types compositional reasoning challenges. Performance is reported as an overall accuracy score. SPEC 28 benchmark designed to test fine-grained spatial and compositional

---

reasoning, It focuses on evaluating how well models can understand and differentiate images based on precise object arrangements and inter-object relationships described in captions. ARO a benchmark that evaluates vision-language models' ability to reason about fine-grained visual attributes and relational compositions in images. challenges models to distinguish subtle differences in object properties and spatial or semantic relationships through carefully designed minimal-pan image-text queries.
D Feature Visualization Analysis
In Figure present comparative visualization of feature activations across different methods, employing the feature extraction technique proposed by Zhou et al. [60]. In the visualization, warmer colors (e.g., yellow) signify higher similarity relevance to the target concept, while cooler colors (e.g., blue) indicate lower relevance or dissimilarity.
Examining the first set of examples, our proposed method demonstrates clear ability to precisely localize and identify specific attributes, such as the tie, and distinguish its color as either yellow or red. In contrast, FG-CLIP (50) focuses broadly the region of the cat without specifically highlighting the tic, thereby lacking fine-grained attribute recognition.
In the second set of examples, our approach continues to exhibit enhanced discriminative capability. It successfully identifies and localizes more nuanced objects distinct entities within complex scenes, such as "helmet," a "bear," or "flamingo" (depending on the specific image query). Conversely, both the original CLIP and FG-CLIP models often produce more diffuse attention maps or fail to accurately pinpoint these specific elements, underscoring the improved fine-grained understanding achieved by our method.
25

---

CLIP
FG-CLIP
Ours
Caption: A black cat wearing yellow bow tie.
Caption: black cat wearing red bow tie.
Caption: Two girls stand together with helmets on, one is holding pink bear.
Caption: Two girls stand together with helmets on, one holding flamingo.
Figure 3: Examples of feature visualization.
26

---

E Examples of Instruction Editing Hard Image-Texx Text Pairs
Instruct:
Replace the dog with a cat
Source Caption: dog's reflection a car door mirror.
Target Caption: cat's reflection car door mirror.
Instruct: Turn the cupcake into magical crystal ball
Source Caption: young girl sitting in chair eating cupcake.
Target Caption: young girl sitting in chair eating magical crystal ball.
Instruct: Replace the text with 'I'm going change my socks
Source Caption: A mug which reads I'm
IIM GONG CHANGE WWOLDD
I'M GONG CHANGE SOCKS
going to change the world.
Target Caption: A mug which reads I'm going to change my socks.
Instruct: Change the cows into flamingos Source Caption: Some cows are hanging out on the beach.
Target Caption: Flamingos casually hang out the beach, adding a touch of elegance and color the serene coastal scene.
Instruct: Make it look like a dessert pizza with chocolate and strawberries
Source Caption: A pizza with pepperoni, broccoli and cucumbers on it. Target Caption: A dessert pizza with chocolate and strawberries on it.
Instruct: Change the pink bear into flamingo
III
Source Caption:
Two girls stand together
with helmets on, one is holding a pink bear.
Target Caption: Two girls stand together with helmets on, one holding a flamingo.
Figure 4: Examples of instruction editing hard image-text pairs.
27


Reviews:
[{"review_id": "yTTnDwIfah", "reviewer": "Reviewer_Kftn", "summary": "This paper tackles the well known issue of CLIP performing poorly at fine-grained visual understanding. Many prior works have tried finetuning CLIP on fine-grained hard negative/positive data pairs to improve its fine-grained capabilities, but a limitation they face is that fine-grained captions typically are longer and require a text encoder capable of understanding long captions. To fix this, the authors propose a new text encoder for CLIP which utilizes rotary embeddings to generalize to longer captions.\n\nTraining proceeds in two stages: (1) In the first stage of training the new long text encoder is initialized through knowledge distillation from the original text encoder. (2) In the second stage CLIP-like contrastive training is carried out using a mix of long caption data and image editing data, which provides hard negative pairs for each modality. \n\nThe resuklts demonstrate that this approach (CLIP-IN) significantly boosts fine-grained perception without eroding CLIP’s broad zero-shot strengths. On the MMVP benchmark their ViT-L/14-224 model leaps from 18.5% to 30.4% average accuracy. Performance on standard benchmarks like ImageNet Zero-Shot classification, COCO Text to Image retrieval improves too. When utilizing the new CLIP-IN model as vision encoder for LLaVA performance on VL benchmarks like MME, MMBench and LLaVA-Wild gets a meaningful jump.", "questions": "- More detailed results comparing with NegCLIP, HardPositiveCLIP etc would be useful\n- Compositionality benchmark evaluations would add to the paper\n- Could the authors make a clear argument for their novelty against prior works ?\n\nI am open to raising my score if the concerns are addressed", "limitations": "The authors only discuss Limitations of their approach in the checklist itself, it would be good to have a limitations section", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n+ The problem addressed is important and the execution of the idea is solid, the results are excellent, with significant boosts on many tasks.\n+ Presentation: well-structured paper with intuitive figures and thorough experimental tables that make the method and results easy to follow\n\n**Weaknesses**\n\n- Some prior works such as NegCLIP [a] and HardPosCLIP[b] have also tried to improve the fine-grained understanding of CLIP models. These works typical test CLIP specifically on compositionality benchmarks like SugarCREPE, including such results in this work would be useful. Also a comparison to NegCLIP and HardPosCLIP would help.\n\n- Training relies on UltraEdit (image-editing using diffusion models) may limit transferability to real-world datasets \n\n- The basic contours of improving fine-grained capabilities of CLIP are well known (using hard negatives) and in the other direction some works such as LongCLIP have demonstrated how to boost the text encoder to longer texts, so the main contribution of this work is bringing these two together, making the novelty a bit limited.\n\n[a] Yuksekgonul, Mert, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. \"When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?.\" In The Eleventh International Conference on Learning Representations.\n\n[b] Kamath, Amita, Cheng-Yu Hsieh, Kai-Wei Chang, and Ranjay Krishna. \"The hard positive truth about vision-language compositionality.\" In European Conference on Computer Vision, pp. 37-54. Cham: Springer Nature Switzerland, 2024.", "was_revised": true, "final_justification": "NegCLIP Comparison and compositional benchmark results satisfied some of my key concerns.", "rebuttal": "We thank Reviewer Kftn for the constructive feedback and insightful comments. We hope to address the concerns of the Reviewer with the responeses below.\n\n### Q3. Contributions, w.r.t. prior works.\n\n**1) Novel Hard-Negative Sourcing**\nOur primary contribution is being the first to repurpose instruction-editing datasets as a novel source of hard negatives for contrastive learning. Unlike prior work that synthetically generates hard negatives via text-to-image models (TripletCLIP) or rule-based text perturbations (NegCLIP) , our approach uses data that provides high-fidelity hard negatives from real, minimally-edited image pairs, enabling a more effective and symmetric contrastive loss. By demonstrating a new, effective use for these datasets beyond their original purpose, our work contributes a valuable insight to the community.\n\n**2) Novel Synergistic Combination**\nWe are the first to propose a framework that synergistically combines these two complementary forms of fine-grained information:\n\n**Instruction Editing Data (Image-centric):** Provides fine-grained information for the visual representation.\n\n**Long Caption Data (Text-centric):** Provides fine-grained information for the textual representation.\n\nThis fusion of two complementary supervisory signals is the core of our framework. Our ablation studies (Tables 5 & 6 in manuscript) confirm this synergy, showing that the combination of both data types achieves the best performance and significantly outperforms models trained on either one alone.\n\nTo illustrate the benefits of our synergistic approach over methods that solely focus on long-text capabilities, we compare our model with LongCLIP. While LongCLIP extends the text encoder, our method complements this with instruction-editing data and uses RoPE for more effective long-text processing. The results below demonstrate the superiority of our approach.\n\n| Method   | Backbone  | CLS IN-1K (%) | Short Retrieval Avg. (%) | Flickr30k T->I | Flickr30k I->T | MSCOCO T->I | MSCOCO I->T |\n|----------|-----------|---------------|--------------------------|----------------|----------------|-------------|-------------|\n| LongCLIP | ViT-L-336 | 73.5          | 68.8                     | 76.2           | 90.0           | 46.3        | 62.8        |\n| Ours     | ViT-L-336 | 77.0          | 73.1                     | 79.3           | 93.8           | 51.1        | 68.2        |\n\nOur model significantly outperforms LongCLIP in both classification and retrieval, demonstrating that our synergistic approach leads to more robust and capable representations. We hope this clarifies the scope and significance of our primary contributions, which we will better stress in the introduction of the paper.\n\n### Q1. Comparions to NegCLIP [a].\n\nAs suggested, we provide a more direct comparison with prior works like NegCLIP[a] on compositional reasoning benchmarks.  We ablate our proposed hard negative loss with the NegCLIP loss.The results indicate that the symmetric LHN​ provides a more robust and well-rounded improvement to the model's capabilities. On standard benchmarks, the symmetric loss shows significant advantages, outperforming the NegCLIP loss by +3.1% on ImageNet-1K classification and +2.4% on the short-text retrieval average. This enhanced performance extends to complex compositional reasoning benchmarks like ARO and Winoground, where the proposed loss also achieves higher scores. While the NegCLIP loss scores higher on the SugarCrepe benchmark, which heavily focuses on text-based object and attribute swaps, our proposed symmetric loss demonstrates stronger overall performance. This suggests that the symmetric LHN​ fosters a more comprehensive fine-grained understanding by learning from both image and text mismatches, rather than specializing in a single type of textual perturbation.\n| Backbone  | CLS IN-1K (%) | Short Retrieval Avg. (%) | SugerCrepe | ARO  | Winoground | Method                            |\n|-----------|---------------|--------------------------|------------|------|------------|-----------------------------------|\n| ViT-L-336 | 73.9          | 70.7                     | 80.01      | 59.3 | 17.8       | Ours (w/ NegCLIP loss)            |\n| ViT-L-336 | 77.0          | 73.1                     | 77.2       | 60.7 | 18.1       | Ours (w/ proposed symmetric L_HN) |\n\n### Q2.Evaluations on Compositional Benchmarks.\n\nWe would like to gently point out that our original manuscript already includes a detailed evaluation on three compositional reasoning benchmarks—SugarCrepe, Winoground, and SPEC in the supplementary materials. To better address the reviewer's valid concern and provide a more comprehensive analysis, we have expanded our evaluation in the table below:\n| Method       | Backbone   | ARO avg     | ARO Relation/Attribute | MMVP | Winoground avg. | Winoground text | Winoground image | Winoground group | SugarCrepe avg | SPEC I->T avg./T->I avg. | SPEC avg |\n|--------------|------------|-------------|------------------------|------|-----------------|-----------------|------------------|------------------|----------------|--------------------------|----------|\n| OpenAI CLIP  | ViT-L/224  | 58.9        | 59.3/58.5              | 18.5 | 15.9            | 28.3            | 10.5             | 8.8              | 75.6           | 33.2/31.3                | 32.3     |\n| Ours         | ViT-L/224  | 64.4        | 64.3/64.4              | 30.4 | 17.1            | 28.0            | 13.8             | 9.5              | 77.5           | 37.6/35.0                | 36.3     |\n| OpenAI CLIP  | ViT-L/336  | 61.0        | 60.1/61.9              | 20.0 | 15.4            | 28.3            | 10.5             | 7.5              | 74.8           | 32.8/31.1                | 32.1     |\n| Ours         | ViT-L/336  | 60.7        | 58.1/63.2              | 26.1 | 18.1            | 33.0            | 11.8             | 9.5              | 77.2           | 35.14/35.2               | 35.2     |\n| Siglip2      | so400m/224 | 49.7        | 49.0/50.4              | 35.6 | 6.9             | 9.0             | 9.3              | 2.5              | 49.5           | 27.4/27.2                | 27.3     |\n| Ours         | so400m/224 | 50.7        | 49.5/51.9              | 36.3 | 8.5             | 14.3            | 7.5              | 3.8              | 50.5           | 30.6/30.4                | 30.5     |\n| Siglip2      | so400m/336 | 48.9        | 47.3/50.5              | 35.6 | 6.7             | 9.3             | 8.5              | 2.3              | 50.9           | 27.6/27.5                | 27.5     |\n| Ours         | so400m/336 | 50.5        | 50.9/50.0              | 36.3 | 7.0             | 13.5            | 5.5              | 2.0              | 51.7           | 30.2/30.8                | 30.5     |\n\nAs evidenced by the results, our CLIP-IN framework consistently outperforms baseline models across a diverse range of compositional benchmarks:\n\nConsistent Improvement over Baselines: Our method significantly surpasses the original OpenAI CLIP (ViT-L/224), with notable gains of +5.5% on ARO, +1.9% on SugarCrepe, and +4.0% on SPEC. This trend persists across nearly all benchmarks when using other backbones.\n\nSuperiority over SigLIP2: Our framework enhances the compositional reasoning capabilities of the robust SigLIP2 model, boosting performance across all four benchmarks when applied to its backbone, which underscores the broad applicability and effectiveness of our approach.\n\n### W4. Transferability to real-world datasets.\n\nWe would like to offer two key pieces of evidence from our paper that demonstrate our method's strong transferability to real-world data:\n\n**1) Grounded in Real Images:** Unlike methods that generate images from scratch, the UltraEdit dataset starts with real images (i.e., high-quality image-caption pairs from diverse real image datasets like COCO) and applies localized, instruction-guided edits. As we state in our paper, this \"mitigates domain biases of purely generated data\". This grounding makes the learned representations more robust.\n\n**2) Strong Zero-Shot Performance:** The most direct evidence of transferability is our model's performance on standard, real-world benchmarks. As shown in Table 1 of original manuscript, our CLIP-IN model achieves significant zero-shot performance gains on ImageNet-1K, Flickr30K, and MS-COCO. For instance, with a ViT-L/14 backbone, our method improves average short-text retrieval accuracy from 62.5% to 73.1% and ImageNet-1K accuracy from 76.6% to 77.0% over the strong OpenAI CLIP baseline. These consistent improvements on diverse, real-world datasets shows that our training strategy generalizes effectively and is not limited to the training distribution."}, {"review_id": "ZfBPFUas1r", "reviewer": "Reviewer_mmoh", "summary": "The authors propose a training pipeline called CLIP-IN to address CLIP's limitations in understanding fine-grained visual details. The authors firstly repurpose instruction-editing datasets as a source of high-quality \"hard negatives\" to train the model to distinguish subtle visual-semantic differences via a symmetric contrastive loss. They then adapt the CLIP text encoder to process long, descriptive captions by integrating RoPE via knowledge distillation. Experiments show that CLIP-IN improves model performance and enhances downstream MLLMs by reducing visual hallucinations.", "questions": "1. Can the author provide ablation studies on using RoPE and the improvements brought by their training loss design as mentioned in weaknesses section.\n\n2. Since the framework relies on the instruction editing dataset, How sensitive is the framework to the selection, variety, and scale of instruction editing data? If certain objects or relations are underrepresented, does the model exhibit new bias or failure modes?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. This work cleverly repurposes instruction-editing datasets to provide a scalable and high-quality dataset for contrastive learning, avoiding reliance on synthetic data, which often lacks realism and control.\n\n2. The paper presents a well-structured two-stage framework to improve training performance. The combination of two complementary data sources is well-motivated. The use of RoPE and knowledge distillation is an effective solution to the long-text limitation. The overall design is thoughtful.\n\n3. This work addresses a clearly stated limitation of CLIP and may benefit some existing work in spatial understanding and MLLMs.\n\nWeaknesses:\n\n1. In some metrics like long caption retrieval, CLIP-IN falls behind FG-CLIP. The authors attribute this to the larger training dataset that FG-CLIP leverages, which might also raise questions about whether CLIP-IN's performance can scale with training data size.\n\n2. The ablation section only discusses the impact of different data sources, but lacks analysis of component-level effects. The authors should discuss the influence of specific design choices, such as the impact of using RoPE and the improvements brought by their training loss design.", "was_revised": true, "final_justification": "The rebuttal resolves most of my concerns and I will main the rating of 4.", "rebuttal": "We thank Reviewer mmoh for the constructive feedback and insightful comments. We are grateful for the acknowledgment of our work's strengths, including the clever repurposing of instruction-editing datasets, the well-structured framework, and its potential to benefit the community. We hope to address the concerns of the Reviewer with the responses below.\n\n### W1. Scaling Long Caption Data\n\nWe agree with the reviewer that demonstrating the scalability of our framework is crucial. To directly address the reviewer's question about scalability, we have conducted a new experiment where we increased the amount of long caption data for training CLIP-IN from 18M to 30M pairs. The results, compared with our initial model and FG-CLIP, are presented below.\n\n| Method         | Long Caption Data | CLS IN-1K (%) | Short Retrieval Avg. (%) | Long Retrieval Avg. (%) | Flickr30k T->I | Flickr30k I->T | MSCOCO T->I | MSCOCO I->T | MMVP | ShareGPT4v T->I | ShareGPT4v I->T | DCI T->I | DCI I->T |\n|----------------|-------------------|---------------|--------------------------|-------------------------|----------------|----------------|-------------|-------------|------|----------------|----------------|---------|---------|\n| Ours (Original)| 18M               | 77.0          | 73.1                     | 76.4                    | 79.3           | 93.8           | 51.1        | 68.2        | 26.1 | 91.6           | 93.5           | 61.9    | 58.4    |\n| Ours (New)     | 30M               | 76.2          | 74.9                     | 85.1 (+8.7)             | 81.5           | 94.2           | 53.5        | 70.3        | 27.4 | 96.0           | 97.9           | 73.8    | 72.7    |\n| FG-CLIP        | 1B                | 76.1          | 73.6                     | 81.8                    | 81.5           | 93.7           | 50.9        | 68.9        | -    | 96.8           | 97.4           | 66.1    | 66.7    |\n\nDoubling the long caption data from 4M to 8M results in a dramatic improvement in the Long Retrieval Average from 76.4% to 85.1% (+8.7 points). Performance also improves on both Short Retrieval (+1.8 points) and the fine-grained MMVP benchmark (+1.3 points). This clearly demonstrates that our framework's performance scales effectively with more data. With this new data, our model now outperforms FG-CLIP on the primary Long Retrieval metric (85.1% vs. 81.8%) and also surpasses it on Short Retrieval. This is achieved while using vastly less data (30M vs. 1B), highlighting the data efficiency of our synergistic training approach. While there is a slight trade-off in general classification, our ImageNet-1K score remains competitive with FG-CLIP.\n\n### Q1. Ablation on RoPE and the Proposed Hard Negative Loss\n\n**1) Impact of Rotary Positional Embeddings (RoPE)**\n\nTo isolate the benefit of our RoPE-based text encoder distillation, we compare it against a strong baseline that uses absolute positional encoding interpolation, as popularized by LongCLIP. We initialize our second stage of training with LongCLIP, a text encoder pre-trained using positional interpolation and compare it to our proposed RoPE-based model. All other settings are identical.\n\nThe results show that our RoPE-based approach is a more effective method for extending the text encoder's context length. It significantly outperforms the interpolation baseline on general classification (IN-1K: +1.2%), long-text retrieval (+6.0%). \n\n| Method                                  | CLS IN-1K (%) | Short Retrieval Avg. (%) | Long Retrieval Avg. (%) | Flickr30k T->I | Flickr30k I->T | MSCOCO T->I | MSCOCO I->T | ShareGPT4v T->I | ShareGPT4v I->T | DCI T->I | DCI I->T |\n|-----------------------------------------|---------------|--------------------------|-------------------------|----------------|----------------|-------------|-------------|-----------------|-----------------|----------|----------|\n| Ours (Initialized with LongCLIP)        | 75.0          | 74.2                     | 79.1                    | 80.9           | 93.3           | 52.7        | 69.8        | 96.5            | 97.0            | 60.8     | 62.2     |\n| Ours (Initialized with RoPE-based text encoder) | 76.2          | 74.9                     | 85.1                    | 81.5           | 94.2           | 53.5        | 70.3        | 96.0            | 97.9            | 73.8     | 72.7     |\n\n\n**2) Impact of the Symmetric Hard Negative Loss (L_HN​).**\n\nTo validate our proposed symmetric hard negative contrastive loss (LHN​), we compare it against two other hard negative mining strategies:\n\nNegCLIP loss: Uses only the image-to-text hard negative term.\n\nTriplet loss: Uses separate image-to-text and text-to-image triplet losses but without the symmetric formulation.\n\nWe use our main ViT-L/14@336 model for this ablation. The results below confirm the superiority of our symmetric loss design.\n\n| Loss Function                      | CLS IN-1K (%) | Short Retrieval Avg. (%) | Flickr30k T->I | Flickr30k I->T | MSCOCO T->I | MSCOCO I->T |\n|------------------------------------|---------------|--------------------------|----------------|----------------|-------------|-------------|\n| Ours (w/ NeCLIP loss)              | 73.9          | 70.7                     | 77.9           | 90.5           | 48.6        | 65.9        |\n| Ours (w/ Triplet loss)             | 74.2          | 71.1                     | 78.1           | 89.8           | 49.4        | 66.9        |\n| Ours (w/ proposed symmetric L_HN)  | 77.0          | 73.1                     | 79.3           | 93.8           | 51.1        | 68.2        |\n\nOur approach consistently and substantially outperforms both alternative strategies across all evaluated benchmarks. Compared to the Triplet loss, our symmetric loss achieves a remarkable +2.8% improvement on ImageNet-1K accuracy and boosts the average short-text retrieval score by 2.0%. The performance gap is even more pronounced when compared to the one-directional NegCLIP loss, where our method shows gains of +3.1% on ImageNet-1K and +2.4% on average retrieval.\n\n### Q2. Sensitivity to Instruction Editing Data.\n\n**1) The Effect of Scale.** \nThe framework's performance scales positively with the amount of instruction editing data. As detailed in the original manuscript's ablation study, increasing the dataset size from 1.4 million to 4 million samples improved ImageNet-1K accuracy by 0.6% while maintaining high performance on retrieval tasks. This indicates that the model benefits from more data.\n\n**2) The Effect of Selection and Variety.**\nTo analyze the impact of data variety and potential biases, we performed two controlled experiments using a 1.4 million sample subset as a baseline. We compared the baseline, which was randomly sampled, against one curated subsets: balanced by the most frequent visual concepts (objects and colors). By curating the dataset to ensure a balanced distribution of core visual concepts, the model achieves substantial performance gains on tasks that require detailed understanding. This strategy leads to a significant improvement in both short-text retrieval (+2.4) and, critically, fine-grained visual perception on MMVP (+2.0).\n\n| Training Data (1.4M)                | CLS IN-1K (%) | Short Retrieval Avg. (%) | MMVP |\n|-------------------------------------|---------------|--------------------------|------|\n| Random Sampling                     | 75.8          | 72.9                     | 26.1 |\n| Balancing by Objects and Colors     | 75.2          | 75.3                     | 28.1 |"}, {"review_id": "Z4XFW4ALz0", "reviewer": "Reviewer_Sser", "summary": "The problem of addressing the fine-grained visual understanding of CLIP-like (dual-encoder) models is considered. This work proposed CLIP with INstruction edit data and INformative long data (CLIP-IN), a new framework for CLIP training, where technical novelties such as a symmetric hard negative contrastive loss, rotary positional encoding are introduced. CLIP-IN is shown to achieve superior performance in text-image retrieval.", "questions": "1. For the column of IN-1K CLS in Table 1, why is EVA-CLIP's 80.4 highlighted? It  is not the best result.\n\n2. When making comparison to FG-CLIP L251-L254, it is unfair to state that CLIP-IN (76.4%) \"trails slightly behind\" FG-CLIP (81.8%) while the Image classification accuracy of CLIP-IN \"significantly exceeds\" that of FG-CLIP (76.1%).", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths  \n1. Collection of hard negative image-caption pairs via instruction editting data.\n2. Technical novelties: a symmetric hard negative contrastive loss, rotary position encoding for long captions.\n\n\nWeaknesses  \n1. The issue of poor fine-grained alignment of CLIP is well-known in the community, and motivates the introduction of the topic of vision-language compositionality. Evaluations on image classification and text-image retrieval are not sufficient to validate the claim that CLIP-IN is superior. More quantitative evaluations on relevant benchmarks for VL compositionality (e.g., SugarCrepe [T1], ARO [T2], VL-Checklist [T3], and more proposed after 2023).  \n\nUpdate: I noticed that there is a comparison between CLIP-IN and OPENAI's CLIP in the appendix. This should be brought up in the main paper, as it is more relevant for the problem of fine-grained visual understanding. Also, evaluation on more recent compositionality and comparison against more competent baselines are necessary.\n\n[T1] SugarCrepe: A benchmark for faithful vision-language compositionality evaluation. NIPS’23  \n[T2] When and why vision-language models behave like bags-of-words, and what to do about it? ICLR’23.  \n[T3] Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations.  \n\n2. Evaluation of 0-shot image classification on ImageNet-1K is far from being enough. Evaluation on a standard set of datasets, e.g. CLIP-Benchmark (https://github.com/LAION-AI/CLIP_benchmark), is missing. \n\n3. Ablation study of different parts of technical novelty in CLIP-IN is missing.", "was_revised": true, "final_justification": "Thank the authors for the response and addressing my concerns. I decided to change my rating of borderline reject to borderline accept.", "rebuttal": "We thank Reviewer Sser for the constructive feedback and insightful comments. We hope to address the concerns of the Reviewer with the responeses below.\n\n### W1. More Evaluations on Compositionality Benchmarks.\n\nWe will ensure these results are featured prominently in the main body of the revised paper. To better address the reviewer's valid concern and provide a more comprehensive analysis, we have expanded our evaluation in the table below:\n\n| Method       | Backbone   | ARO avg     | ARO Relation/Attribute | MMVP | Winoground avg. | Winoground text | Winoground image | Winoground group | SugarCrepe avg | SPEC I->T avg./T->I avg. | SPEC avg |\n|--------------|------------|-------------|------------------------|------|-----------------|-----------------|------------------|------------------|----------------|--------------------------|----------|\n| OpenAI CLIP  | ViT-L/224  | 58.9        | 59.3/58.5              | 18.5 | 15.9            | 28.3            | 10.5             | 8.8              | 75.6           | 33.2/31.3                | 32.3     |\n| Ours         | ViT-L/224  | 64.4        | 64.3/64.4              | 30.4 | 17.1            | 28.0            | 13.8             | 9.5              | 77.5           | 37.6/35.0                | 36.3     |\n| OpenAI CLIP  | ViT-L/336  | 61.0        | 60.1/61.9              | 20.0 | 15.4            | 28.3            | 10.5             | 7.5              | 74.8           | 32.8/31.1                | 32.1     |\n| Ours         | ViT-L/336  | 60.7        | 58.1/63.2              | 26.1 | 18.1            | 33.0            | 11.8             | 9.5              | 77.2           | 35.14/35.2               | 35.2     |\n| Siglip2      | so400m/224 | 49.7        | 49.0/50.4              | 35.6 | 6.9             | 9.0             | 9.3              | 2.5              | 49.5           | 27.4/27.2                | 27.3     |\n| Ours         | so400m/224 | 50.7        | 49.5/51.9              | 36.3 | 8.5             | 14.3            | 7.5              | 3.8              | 50.5           | 30.6/30.4                | 30.5     |\n| Siglip2      | so400m/336 | 48.9        | 47.3/50.5              | 35.6 | 6.7             | 9.3             | 8.5              | 2.3              | 50.9           | 27.6/27.5                | 27.5     |\n| Ours         | so400m/336 | 50.5        | 50.9/50.0              | 36.3 | 7.0             | 13.5            | 5.5              | 2.0              | 51.7           | 30.2/30.8                | 30.5     |\n\nAs evidenced by the results, our CLIP-IN framework consistently outperforms baseline models across a diverse range of compositional benchmarks:\n\nConsistent Improvement over Baselines: Our method significantly surpasses the original OpenAI CLIP (ViT-L/224), with notable gains of +5.5% on ARO, +1.9% on SugarCrepe, and +4.0% on SPEC. This trend persists across nearly all benchmarks when using other backbones.\n\nSuperiority over SigLIP2: Our framework enhances the compositional reasoning capabilities of the robust SigLIP2 model, boosting performance across all four benchmarks when applied to its backbone, which underscores the broad applicability and effectiveness of our approach.\n\n### W2. More Evaluations on CLIP-Benchmark.\n\nWe agree with the reviewer that it is crucial to provide more evaluations on CLIP-Benchmark and we include more results in the following table. It shows the our proposed method obtains consistent improvement over baseline in the zero-shot image classification task.\n\n| Method       | Backbone       | IN1k  | IN-A  | IN-R  | IN-S  | IN-V2 | Avg.  |\n|--------------|----------------|-------|-------|-------|-------|-------|-------|\n| OpenAI CLIP  | ViT-L/224      | 75.5  | 70.8  | 87.8  | 59.6  | 69.8  | 72.7  |\n| Ours         | ViT-L/224      | 76.3  | 67.7  | 89.4  | 61.0  | 70.1  | 72.9  |\n| OpenAI CLIP  | ViT-L/336      | 76.6  | 77.5  | 89.1  | 61.0  | 70.9  | 75.0  |\n| Ours         | ViT-L/336      | 77.0  | 76.0  | 90.6  | 62.9  | 71.3  | 75.6  |\n| Siglip2      | SO400m/224     | 83.2  | 81.5  | 95.3  | 68.1  | 74.3  | 80.5  |\n| Ours         | SO400m/224     | 83.4  | 81.3  | 95.7  | 68.9  | 74.6  | 80.8  |\n| Siglip2      | SO400m/384     | 84.1  | 83.2  | 96.7  | 70.2  | 76.1  | 82.1  |\n| Ours         | SO400m/384     | 83.7  | 83.2  | 96.4  | 70.7  | 75.7  | 82.0  |\n\n\n### W3. More Ablation Studies.\n\nWhile our original manuscript (Table 5 and Table 6) included an ablation study on the impact of our two primary data sources (instruction editing data and long caption data), we agree that more detailed ablations on the core technical components of our framework are beneficial. In response, we have conducted two new component-level ablation studies to validate the effectiveness of our proposed symmetric hard negative loss and our RoPE-based text encoder distillation.\n\n**1)  Ablation on RoPE-based Text Encoder Distillation.**\n\nTo evaluate the benefits of our RoPE-based text encoder distillation, we compared it to a robust baseline utilizing absolute positional encoding interpolation, commonly adopted by LongCLIP. We initialized the second training stage with LongCLIP, a text encoder pre-trained with positional interpolation, and contrasted it with our RoPE-based model, keeping all other settings constant.\n\nThe results show that our RoPE-based approach is a more effective method for extending the text encoder's context length. It significantly outperforms the interpolation baseline on general classification (IN-1K: +1.2%), long-text retrieval (+6.0%). \n\n| Method | CLS IN-1K (%) | Short Retrieval Avg. (%) | Long Retrieval Avg. (%) | Flickr30k T->I | Flickr30k I->T | MSCOCO T->I | MSCOCO I->T | ShareGPT4v T->I | ShareGPT4v I->T | DCI T->I | DCI I->T |\n|--------|---------------|--------------------------|-------------------------|----------------|----------------|-------------|-------------|-----------------|-----------------|----------|----------|\n| Ours (Initialized with LongCLIP) | 75.0 | 74.2 | 79.1 | 80.9 | 93.3 | 52.7 | 69.8 | 96.5 | 97.0 | 60.8 | 62.2 |\n| Ours (Initialized with RoPE-based text encoder) | 76.2 | 74.9 | 85.1 | 81.5 | 94.2 | 53.5 | 70.3 | 96.0 | 97.9 | 73.8 | 72.7 |\n\n**2) Ablation on Symmetric Hard Negative Loss (L_HN​).**\n\nTo evaluate the effectiveness of our symmetric hard negative contrastive loss (L_HN), we compared it with two other hard negative mining strategies: NegCLIP Loss, which uses only the image-to-text hard negative term, and Triplet Loss, which employs separate image-to-text and text-to-image triplet losses but without a symmetric formulation.\n\nUsing our primary ViT-L/14@336 model for this ablation, the results clearly demonstrated the superiority of our symmetric loss design, consistently and significantly outperforming both alternatives across all evaluated benchmarks. Compared to the Triplet loss, our symmetric loss achieved an impressive +2.8% improvement in ImageNet-1K accuracy and a 2.0% boost in average short-text retrieval scores. The performance gap is even more pronounced when compared to the one-directional NegCLIP loss, with gains of +3.1% on ImageNet-1K and +2.4% in average retrieval.\n\n| Loss Function | CLS IN-1K (%) | Short Retrieval Avg. (%) | Flickr30k T->I | Flickr30k I->T | MSCOCO T->I | MSCOCO I->T |\n|---------------|---------------|--------------------------|----------------|----------------|-------------|-------------|\n| Ours (w/ NeCLIP loss) | 73.9 | 70.7 | 77.9 | 90.5 | 48.6 | 65.9 |\n| Ours (w/ Triplet loss) | 74.2 | 71.1 | 78.1 | 89.8 | 49.4 | 66.9 |\n| Ours (w/ proposed symmetric $L_{HN}$) | 77.0 | 73.1 | 79.3 | 93.8 | 51.1 | 68.2 |\n\n### Q1.\nThat was a typo in our draft. We will revise it in the final version.\n\n### Q2. Comparisons to FG-CLIP.\nThe identified gap in long caption retrieval performance appears to stem from the disparity in training data scale (18M pairs vs. FG-CLIP's 1B pairs). To address this, we expanded our training dataset from 18M to 30M pairs. The results demonstrate our framework's scalability, as shown in the table below:\n\n| Method         | Long Caption Data | CLS IN-1K (%) | Short Retrieval Avg. (%) | Long Retrieval Avg. (%) | Flickr30k T->I | Flickr30k I->T | MSCOCO T->I | MSCOCO I->T | MMVP ShareGPT4v T->I | MMVP ShareGPT4v I->T | DCI T->I | DCI I->T |\n|----------------|------------------|---------------|-------------------------|------------------------|----------------|----------------|-------------|-------------|----------------------|----------------------|----------|----------|\n| Ours (Original)| 18M              | 77.0          | 73.1                    | 76.4                   | 79.3           | 93.8           | 51.1        | 68.2        | 26.1                 | 91.6                 | 93.5     | 61.9     | 58.4     |\n| Ours (New)     | 30M              | 76.2          | 74.9                    | 85.1 (+8.7)            | 81.5           | 94.2           | 53.5        | 70.3        | 27.4                 | 96.0                 | 97.9     | 73.8     | 72.7     |\n| FG-CLIP        | 1B               | 76.1          | 73.6                    | 81.8                   | 81.5           | 93.7           | 50.9        | 68.9        | -                    | 96.8                 | 97.4     | 66.1     | 66.7     |\n\nOur updated model, trained on 30M pairs, achieves a long caption retrieval accuracy of 85.1%, surpassing FG-CLIP's performance of 81.8% by +3.3%. This highlights our framework's capability to excel with significantly less training data (over 30 times less data). Despite the improvements in retrieval performance, our classification results remain similar to those of FG-CLIP, ensuring a balanced overall performance across various metrics."}, {"review_id": "iY6VVUBYDd", "reviewer": "Reviewer_RrCr", "summary": "This paper proposes a framework called CLIP-IN, which involves two stages of training. In the first stage, a vision-language model (VLM) is used to generate long-form descriptions, and a distillation approach is applied to learn the text encoder’s RoPE (Rotary Positional Embedding). In the second stage, large-scale hard negative samples are constructed using existing instruction editing datasets—covering both image-side and text-side perturbations—and contrastive learning with these challenging negatives is employed to enhance the model’s ability to distinguish fine-grained visual-semantic differences. The method demonstrates improvements across a range of downstream benchmarks, supported by comprehensive comparative and ablation studies.", "questions": "See the above weakness.", "limitations": "The authors are encouraged to include a limitation section in the final version.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. The idea of converting instruction editing datasets into hard negatives is interesting, as it leverages existing data resources and avoids unnecessary waste.\n2. The reviewer finds the text encoder distillation in the first stage particularly intriguing, as it appears beneficial for improving CLIP’s ability to handle long-text inputs.\n3. The overall approach is easy to understand, with a clear and well-structured methodology.\n4. The experiments are relatively comprehensive, covering short-text retrieval, long-text retrieval, zero-shot classification, and a series of downstream tasks with LLaVA.\n\n## Weakness\n1. While there is some degree of novelty, it appears to be limited. The authors essentially combine fine-grained perception (e.g., MMVP) with long-text understanding (e.g., ShareGPT4V), forming the basis of their two-stage approach: the first stage aims to enhance long-text handling, while the second focuses on improving the model’s sensitivity to fine-grained visual-semantic differences.\n2.  The ablation study is not fully complete. For instance, it lacks an analysis of the impact of the text encoder distillation in the first stage. It would be important to evaluate whether simpler alternatives—such as Long-CLIP's positional interpolation strategy—could similarly improve long-text understanding.\n3. The reviewer questions whether long-text capability is truly necessary. For example, Table 6 shows that after incorporating LoneData, MMVP performance drops from 24.7 to 20.9. Moreover, the reviewer believes that even without the first-stage distillation or the re-captioned data, simply using the original 18M dataset combined with the hard negative samples could still enhance fine-grained perception (MMVP). \n4. Why use InternVL2 to generate detailed captions?", "was_revised": true, "final_justification": "Thanks for the author's response, which addresses most of my concerns, and I will update my score.", "rebuttal": "We thank Reviewer RrCr for the constructive feedback and insightful comments. We hope to address the concerns of the Reviewer with the responses below.\n\n### W1. Novelty and Contributions\n\nWhile it is true that our framework focuses on enhancing both long-text understanding and fine-grained perception, the essence of our contributions and innovation lies in the specific methodologies employed to achieve these objectives, and most importantly, in the synergistic integration of these components.\n\n**1) Novel Hard-Negative Sourcing**\n\nOur primary contribution is being the first to repurpose instruction-editing datasets as a novel source of hard negatives for contrastive learning. Unlike prior work that synthetically generates hard negatives via text-to-image models (TripletCLIP) or rule-based text perturbations (NegCLIP), our approach uses data that provides high-fidelity hard negatives from real, minimally-edited image pairs, enabling a more effective and symmetric contrastive loss. By demonstrating a new, effective use for these datasets beyond their original purpose, our work contributes a valuable insight to the community.\n\n**2) Novel Synergistic Combination**\n\nWe are the first to propose a framework that synergistically combines these two complementary forms of fine-grained information:\n\n- **Instruction Editing Data (Image-centric):** Provides fine-grained information for the visual representation.\n- **Long Caption Data (Text-centric):** Provides fine-grained information for the textual representation.\n\nThis fusion of two complementary supervisory signals is the core of our framework. Our ablation studies (Tables 5 & 6 in manuscript) confirm this synergy, showing that the combination of both data types achieves the best performance and significantly outperforms models trained on either one alone.\n\n### W2. Ablation on RoPE-based Text Encoder Distillation\n\nTo isolate the benefit of our RoPE-based text encoder distillation, we compare it against a strong baseline that uses absolute positional encoding interpolation, as popularized by LongCLIP. We initialize our second stage of training with LongCLIP, a text encoder pre-trained using positional interpolation and compare it to our proposed RoPE-based model. All other settings are identical.\n\nThe results show that our RoPE-based approach is a more effective method for extending the text encoder's context length. It significantly outperforms the interpolation baseline on general classification (IN-1K: +1.2%), long-text retrieval (+6.0%).\n\n| Method                                    | CLS IN-1K (%) | Short Retrieval Avg. (%) | Long Retrieval Avg. (%) | Flickr30k T->I | Flickr30k I->T | MSCOCO T->I | MSCOCO I->T | ShareGPT4v T->I | ShareGPT4v I->T | DCI T->I | DCI I->T |\n|-------------------------------------------|---------------|--------------------------|-------------------------|----------------|----------------|-------------|-------------|-----------------|-----------------|----------|----------|\n| Ours (Initialized with LongCLIP)          | 75.0          | 74.2                     | 79.1                    | 80.9           | 93.3           | 52.7        | 69.8        | 96.5            | 97.0            | 60.8     | 62.2     |\n| Ours (Initialized with RoPE-based text encoder) | 76.2     | 74.9                     | 85.1                    | 81.5           | 94.2           | 53.5        | 70.3        | 96.0            | 97.9            | 73.8     | 72.7     |\n\n### W3. Capability of Long Caption Data\n\nIn response to the reviewer's query, we provide the following analysis and new experimental data to clarify the necessity of the long-text component within our synergistic framework.\n\n- The reviewer's observation regarding the drop in MMVP performance when using only long-text data (as shown in our initial ablation study in Table 6) highlights the central focus of our work. It demonstrates that the components are synergistic, and their true potential is realized only when they are combined. While a single data source in isolation may not improve every metric, the final model combining both instruction data and long captions showed a +3.3% improvement on the MMVP MLLM benchmark over the baseline.\n\n- To directly test the reviewer's valuable hypothesis that a simpler model (using 18M original short captions and hard negatives) could enhance fine-grained perception, we conducted a new experiment. The results are detailed below, comparing the simpler model (\"w/o long captions\") against our full framework.\n\n\n| Method               | CLS IN-1K (%) | Short Retrieval Avg. (%) | Long Retrieval Avg. (%) | Flickr30k T->I | Flickr30k I->T | MSCOCO T->I | MSCOCO I->T | ShareGPT4v T->I | ShareGPT4v I->T | DCI T->I | DCI I->T |\n|----------------------|---------------|--------------------------|-------------------------|----------------|----------------|-------------|-------------|-----------------|-----------------|----------|----------|\n| Ours (w/o Long Captions) | 76.3          | 65.45                    | 69                      | 20.7           | 70.7           | 89.2        | 40.7        | 61.2            | 84.3            | 85.2     | 53.6     | 52.9     |\n| Ours (Full Model)    | 77            | 73.1                     | 76.35                   | 26.1           | 79.3           | 93.8        | 51.1        | 68.2            | 91.6            | 93.5     | 61.9     | 58.4     |\n\nThe results from this new experiment provide a definitive answer:\n\n- **Fine-Grained Perception:** The data shows the simpler model does not enhance fine-grained perception; in fact, it achieves an MMVP score of only 20.7. Our full framework, with the long-text component, reaches a substantially higher score of 26.1.\n\n- **Retrieval Performance:** Our full model also demonstrates significantly stronger performance across all retrieval tasks, outperforming the simpler model on both short-text retrieval (73.1 vs. 65.45) and long-text retrieval (76.35 vs. 69.0).\n\n- **Classification:** The full framework also maintains a stronger ImageNet-1K classification score (77.0 vs. 76.3).\n\nThis experiment confirms that the long-text capability is a necessary component. The synergistic combination of long-caption context and hard-negative training is essential to achieve the reported gains in fine-grained visual perception and overall model performance.\n\n### W4. Model for Long Caption Generation\n\nWe selected InternVL2 because it was one of the state-of-the-art open-source vision-language models available at the time of our research. Our method is not dependent on this specific choice and could easily incorporate detailed captions generated by other current or future state-of-the-art models."}]

Author rebuttal:

Meta-review: This papers proposes to improve CLIP's fine-grained perception 1) hard-negative training on instruction-editing datasets using a symmetric hard negative contrastive loss, 2) long descriptive caption training that utilizes rotary positional encodings to capture rich semantic context missed by standard CLIP. Reviewers appreciate that this work addresses a clear limitation of CLIP (Kftn, mmoh), cleverly converts instruction editing datasets into hard negatives (RrCr, mmoh, Sser), and combines two complementary data sources (long-text and hard-negative) in a well-motivated way (mmoh). All reviewers recommend acceptance of the paper. After considering the reviews and rebuttal, AC agrees with the reviewers' assessment and recommends acceptance.
Decision: Accept (poster)