VisMin: Visual Minimal-Change Understanding
Authors: Rabiul Awal
Saba Ahmadi
Le Zhang
Aishwarya Agrawal
Venue: NeurIPS 2024 poster (2024)
Keywords: vision-language
fine-grained understanding
multimodal representation
Primary area: machine_vision

Abstract: Fine-grained understanding of objects, attributes, and relationships between objects is crucial for visual-language models (VLMs). To evaluate VLMs' fine-grained understanding, existing benchmarks primarily focus on evaluating VLMs' capability to distinguish between two very similar captions given an image. In this paper, our focus is on evaluating VLMs' capability to distinguish between two very similar images given a caption. To this end, we introduce a new, challenging benchmark termed Visual Minimal-Change Understanding (VisMin), which requires models to predict the correct image-caption match given two images and two captions. Importantly, the image pair (as well as the caption pair) contains minimal changes, i.e., between the two images (as well as between the two captions), only one aspect changes at a time from among the following possible types of changes: object, attribute, count, and spatial relation. These four types of minimal changes are specifically designed to test the models' understanding of objects, attributes of objects (such as color, material, shape), counts of objects, and spatial relationships between objects. To curate our benchmark, we built an automatic pipeline using large language models and diffusion models, followed by a rigorous 4-step verification process by human annotators. Empirical experiments reveal that current VLMs exhibit notable deficiencies in understanding spatial relationships and counting abilities. Furthermore, leveraging the automated nature of our data creation process, we generate a large-scale training dataset, which we use to finetune CLIP (a foundational VLM) and Idefics2 (a multimodal large language model). Our findings show that both these models benefit significantly from fine-tuning on this data, as evident by marked improvements in fine-grained understanding across a wide range of benchmarks. Additionally, such fine-tuning improves CLIP's general image-text alignment capabilities too. All resources including the benchmark, the training data, and the finetuned model checkpoints will be released.

VisMin: Visual Minimal-Change Understanding
Rabiul Awal" Saba Ahmadi"
Le Zhang'
Aishwarya Agrawal
Mila Quebec AI Institute
Université de Montréal
{rabiul awal, ll. .eang,agggg.. wwi.ggiissaaaaaaaaee ammr quebec
Abstract
Fine-grained understanding of objects, attributes, and relationships between objects is crucial for visual-language models (VLMs). To evaluate VLMs' fine-grained un- derstanding, existing benchmarks primarily focus on evaluating VLMs' capability to distinguish between two very similar captions given image. In this paper, our focus is on evaluating VLMs' capability to distinguish between two very similar images give caption. To this end, we introduce new, challenging benchmark termed Visual Minimal- Chninal-Change Understanding (VisMin), which requires models predict the correct image-caption match given two images and two captions. Importantly, the image pair (as well as the caption pair) contains minimal-changes, i.e., between the two images (as well between the two captions), only aspect changes at a time from among the following possible types of changes: object, attribute, count, and spatial relation. These four types of minimal-changes are specifically designed to test the models' understanding of objects, attributes of objects (such color, material, shape), counts of objects and spatial relationship between objects. To curate our benchmark, we built an automatic framework using large language models and diffusion models, followed by rigorous 4-step verifica- tion process by human annotators. Empirical experiments reveal that current VLMs exhibit notable deficiencies in understanding spatial relationships and counting abilities. Furthermore, leveraging the automated nature of our data creation process, we generate large-scale training dataset, which we use to finetune CLIP (a founda- tional VLM) and Idefics2 (a multimodal large language model). Our findings show that both these models benefit significantly from fine-tuning on this data, as evident by marked improvements in fine-grained understanding across wide range of benchmarks. Additionally, such fine-tuning improves CLIP's general image-text alignment capabilities too. We release all resources including the benchmark, the training data and the finetuned model checkpoints at httpps: /tte:s///iiiie net/
Introduction
Fine- inee-graind understanding of objects, attributes, and their relationships is critical for Visual- Language Models (VLMs) to generalize effectively to new, unseen scenes and compositions. Previous studies such as ARO 43 and Sugarcrepe M. highlighting the deficiencies of VLMs in this domain predominantly focus on understanding fine-graie fine-grained differences between two very similar captions a human- human-written written caption and automatically generated negative caption differs from the original caption only with respect to an object, or an attribute or relationship between two objects. While such hard-negative examples for captions can be syn- thesized using rule-based approaches, synthesizing such hard- nard-negative examples for images is very
mew.aiiviaaaattiioiiooooooooo where the hard-
"denotes equal contribution
IIn the context of contrastive learning, hard negative is a specific type of negative example that is particularly challenging distinguish from the positive example.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Object Change
Attribute Change
table with several dishes on it with a stereo.
toble wi th several dishes on it with a television.
man with block Man wi th green silver stripes stands at outdoor event.
white pol ko dots stards at outdoor event.
Count Change
Spatial Relation Change
One toy car and two teddy bears.
Three tay cars and two teddy bears.
hydrant and a trash can to the right in street.
trosh can a fire hydrant to the right in street.
Figure I: Overview of our VisMin benchmark. VisMin consists four types of minimal-changes object, attribute, count and spatial relation between two image-captions pairs. The evaluation task requires a model to predict the correct caption match given: 1) two images and caption, 2) two captions and one image.
challenging. Existing benchmarks presenting visual hard- negatives suffer from two main limitations: Limited Difficulty: In benchmarks such Winoground (36]. MMVP(3),, the original images and their hard-negative counterparts differ in multiple aspects (objects, attributes of objects, image background, etc.). This multiplicity limits the difficulty of the benchmark and makes challenging to precisely evaluate the models' fine-grained understanding of specific aspects. 2) Limited Com- plexity: Although benchmarks such as EQBEN [38). SPEC (28) have controlled hard sard-negatives the visual domain is limited to graphic engines, few video domains reliance purely synthetic images depicting simplistic scenes.
Motivated by these observations, we propose a new benchmark, Visual Minimal-Chame Under- standing (VisMin) built on top of the images from the COCO [22) dataset that consists of complex everyday scene images. VisMin is designed to measure VLMs' ability to comprehend minimal changes, i.e., changes with respect to only one aspect (see Fig. D.. from among the following aspects: object, attribule, count, and sputial relation, while keeping other aspects unchanged as much as possible.
The evaluation task for model is to predict the correct image-caption match given: 1) two images and one caption, 2) captions and one image. To curate Vis.inn we built an automated pipeline using large language models and diffusion models. To ensure the quality of our benchmark, the synthetic data generated using the automated pipeline undergoes rigorous 4-step verification process by human annotators, with data retained in the benchmark only if it passes all four steps. We meticulously designed the benchmark ensuring uniformity across various categories the extent possible. We conduct detailed analysis of our benchmark, which enables more transparent assessment of the various strengths and weaknesses of the models.
We conducted empirical tests on eight open-source VLMs, including foundational models like CLIP 31 and Multimodal Large Language Models (MLLMs) such as Llava (4) and Idefics2 We also evaluated two closed-source APIs, GPT-4 and Gemini. Our findings suggest that both foundational models and MILMs perform relatively well in understanding minimal changes in objects and attributes. Surprisingly, MILMs underperform foundational VLMs in object and attribute understanding! For spatial relation understanding, although MLLMS perform better than VLMs. both families of models perform below random chance! Similarly, both families of models show considerable room for improvement counting capabilities. Our results underscore the need for emphasis on spatial reasoning and counting understanding over attribute/object recognition in VLM evaluations. We anticipate that our benchmark will catalyze advancements these critical areas within the community.

---

Lastly, owing to the automated nature of our synthetic data creation process, we generated a large- scale (64,392 samples) minimal-change image-text data for fine-tuning the VLMs to enhance their fine-grained understanding. Fine-tuning CLIPP (a foundational VI.M) and Idefics2 (a MI.LM) on our minimal-change data, without any additional modifications to the model architecture or loss functions, results in significant improvements in fine-grained understanding across various benchmarks. Notably, such fine-tuning also enhances foundational VLMs' general image-texi alignment capabilities as evident by marked improvements in CLIP's image-text retrieval performance on COCO. These observations suggest that our minimal-change dataset can serve as model-agnostic, general-purpose resource to enhance the capabiliities of VLMs.
To summarize, our contributions are threefold: 1) A controlled and challenging benchmark. We introduce the VisMin benchmark, which challenges models to detect semantic differences between visually similar but semantically different images. Extensive testing on foundational VL.S and MILMSs reveals their difficulties with this task, highlighting areas for improvement. 2) A pipeline for automated data creation and benchmark development. We create an automated pipeline to generate visual minimal-change data at scale using large language models and diffusion models, with a rigorous four-step human verification system to ensure high data quality. 3) Enhancement of V LLMs' fine-grained understanding with fine-tuning the fine- teeiged understanding of CLIP and Idefics2 by fine-tuning them on our large-scale minimal- change image-text data, demonstrating improved image-text alignment and overall performance.
minimal-change data. We improve
2 Related work
Fine-grained understanding benchmarks: Most existing benchmarks focus on understanding fine-grained textual differences, such as VL-checklist (47), ARO [43]. and Sugarcrepe [8] Bench- marks presenting visual hard-negatives, such EQBEN [38] Winoground (36). ImageCode (I4), SPEC (28). either lack minimal changes or have limited visual complexity - -graphic engines, a few video domains or purely synthetic images depicting simplistic scenes. Our benchmark addresses these gaps by utilizing the advances in LLMs [10] and diffusion models [30] 20 21 to achieve minimal changes in complex COCO-like scenes without compromising the naturalness of the images, thus providing a more robust evaluation of fine-grained visual understanding in VLMs. Detailed
comparisons of benchmarks are provided in section 4
Automatic approach to generate visual hard negatives: Existing approaches to automatically generate visual hard negatives fall into three broad categories: (i) using nearby video frames with semantic changes [4] 38). (ii) using graphic engines [38]. (iii) using diffusion models (28 38 17].
Our proposed framework falls in the third category. DEMON[[7] is the closest to our work, creating training data using diffusion models to improve the learning of a given vision-language model. They use diffusion models perform local editing on the images given the target object mask. However, this approach requires attention masks from the vision-languse proposes a diffusion-bart based canvas-filling method for generating minimally-differeet image pairs limited to four types of minimal changes: size, position, count and existence. Compared to these existing methods, our automated pipeline to generate minimal-chang data more involved in order to achieve minimal-changee in complex scenes while maintaining the photo-realism of the scene and controlling changes across diverse categories. Our pipeline also has more a comprehensive automated filtering mechanism compared to previous pipelines that mainly rely on CLIP-based filtering.
model being studied. SPEC [28)
Enhancing fine-grained understanding in VLMs with hard negatives: Most methods to enhance fine-grained understanding in VLMS like CLIP focus on fine-tuning with caption-based hard negatives and optimizing loss functions to better use these signals 442
46 33 Common strategies for
generating textual hard negatives include hcuristic rules 433 language models [46] [5 scene-grapy graph information 7 33 and LLMs integrated with semantic segmentation O. In contrast, fewer works explore visual hard negatives; methods like NegCLIP 42) and General Scene Difference [16] rely on nearest-neighhor images, which often differ too much or too little in context, limiting fine-graine -raiined learning. Our approach is closest to SPEC (28) and CounterCurate [45), which also fine-tune VLMs using minimal-change visual hard negatives. Unlike SPEC, we and CounterCurate extend this to multimodal large language models, but our work evaluates performance on 10 out-of-distriiution benchmarks (compared to or 2 in SPEC and CounterCurate) and outperforms baseline models in most cases, demonstrating the strength of our approach (see Tables 3 and
3

---

3 Minimal-Change Image- Text Dataset Creation
We devised a framework to synthesize large-scale minimal-change data and introduce the VisMin benchmark (see overview fig. 2) The pipeline includes three stages: Minimal-Change Pairs Synthesis. where we minimally edit image and text pairs; Automatic Filtering, which verifies the faithfulness of the texts and synthesized images; and Human Verification, a four-step process to ensure that only data meeting all quality criteria is included. We will discuss each stage in detail.
Our dataset creation pipeline includes three stages: (i) Minimal-Chang Change Pairs Synthesis: We develop methods for synthesizing minimal -hangge image-caption pairs involving Objects & Attributes and Counting & Spatial Relations. (ii) Automatic Filtering: An LLM generates questions and answers based on captions, and a VQA model predicts answers from images. Synthetically generated tunimal-change data are excluded if answers don't ' match. (iti) Human Verification: Synthetically generated minimal-change data undergoes rigorous 4-steps human verification, and only examples passing all stages are included the benchmark.
Figure 2:
Stage 1:
Minimal- Minimal-Change Pairs Synthesis
Stage 2: Automatic Filtering
Object & Attribute Edit
Modil
Manch Buth??
Sermantic Model
Inseinting
LLM
Banke Dipalics
Chicps
Guesdluch la Ancoarch Par
Mitodihrrse vis!
Quveroe Amoower
Count & Spatial Relation Edit
Stage 3: Human Verification
Prompt
LLM
Loyout Difusioo
Naturalnuus
Beandtsp
Matching
go a
Clrnae Copalee
if sullie
2011,
instrction Verticatton
Raeppes flae of
CC
Amasen
Niximal Change
Syntheti
Mecohaniica
Orage
Diffusiinn
Image-Pact data
Tertanl Dat Vertizutton
Stage 1: Minimal- Changee Pairs Synthesis In the first stage of our pipeline, we focus on synthesizing minimal-change image-text pairs across four strategic categories: objects, attributes, counting, and spatial relations. These categories are specifically chosen to test various levels of visual-linguistic comprehension. We generate minimal-change text pairs using LLM and then generate minimal- change image pairs using diffusion models. Our synthesis process distinctly tailors the creation of image-text pairs to the specific needs of each category, depicted in Stage in Figure (Object & Attribute Edit and Count& & Spatial Relation Edit blocks).
LLM-guided Edit Instructions Generation To generate Himimal-channee text pairs, start with source captions and then prompt an LLM (Mistral 47B (10) to generate both the edit instructions specific to each edit category and the corresponding edited caption (see Appendix A.2. for the prompt used). For Object and Attribute edits, we use human-written captions from COCO [2 and VSR (23) datasets as source captions. The LLM processes these captions suggest edits targeting specific objects or attributes. For example, given the source caption "A dog in the middle of the couch", the LLM generates the edit instruction "change dog doll" which contains both the source phrase ("'og"") and the edited phrase ("doll"). The LLM also generates the edited caption "A doll in the middle of the couch". We generate five plausible (based on the criteria outlined for LLM prompting) edit instructions and edited captions per source caption. To ensure the edited captions are minimally changed w.r.t the source caption and contain visually plausible changes, we prompt the LLM again for filtering, removing 40% of the total LLM outputs that do not meet those criteria (see appendix A.2.2 for details on the criteria). For Counting and Spatial Relation edits, we generate the source captions synthetically due to the absence of suitable human- written captions dataset containing descriptions of counts and spatial relations of objects. We prompt the LLM to create captions and outline object layouts and bounding boxes. For instance, the LLM might generate a caption like "A plate of cookies on table and cup of coffee to its right," with the corresponding bounding boxes: ("plate of cookies"": [200, 300, 120, 80]: "cup of coffee": [320. 300, 60, 80) The I.M generates a large pool of such synthetic captions. The edit instructions and the corresponding edited captions are generated using rule-based method aimed at swapping the object positions for

---

spatial relation edits (e.g. edited caption: "A cup of coffee on a table and plate of cookies to its right", swapped bounding boxes: Istt crop": [200, 100, 80); "2nd crop": [320, 300, 60, 80)) or adjusting the object counts for counting edits (e.g. edited caption: "A cup of coffee on table"), removed bounding boxes: [2200,300, 120, 80]], in this example removing the plate of cookies from the image.
Diflusion-guided Image Synthesis We modify images according to the edit instructions generated by the LLM in the previous step. For Object and Attribute edits, we first mask the object to be edited in the source image using the Grounding- Prounding-DINO model [25]. We obtain the source images from the COCO dataset. The object to be edited is specified in the source phrase of the edit instruction (e.g.. "a dog" in the edit instruction "change dog to doll"). We then apply the SDXL inpainting model [30,, using the imput image, masked region, and edited phrase (obtained from the edit instruction, e.g.," "a doll" in the edit instruction "change dog to doll") to alter the masked image region to match the desired outcome, e.g., changing "a dog" to "a doll." For Counting and Spatial Relation edits, we create a synthetically generated source image dataset based on LLM--eggttt layouts from the previous step, using the LLM-grounded Diffusion (LMD) model [21] for image synthesis. To create an edited image, for the spatial relation edits, we first reposition the source image's 's bounding boxes using a rule-based method. We then obtain image crops from the source image corresponding to the objects which we need to reposition w.r.t each other. Lastly, we use the GLIGEN layout-diffin -diffusion model (20] to smoothly insert the obtained crops into the source image at the repositioned bounding box locations. For counting edits, we obtain the edited image by always removing one or multiple objects from the source image. The object to be removed is specified by masking and we the Lama model 34 to carry out the object removal. We employ layout-based diffusion models [21 20) instead of using end-to-end diffusion models like Stable diffusion [30 the layout-based model facilitates precise control over the object positions and counts and thus ensures the changes are faithful to the edit instruction well minimal. Unfortunately, end-to-end models such as Stable Diffusion are not good at precisely editing object positions and counts.
Stage 2: Automatic Filtering To ensure consistency of synthesized hard-negative images, we use a VQA-based filtering system, which is more effective than object detection (see Stage 2 in Figure 2). Questions generated by LLM [10] based on the edit instruction and caption (following TIFA [9] verify that edits align with the caption and that the positive caption no longer applies to the negative image. We use LLa Va 7B (24] to answer these questions, with region-specific questions for object/atrribue edits and global questions for background consistency. This process removes 75% of synthesized images, ensuring dataset quality.
Stage 3: Human Verification To ensure high-quality of the benchmark, on top of automated filtering, we conduct human verification using the Amazon Mechanical Turk platform. The images and captions undergo four steps of verification, requiring agreement from at least four out of five annotators at each step pass the human verification. The steps are: 1) Naturalness and Image-Text Matching Verification: Annotators assess if (a) the image looks natural, (b) the caption is sensical, and (c) the image matches the caption. Only 26% of synthetic images pass this step, mainly due to the criterion (a), where counting and spatial relation images often look unnatural. See Appendix Table for detailed acceptance rates. 2) Visual Edit Verification: Annotators assess if the images faithfully reflect the specified minimal edits without additional changes, with an acceptance rate of 80%. 3) Edit Instruction Verification: Annotators assess if the LLM-generated edit instructions are minimal, i.e., the suggested edit modifies only one aspect of the sentence (out of object, attribute, counting, or spatial relation), with an 84% acceptance rate. 4) Textual Edit Verification: Annotators assess if the edited sentence faithfully reflects the specified minimal edits without additional changes, with a 95% acceptance rate. Annotators also verify the LLM's categorization of the types of edits (object, attribute, counting, or spatial relation). These steps ensure precise, minimal changes in images and captions, delivering high-quality benchmark for fine-grained visual understanding. See Appendix 1.4.1 for annotator instructions.
4 Training and Benchmark sets
In our study. we create training and benchmark sets to improve and assess fine -re-graided understanding in VLMs. The training data is generated through a scalable pipeline with automatic filtering, while the benchmark data undergoes additional rigorous human verification to ensure high quality (as explained above). For object and attribute edit types, that make use of natural images, the train- ing data is sourced from VSR (images sourced from COCO) and the COCO 2017 training split
5

---

(118K images), while the benchmark data is sourced from the COCO 2017 validation split (5K images). This ensures benchmark images are unseen during training, maintaining evaluation re- liability by community standards. The Training dataset has 64,392 samples (37,017 objects, 10, attributes, 10,050 counting, 6,973 relations), while the VisMin bench- mark has 2,084 samples (579 objects, 294 attributes, 589 counting, 622 relations). We aimed for a balanced benchmark across categories. However, the number of attribute samples in VisMin is relatively low be- cause the LLM suggested attribute edits for only 2000 samples in the COCO 5K validation set. Moreover, most of these suggested edits were color edits. So we further downsampled the color edit instances to balance the distribution of the types of attribute edits. Figure 3 shows subcategories of the changes in VisMin For detailed training subcategories, see Appendix 15 For qualitative samples, refer to Appendix 13 and 14
right 177
melatoon 622
left 200
counting
object 379
In table D we compare VisMin with related benchmarks. Vi- sual Minimal HN: This criterion evaluates if the visual hard negatives contain minimal changes. In Winoground and MMVP, hard negatives differ across multiple aspects (object, attribute, background, etc.). In contrast, VisMin's 's hard negatives vary only in one aspect, keeping others unchanged as much as possi- ble. This minimal-change property is also present in What's EQBEN, SPEC, and in a subset of images from ImageCoDe and CounterCurate. Visual Com- plexity: This criterion assesses the complexity of visual scenes. ImageCoDe and EQBEN mainly feature images from limited video domains and graphic engines, while What'sUp uses simple house- hold and tabletop images. SPEC generates simplistic scenes using diffusion models. In contrast, Winoground uses expert-curated Getty Images, and MMVP uses ImageNet and LAIONAesther ics. VisMin and CounterCurate (concurrent work) stand out by using diverse, complex everyday scenes from COCO [22 and Flicker30K Entities 22 featuring common objects in natural contexts.
72
Color
Figure 3: VisMin categories and subcategories.
Textual Complexity: Bench- marks like ImageCoDe,
Table I: Comparison of benchmarks offering visual hard negatives (HN) across: @ minimal HN, P visual complexity, textual complexity, aapman-approved captions ( and images and size. v: cri- terion holds for subset of the benchmark.
Winoground, and MMVP use free-form human-written cap- tions. In contrast, What's Up (focused on spatial changes) and SPEC (focused on controlled changes) use captions,
21 2
Wainta do W
Dosurhonk Sallnng
Quen
1232
Lobire rolen Aoman Cryple right
Now (ver (Hoarry Treskey
3,00
template-based
HUN Comme Wobbe (0m))
Air Peodeomm/eens 12M1 Sonpar
45,000
COCOL Systerio Apaaton
which often lack diversity.
EQBEN and CounterCurate mix free-form (human or LLM- ALM-generated) and template-based based captions. VisMin combines human-writeer and LLM-generated free-form captions, , yielding sufficient textual complexity to the benchmark. Human Verification: For benchmarks using synthetic images like EQBEN, SPEC, CounterCurate, and VisMin, human evaluation is essential to ensure images look natural. It's also crucial for benchmarks with automatically generated hard negative captions, as these may be nonsensical unless well-defined templates like What's are used. Nonsensical captions make it easier for VLMs to identify them as incorrect [ VisMin is notably the only benchmark with full human verification, ensuring both captions and images are high quality. CounterCurate also conducts human verification but only checks for image-caption consistency (on 300 examples), without verifying image naturalness caption sensibility. Size: This criterion assesses the dataset's 's size. VisMin excels by combining controlled minimal changes with complex, natural scenes and captions, providing optimal balance for robust evaluation.
5 Benchmarking VLMs on VisMin Benchmark
Setup We have comprehensively benchmarked existing state-of-the-art VLMs on VisMin, en- compassing both foundational VLMss-uch as CLIP (31), SigLip 44,, BLIP [8) and Coca 400 and generative MILMMs including Ilava 24.. Idefies2 [15] and InternVL.L.. 2 Additionally, closed-source MLLMs such as GPT4-o || and Geminil.0 Pro 33 are also evaluated.

---

For foundational models like CLIP. we conducted an image-text matching task using cosine similarity, following [36]. The tasks involved two settings: choosing the correct image from two captions and selecting the correct caption from two images. In VisMin examples (see Fig. [ with pairs ((T1,)) (72, C2)]. the text score is 1 if (s(Co. To) > s(C,, To)) A (s(C1, h1)
> s(Co, [1)), and the image score is 1 if (s(Co. To) > s(oo, I.)) A (s(C1, I1) s(C,, (o)): the group score is 1 when both scores are I. For MLLMs, we adapted these tasks to a visual question answering format with binary questions about the matching relationship between images and captions {(4), CI), (12, C2)). To calculate the text score, we presented the model with one image and two captions, using the prompt " "Does this image depict: [C1 C2/?" To calculate the image score, we presented the with two images and one caption, using the prompt "Which image better aligns with the description: '/CJ'? The first the second image?". The score is 1 if the predicted answer matches the ground truth. Once both scores are obtained, the group score is if both individual scores are 1.
Table 2: Performance of foundational variants and MLLMs across categories on the VisMin Dataset. Columns "I," 'T.' and 'G" denote Image, Text, and Group scores from Winoground [36.. AVG denotes the average across columns. The best results are highlighted in bold.
Object
Attribute
S. Relation
Count
AVG
Random Chance
16.67
16:67
25
16.67
25
25
16.67
22.22
MTuk Human
86.87
95,50
83.07
82,31
91.15
76.87
81.67
92.76
76,20
88.96
96.77
86.41
86.54
CLIP (VTT-B/32) [1
79.62
77.89
67.53
72.11
65.99
55.1
8.2
4.34
0.48
31.24
20.71
10.53
41.15
CLIP (NST-B/16)
86.53
79.1
71.68
70.75
65.31
$2.38
8.84
3.22
0.8
34.3
22.58
13.58
42.42
CLIP (V/T-L/14)
87.56
83.59
78.07
74.49
69.73
57.82
9.16
4.66
1.45
37.01
30.56
18.17
46.42
NegCLIP [+]
87.74
87.05
80.66
81.63
80.27
71.77
10.13
4.66
1.13
55.01
57.72
42.28
48.96
Sigl-ip (VIT
90.5
88.95
83.25
86.05
79.25
73.13
11.58
6.43
1.77
60.95
47.03
38.37
55.61
Sigl.ip.(WTTT///11
93.44
88.43
84.46
84.35
78.23
68.37
10.29
4.82
1.29
61.8
$7.05
44.14
56.39
Flava [32
81.69
75.12
66.66
74.49
60.54
52.04
6.75
5.79
0.96
35.99
27.5
15.45
41.91
BLIPP 18]
92.4
92.57
87.05
88.44
86.73
78.57
11.25
4.98
2.09
52.97
46.01
33.28
56.36
Coca (40
84.97
81,52
73.58
78.57
66,33
57.82
11.25
5.95
1,77
60.1
35.82
28.52
48.85
LlaVal
93.0
32.8
32.2
92.2
34.4
33.3
91.8
7.8
7.4
73.6
25.0
20.2
38,28
Idefies2 (8B) [5]
95.4
69.4
67.6
89.1
71.4
67.0
18.6
18.8
4.8
72.2
50.6
47.0
55,99
CogVLM (7B) 39
94.64
89.63
87.56
11
88.43
81.63
21.70
11.09
2.25
58, 40
6.45
3.56
52.87
IntemVL1..5 (25B)
2
94.65
40.24
39.72
91.16
42.86
41.16
74.28
14.79
11.74
73.51
31.58
27.5
48.60
Geminil.0 0 Pro
94.99
79.97
78.76
91.84
74.83
72:45
52.57
15,43
9.81
67.74
44.14
37.52
49.63
GPT4-o
95.51
96.2
93.44
92,18
90.48
87.07
89.07
50.48
46.78
77.42
78.27
68.42
73.93
Results Insights from Table highlight key capabilities and limitations of current models. Text scores generally surpass image scores, especially in MLLMs, where text scores are often two to three times higher. In contrast, foundational VLMs show a modest discrepancy between image and text scores. We hypothesize that for MLLMs, the image score is lower compared to the text score because they lack training with multiple images, and simple vertical concatenation does not provide sufficient visual signals, leading to suboptimal alignment with captions. Notably, Idefics2, which supports multi-image processing, performs similarly on text and image scores, underscoring the importance of multi-image data during pretraining- Foundational VLMs' higher text scores suggest that distinguishing between captions is easier than between images, highlighting the need for our visual minimal change benchmark. Interestingly, foundational VLMs generally outperform MLLMs due to the latter's 's lower image scores.
All models perform well on Object and Attribute splits, indicating that understanding semantic changes correlates strongly with recognition capabilities. Models excelling in image classification tend to perform better, reflecting foundational understanding that does not require advanced reason- ing. For instance, Idefics2, using the SigLip (ViT-L/16) vision encoder, performs worse with strong LLMs compared to its foundational VLM counterpart, likely due limited multi-image understand- ing in MILMs. Notably, VLM shows superior performance over other MLLMs in understanding object and attribute changes. This advantage likely stems from its integration of grounding tasks and nigh-quality human-amm annotated datasets like Flickr30K Entities (2).. RefCOCO [12] and Vi- sualGenome (13) in its pretraining. On the other hand, the Spatial Relation split relies heavily on reasoning capabilities, with MLLMs outperforming foundational models. This suggests that LLMs can parse object relationships through reasoning. However, existing VLMs struggle with spatial relations, often scoring below random chance, indicating potential biases in models and highlighting an area for further research.
For single-image models, such as Llava, we combine the two images vertically with a clear border.
7

---

We document human baseline performance on our benchmark via Amazon Mechanical Turk (see Appendix A.4.2). Humans generally outperform models on image scores, except in the attribute category where GPT4-o excels. Models typically surpass humans in text scores, especially with attributes and objects. However, in spatial relations and counting, humans significantly outperform models in group scores, highlighting areas for model improvement and the robustness of human scene comprchension.
Enhancing fine-grained understanding in VLMs
We use a synthetic minimal-change dataset to enhance fine- tee-graiied understanding through additional fine-tuning of VLMs. Training with pairs of images and captions with minimal differences provides richer training signal, improving model performance in fine-grained understanding tasks. We demonstrate improvements on top of both foundational VLMs and MLLMS by conducting extensive evaluations across various benchmarks: (1) Single image benchmarks test the alignment between single images and multiple captions: VSR (23), CountBench [26). VALSE [27]. SPEC [28], and
Sugarcrepe 8 (2) Multiple image benchmarks test the alignment between multiple images and
captions: ImageCode [4], .MVP [37. Whatsup [Л]. Winoground [36], EQBEN (38, and our
VisMin benchmark.
6.1 Fine-tuning Foundational VLMs
Enhancement with Minimal-Change Data Our approach uses a synthetic minimal-change dataset to improve visual representation without altering the training methodology. We construct training batches with both source and edited image-text pairs: In the original CLIP training. a mini-batch B = [(C11 1, (C2, I2)..... (Cn, [,),, with pairs randomly sampled from the dataset as random negatives. With minimal-change data, we add edited image-text pairs hard negatives, resulting in B = {(11, T1 (C1, 7, (C2, 12), (C), T2 .) where (C%, J, is the edited pair of (Ca, Ix). We use a total batch size of 128 with 4 A 100 GPUs and retain other training protocols and hyperparameters as default from OpenCIIIP B. including a learning rate of le-05, weight decay of 0.2, Adam 8n of 0.9, 32 of 0.98, an eps of le-06, and a cosine scheduler. The training runs for epochs, and we select checkpoints based on separate VisMin validation set.
We fine-tuned pre-trained CLIP our minimal-change data, calling it VisMin-CLIP. For comparison, we implemented three models using the same pre-trained CLIP: NegCLIP 42), CounterCurate- CLIP 45). and SPEC-CLIP (28). In NegCLIP, we fine-tuned CLIP with automatically gener- ated hard negative captions and nearest- nearest-neighhor images paired with human- captions. For we used hard-negative data (attribute, position, counting) but trained one model on all three types, unlike the original approach of training separate models. SPEC-CLIP was fine- tuned on six combined gategory-speciffi splits (size, spatial, existence, count). Hard-negatives were included in all batch constructions, with excess negatives rolled into the next batch if needed. All models used ViT-L/14 as the backbone and the original CLIP loss, initialized from OpenAl check- points. Best checkpoints were selected from validation sets for NegCLIP and CounterCurate-CCn and from average benchmark performance for SPEC-CLIP This controlled comparison evaluates the impact of different hard- negativee data approaches improving CLIP's ' fine- fine-grained understanding.
CounterCurate-CI.
Table 3: Performance of fine-tume lne-tunndd CLIP and Idetics2 across categories on the VisMin Datasct. The symbol indicates the reproduced model checkpoints based on their respective training data.
Object
Attribute
S. Relation
Count
AVG
T
G
I
T
G
CLIP((VT-LL11)
87.56
83.59
78.07
74.49
69.73
57.82
9.16
4.66
1.45
37.01
30.56
18.17
46.02
NegCLIP"
87.74
87.05
80.66
81.63
80.27
71.77
10.13
4.66
1.13
55.01
57.72
42.28
55.00
CounterCurate CLIIpi
89.81
91.02
84.46
82.99
80.27
72.79
20.1
11.41
7.4
49.24
45.16
31.92
55.54
SPEC-CLIP
86.53
86:01
78.58
78.57
71.77
63.95
9.16
5.31
45.5
47.71
32.43
50.55
VisMin-CLIP
91.54
91.19
86.36
85.03
83.67
75.85
11.9
3.38
1.29
82.34
79.97
72.33
63.74
Idefies2
95.4
69.4
67.6
89.1
71.4
67.0
18.6
18.8
4.8
72.2
50.6
47.0
55.99
VisMin Idefic22
96.5
95.7
93.3
83.0
76.0
85.4
87.8
80..
86.43
Results We evaluated these models on the VisMin benchmark (results in Table 3). Fine-tuning with minimal-change data significantly improves CLIP's performance on the Object, Attribute, and Count categories, demonstrating the usefulness of our minimal-change data in enhancing the fine-grained
8

---

Table 4: Evaluation on other single and mult-image visual fine-grained understanding benchmarks. All models adopt VIT-L-14 as the vision encoder. CB refers CountBench, SG refers Sugarcrepe, IC refers to Imagecode. 12T and T2I indicate standard standard image-to-text and text-to-image retrieval metrics. Best-performing models in the CLIP CLIP-family are highlighted in blue, and best perforrerninng MLLM models are highlighted in green.
Samples
SENGLE VALSE
MALTI-DMGGE
Whatopp
SPBC
MBVV
Winggrond
EQREN
ViaMia
127
G
0
69.1
12.7
17.81
92:0
1.5
13.57
41:13
NecCLIP
400
75.41
37:23
[67.33
55.63
57.42
SPEC
32.00
75
87.04
7,75
4.7
30.71
52.7
ViaMia CIPP
65k
28.00
72.34
$1.44
43.99
44.23
35.98
05.81
33.99
32.75
14.35
9.75
56.29
10.71
33.57
60.7
64.55
$8.96
91.11
88.91
69.04
14.37
64.4
33.75
22.5
51.56
understanding of foundational VLMs such as CLIP. VisMin-CLIP consistently outperforms NegCLIP, CounterCurate-CC CLIP and SPEC-CLIP across all categories except spatial relations. This suggests that the visual minimal-change data is more helpful in improving the fine-grained understanding capabilities of the CLIP model compared to the nearest neighbor images in NegCLIP and not fully minimally-changed CounterCurate and SPEC data.
We further conduct zero-shot evaluation of the fine-tuned CLIP models on other fine-grained understanding benchmarks (beyond our VisMin benchmark) to test their generalization capabilities (see Table VisMin-CLL. performs best in 11 out of 18 tasks, while NegCLIP, CounterCurate CLII. CLIP, and SPEC-CLIP lead in 3, 1. and 3 tasks, respectively. All models outperform the pre-trained CLIP model across benchmarks. For counting and spatial reasoning tasks, VisMin training data shows significant improvements. On CountBench, we observed 9%, 19%, and 17% gains over NegCLIP, CounterCurate-CLa and SPEC-CLIP, respectively. Similarly, spatial reasoning benchmarks (SPEC, Whatsup, VSR), VisMin-CLL (LIi showed an average 7.79% improvement over NegCLIP and 5.21% over CounterCurate- CLIP. While SPEC-CLIP performed best on the in-distribution SPEC benchmark, VisMin- still outperformed other models, indicating its effectiveness in improving fine-grained understanding. For VALSE and SugarCrepe, NegCLIP performed best, likely due to similarities in the textual hard-negative generation process used in those benchmarks and NegCLIP's fine-tuning data.
Furthermore, our minimal-change data significantly outperforms others in multi-image understanding benchmarks. Fine-tuning on this data enhances the model's ability to distinguish between similar images, showing improvements on challenging benchmarks like Winoground, MMVP, and EQBEN, which test compositional reasoning and fine- ane-grained understanding. VisMin-CLIL improved Text scores by 6% on Winoground and 18% on EQBEN over baseline CLIP, demonstrating effective alignment of visual and textual feature spaces. VisMin-CLIP also outperforms other models on most tasks and achieves comparable results on remaining benchmarks (except SPEC), despite using fewer samples (e.g., K5 in VisMin vs. 637 K in SPEC).
Figure 4: VisMin fine-tuned models show greater im- provements with larger models. The circle radius reflects the number of model parameters.
Figure 5: (left) Recall results with ViT-L/14 on COCO benchmark (right) standard benchmark re- sults on Idefics2.
CLI Vieit DoaMin-Ul CUP
CL VIS
135
Tus:20
VYNU
PORE
80
62
TO
20
55
CUP
MoOCUIF VISW-CUUP
idefo2
Additional Findings Further experiments reveal several key findings: (1) Scalability: As illustrated fig. 4 we evaluated varying sizes of OpenAl's CLIP models- B/32 and L/16. Larger models demonstrated improved performance across both single-image and multi-image benchmarks after training on our synthetie data. This improvement is likely because understanding minimal changes is complex task, demanding robust model capabilities. For instance, the smallest tested model,

---

ViT-B/32 (149.62M parameters), exhibited improvements of 2.37 and 3.24 in single and multiple image benchmarks, respectively, when comparing VisMin- CLIP against the baseline CLIP. When the model's s capacity was expanded to ViT-L/14 (427.62M parameters), the improvements increased to 6.88 and 9.21, respectively. These results highlight the scalability and efficacy of our data in enhancing model performance. (2) Enhanced Original Capabilities: In addition to improvements in fine-grained understanding tasks, training on our data also enhances performance in standard retrieval tasks, as shown in fig. 5 This suggests that models achieve better alignment from training minimal change tasks, indicating that our data is generally applicable across various cross-modal tasks.
6.2 Fine-tuning Multimodal Large Language Models (MLLMs)
We utilized Idefies2 [1]
improve fine-grained understanding, employing our instruction-formaatedd dataset. Given its proficiency in multimodal interactions and advanced multi-image processing, Idefics2 was chosen for its open-source accessibilityy model size and leading zero-shot performance.
Dataset and QLoRa Fine-tuning Our dataset, VisMin Instruct-IT, includes image-text pairs created using rule-based approach (see A.2.3 for details). We reformulated these pairs for MLLMs, where the task is to select the correct image from two options based on a given caption or choose the appropriate caption for image from two possibilities. While the base Idefics2 model was trained with a variable number of images in a sequence, we limited it to two images to include one positive and one hard negative example from VisMin. We fine-tuned the Idefics2-8B model using the QLoRa technique [4], updating adapters in the language model and modality connector including perceiver resampler with A100 80GB GPU. We used 4-bit quantization, with 64 and = 16 for LoRa, and a learning rate of le 5. The model was fine-tuned for one epoch with an accumulated batch size of 64.
Results The fine-tuned Idefics2 model shows significant improvement on VisMin (see Table 3 across all categories, comparable to GPT4-o (see T. demonstrating the effectiveness of our minimal-change data in enhancing MLLM fine-grained understanding. Notably, in the Spatial Relation category, gains of 64.4%. 57.2%. and 64.5% were observed for Text, Image, and Group, respectively, unlike CLIP, where fine-tuning did not improve spatial understanding. Fine-tuning Idefics2 also transfers well to other fine-grained benchmarks, achieving over 5% overall improvement (see Table[4) 4). To assess generalization, we evaluated its zero-shot performance on non-fine- -raiigg tasks (MMMU [11 and POPE [9]; results in Fig. 5 (right)). The model maintained comparable performance on POPE but dropped on MMMU, likely due to the binary-choice format in our fine- tuning data. This suggests further gains could be made by combining our data with instruction-tuning datasets, though this wasn't 't pursued due to the GPU demands of fine-tuning an 8B model with additional data.
In addition to our main results, we provide further analysis in the Appendix A.3 including detailed evaluations on additional multimodal benchmarks, zero-shot image classification task, and video understanding tasks.
7 Conclusion and Limitations
We present VisMin benchmark for evaluating fine-grained visual understanding in VLMs such as CLIP, SIgL.P, LLaVA, and Idefics2. While these models perform well in object and attribute recognition, they struggle with counting and spatial relationships. To address this, we fine-tuned CLIP and Idefics2 on our minimal-chan -change dataset, yielding significant improvements in objects, attributes, and counting. For spatial relations, CLIP showed limited gains, while Idefies2 made notable progress. Fine-tuning also enhanced CLIP's image-text alignment, demonstrated in COCO retrieval tasks, underscoring our dataset's 's value training resource for VLMs. Limitations: Despite automatic filtering, the dataset contains noise, including image deformations and text-image mismatches due to diffusion model limitations. Future diffusion model advancements should improve minimal-change editing. Additionally, our model may inherit social biases from the base models (CLIP/Idefics2), as no specific mitigation measures were applied during fine-tuning. Our use of uniform prompts for evaluation may have influenced performance variably.
10

---

Acknowledgments
We express our gratitude to Mojtaba Faramarzi and Yash Goyal for their constructive feedback throughout the different stages of the project. We also acknowledge the valuable feedback provided by Qian Yang. Kanish Jain, and Osear Manas on the early draft. The technical support extended by the Mila IDT team in managing the computational infrastructure is greatly appreciated. Additionally, Aishwarya Agrawal received support from the Canada CIFAR Al Chair award throughout this project.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Zhe Chen, Jianman Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic assual-lingusstin tasks. arXiv preprint arXiv:2312.1 14238888,. 2023.
[3] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818-2829, 2023.
[4] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuming of quantized lims. Advances in Neural Information Processing Systems, 36, 2024.
[5] Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky, Teaching structured vision & language concepts to vision & language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2657-2668. 2023.
[6] Sivan Doveh, Assaf Arbelle. Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante- Bonilla, Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, al. Dense and aligned captions (dac) promote compositional reasoning in vl models. Advances in Neural Information Processing Systems, 36, 2024.
[7] Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, and Amir Globerson. Incorporating structured representations into pretrained vision & language models using scene graphs, 2023.
[8] Cheng- Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sug- arcrepe: Fixing hackable benchmarks for vasion-language compositionality. arXiv preprint arXiv:2306. 10, 2023.
[9] Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf. Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303. 1897, 2023.
[10) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint arXiv:2401.044 04088, 2024.
[11] Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's "up" with vision-language models? investigating their struggle with spatial reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9161-9175, Singapore, December 2023. Association for Computational Linguistics. doi: 2023. maimlppmmin.560
URL
org/
[12] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects photographs of natural scenes. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
11

---

[13] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[14]
Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy, Image retrieval from contextual descriptions. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 3426-3440, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18665//12022..cloooo..... URL
https://aclarthologyy org acl.onn....onn 241
[15] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelies: An open web-scale filtered dataset of interleaved image-text documents. Advances Neural Information Processing Systems, 36, 2024.
[16] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint 2xXiv:2306.05425 2023.
[17] Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang. Wei Ji, Wenqiao Zhang, Tat seng Chua, Siliang Tang, and Yueting Zhuang. Fine-tuning multimodal LLMs to follow zero-shot demonstrative instructions. In The Twelfth International Conference on Learning Representations, 2024. URL httpss://ppeereeeeee
[18] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified Dision-language understanding and generation. In International conference on machine learning, pages 12888-12900. PMLR, 2022.
[19] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- laniuae models. arXiv preprint arXiv:2305.103: 2023.
[20] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. ArXiv, 0bs/2301.07099, 2023. URL https://api. semanticscholar www.oollaarrrrcooppuuue Crg/Corps 255942528
[21] Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Lim-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305. 13655, 2023.
[22] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th Europeun Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt 13, pages 740-755. Springer, 2014.
[23] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguisties, 11:635-651 2023.
[24] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[25] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre- pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023.
[26] Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten, 2023.
[27] Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gau. VALSE: task-independent benchmark for vision and language models centered on linguistic phenomena. In Proceedings of the 60th Annual Meeting of the Association for
12

---

Computational Linguistics ( (Volume 1: Long Papers), pages 8253-8280, Dublin, Ireland. May 2022. Association for Computational Linguistics. doi: 10. 00.18665.1/2022..ulll-nn.g.. URL
httpp:://aclanthology wwg.oot2............ 2022. acl long. 557
[28] Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. Synthesize, diagnose, and optimize: Towards fine-grained vision-language understanding. 2023.
[29] Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to cesioontoo-hhenn to-prasee correspondences for richer image-to-sentence models. IJC,, 123(1):74-93, 2017.
[30] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 1748-87762 PMLR, 2021.
[32] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- sion, pages 15638-15650, 2022.
[33] Harman Singh, Pengchuan Zhang, Qifan Wang, Mengjiao Wang, Wenhan Xiong, Jingfei Du, and Yu Chen. Coarse-to corsee-tofinns fine contrastive learning in mmage-text-grapeee space for improved vision-language compositionality, 2023.
[34] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Vietor Lem- pitsky. Resolution-robust large mask inpainting with fourier convolutions. arXiv preprint 10X1v:2109.07161, 2021.
[35] Gemini Team. Gemini: A family of highly capable multimodal models, 2024.
[36] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5238-5248, June 2022.
[37] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal Ilms, 2024.
[38] Tan Wang, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Equivariant similarity for vision-language foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1 11998-12008 October 2023.
[39] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, al. Cogylm: Visual expert for pretrained language models. arXiv preprint arXi::2311.03079, 2023.
[40] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint 20X1::2055.01917, 2022.
[41] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311. 16502, 2023.
[42] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2022.
13

---

[43] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what do about it? In The Eleventh International Conference on Learning Represeniations, 2023. URL https
//openreview.o wwt.:/nee//e.eee wwtp:/eeeeeettotndttteeee oo
[44] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 11975-11986, 2023.
[45] Jianrui Zhang, Mu Cai, Tengyang Xie, and Yong Jae Lee. Countercurate: Enhancing physical and semantic visio-linguistic compositional reasoning via counterfactual examples. arXiv preprint arXiv:2402. 13254, 2024.
[46] Le Zhang, Rabiul Awal, and Aishwarya Agrawal. Contrasting intra-modal and ranking cross- modal hard negatives enhance lisio-linguistic fine- tne-grained understanding. arXiv preprint 2tt::23000088322 2023.
[47] Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. An explainable toolbox for evaluating pre-trained vision-language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 30-37, Abu Dhabi, UAE, December 2022. Association for Computational Linguisties. URL https ptttps://aclantholoogg. org/ 2022. emnlp demos 4
14

---

A
Appendix
A.1
Societal Impact
Our work shares the broader positive societal implications of vision- vision-language research, particularly improving the accuracy of vision-language models (VLMs). These advancements could benefit various applications, such as helping visually impaired individuals better interpret their surroundings, creating interactive learning tools for children, and enabling more effective interaction with in-home robots. However, like most technologies, accurate vision-language systems also present potential risks. For instance, they could be misused to extract sensitive information from public CCTV footage inadvertently leak personal data, such credit card information, when assisting visually impaired users. To address these concerns, robust privacy safeguards are essential to mitigate these risks.
A.2 Prompt Templates
A.2.1 Prompts Used for Edit Instruction Generation
We utilize the Mixtral 47B LLM to generate edit suggestions across selected categories: objects, attributes, counting, and spatial relations. The specific prompts used for each category are detailed in the Boxes below. These prompts guide the LLM in creating minimal, precise modifications to captions, ensuring the generation of high-quality hard-negative instances.
In-context demonstrations for objects edit suggestions.
As AI language model, your task involves making minimal, targeted edits to a caption that describes an image, guiding corresponding visual changes to be made in the edited version of the image. Follow these structured steps:
Suggest Edit: Identify and suggest specific edit from the given caption. Your edit should only change an object in the scene.
*The edit must adhere the following criteria:
Object changes should be visually distinct and mutually exclusive. not replace an object with its synonyms or closely related sub- sub-categories. The replacement object must fit within the original object's region and visually plausible within the scene. Maintain a one-to-one replacement ratio; do not introduce additional objects.
General Criteria for Edits: Avoid any changes related to action, spatial relations, counting, the size of objects. Edits must be visually and contextually plausible, ensuring the scene remains coherent and the edit does not introduce inconsistencies in other parts of the image. Create an Edited Caption: Draft a new caption reflecting the image post-edit.
Specify Edit Category: Clearly state that your suggested edit falls under 'Object'.
In-context demonstrations for attributes edit suggestions.
As an AI language model, your task involves making minimal, targeted edits to caption that describes an image, guiding corresponding visual changes in the edited version of the image. Follow these structured steps:
Suggest Edit: Identify and suggest a specific edit from the given caption if it is editable. Your edit should modify attribute of existing objects in the caption.
*The edit must adhere the following criteria: Change only the attributes of an object from one of the allowed sub-category: color, pattern, shape, appearance, material, state, condition. Replacing object itself or introducing new attributes is not allored. Attribute
15

---

changes must be distinct, mutually exclusive, and not synonymous. The edited attribute should make visual and contextual sense within the image without altering its overall composition.
General Criteria for Edits: Do not suggest attribute changes related spatial relations, counting, the size of objects. Edits must be visually and contextually plausible, ensuring the scene remains coherent and the edit does not introduce inconsistencies. Create an Edited Caption: Draft a new caption reflecting the image post-edit.
- Specify Edit Category: Clearly state that your suggested edit falls under 'Attribute'. If you cannot change an attribute due to a missing attribute the given caption, please output 'EMPTY'.
In-context demonstrations for counting image description generation.
*Task Instruction Counting*
Your goal
suggest textual description for creating visual scene within a 512x512 pixel canvas, focusing specifically on counting task involving objects numbered between two and nine. Follow these steps to ensure comprehensive output:
**Input**: Propose a scene with groups of distinct objects, ensuring each group contains different number of items ranging from two to nine. These objects should be easily countable and distinguishable. The objects should be spread out across the canvas; cluttering or overlapping is prohibited. Detail the arrangement of these objects within the scene, considering their visibility for accurate counting. Do not mention any activity in the scene.
2. **Bounding Boxes and Object Counts**: For each group of objects, provide bounding box data in the format 'object name', [x, y, width, height]. This data should reflect the object's position and size within the 512x512 image, aiding in identifying the exact number of objects present.
3. **Background Prompt**: Suggest a fitting and simple background that does not detract from the main task counting the objects.
4. **Negative Prompt**: List elements that should be avoided in the image ensure clarity and focus on the counting task.
Identify the most relevant category and sub-category for the scene, based on the objects and their arrangement. Choose categories that enhance the clarity and focus of counting task.
5. **Category (sub-catogory)
In-context demonstrations for relation image deseription generation.
*Task Instruction Relation*
Your goal is suggest textual description and structured data for creating a visual scene within a 512x512 pixel canvas. Follow these steps to ensure comprehensive output:
1. Input: Suggest description with pairs of distinct objects that scene. These objects could be swapped their positions if needed later on. Avoid suggesting common background objects in this step. You may occasionally mention the relative size, distance and orientation between objects if applicable.
commonly occur together
Bounding Boxes and Object Pairs: Detail each identified object with its bounding box in the format 'object name', [x, y, width, height], indicating the object's position and size within the 512x512 image. Be specific about the spatial relationship of objects, such as whether they are on the 'top', 'down', 'above', 'below', 'left', or 'right' of each
16

---

other.
3. Background Prompt: Based on the scene described in the input, suggest fitting and straightforward background.
4.
Negative Prompt: List elements to be excluded from the image.
5. Category (sub-category): Identify the most relevant category and sub- cub-category for the scene based on the objects and their arrangement Limit the options for sub-categories based on the provided reference examples.
A.2.2 Prompts Used for Edit Instruction Verification
We employ the Mixtral47B LLM to verify edit suggestions in the object and attribute categories. The specific prompts used for this verification process are illustrated the Box [A.222] Verification is crucial because initial suggestions by the L.M can occasionally be implausible or violate the intended edit category. By thoroughly verifying the edits, we ensure the generation of accurate and high-quality hard-negative instances, which are essential for effectively testing and improving VLMs.
In-context demonstrations for LLM suggested edit verification object attrihute category.
**Task Instruction Object**
Determine the acceptability of edits to an image caption based on following criteria:
Edits must only involve object changes within a specified scene region. No changes related to attributes, spatial relations, counting, any unrelated alterations.
The new object must be visually distinct and not closely related to the original. Edits should be confined to one region without needing adjustments elsewhere for scene consistency.
Adhere to a one-to-one replacement rule, introducing no additional objects. Reject edits that add unnecessary specificity make assumptions not clear from the original caption.
Reject if information is insufficient uncertain. Strict filtering: When in doubt, reject the edit.
**Task Instruction Attribute** Evaluate edits to image caption that change object's attributes based on following criteria:
Reject changes involving the objects themselves, their spatial relations, counting, or size.
Acceptable modifications are distinct changes in color, pattern, shape, texture, material, state, condition. Attribute changes must be distinct and not synonymous.
Reject edits suggesting synonym replacements or adding new attributes. Reject if information is insufficient or uncertain. Strict filtering: When in doubt, reject the edit.
A.2.3 VisMin Instruct-It Dataset
In the VisMin Instruction dataset, a sample consists of a source image-caption pair (To, Co) and its minimally edited version (T1 C1). To construct the VisMin Instruct-ITT dataset, we create four instruction samples from each VisMin sample:
Two samples for the Image- Task (selecting the best matching image given a caption):
1. Given Co, select between To and I
2. Given C1, select between To and Ij
Two samples for the Text-Task (selecting the best matching caption given an image):
17

---

1. Given To, select between Co and C1
2. Given 4, select between Co and C1
Please refer to Table in the rebuttal PDF for the exact set of instruction templates used. We randomly sample one template for each task type when creating the instruction samples.
In total, we create 65k instruction samples for training. Additionally, we include 16k randomly se- lected image-caption pairs (either (To: Co) or (71, C1) to retain the model's general image captioning ability.
Table 5: VisMin Instruct-It dataset creation templates
Template Type
Template Examples
"You are given two images. Which one better aligns with the deseription: [caption|? ? The first or the second image""
Image-Tack
"Question: You given images. Which one better aligns with the description? (caption) Choices: First, Second."
"Based on the descriptio:: [ssption]. please select one of the given options that best marches the image. Choices: First, Second."
"Question: Does this image depiet: (A) (candidase text
(B) [cundidate text B Choices: B."
Text-Task
this image best represent: (A) (candidate A1, or (B) (cundidate text R)?"
"Which of the Collowing best matches the image: (A) (candidate text A), or (B) (candidate text B)r"
A.3 Additional Results
We report additional results to demonstrate the impact of fine-tuning on our dataset across different aspects of scene understanding. We evaluated our fine-tuned models, VisMin-CLID and VisMin- Idefics2, on various tasks designed to assess distinct elements of multimodal comprehension.
Table 6: Performance breakdown of EQBEN video splits for both CLIP fine-tuned and Idefics2 models. The table also includes results additional multimodal benchmarks for Idefics2 family of models and ImageNet results for various CLIP models. We leave CLIP results on the additional multimodal benchmarks and the Idefics2 results on ImageNet blank due to the form-foratt not being suitable for ovaluation.
Eo-
EQ-GEBC
EQ-AG
Additional Muttinndal Benchmarks
ImageNert
G
T
MathVista MMBench
TeuVDA
VQAv2
top5
C1IPENTT-L/14
50.00
65.00
40.00
15.00
10.00
0.50
15,00
25.00
10.00
75.53
04.50
NecCLIP
65.00
60.00
50.00
20.00
20.00
0.00
15,00
15.00
10.00
65,50
90.00
ViMMio-CLIP
68.20
Mediie
84.62
53.85
25.00
15,00
47.37
53.00
73.40
78.82
VidMin-ldefies2
35.00
15.00
71.89
79.75
We assessed performance of multimodal LLMs (Idefics2 and Vismin- Idefics2) benchmarks like Vattaista, MMBench, TextVQA, and VQAv2 to understand how fine-tuning on our data impacts tasks requiring both general scene understanding and specialized skills (e.g., OCR and math) (see 6 6 middle block) [ Results show slight declines specialized tasks MMathVista, MMBench, VeA but notable improvements on VQAv2 that tests for general scene understanding, supporting our hypothesis that our dataset strengthens general scene comprehension while not losing performance much on more specialized tasks.
Multimodal Benchmarks:
Zero-Shot Image Classification: To evaluate the robustness of learned representations of the contrastive models, we included evaluation on zero-shot image classification on ImageNet (see Table 6 last block). Both NegCLIP and VisMin-CLe underperform compared to the base CLIP model. However, VisMin-CL.IP shows a smaller performance drop compared to NegCL.IP, indicating that it retains stronger generalization capability even after fine-tuning.
Performance on Video Understanding Tasks: Finally, we tested if fine-tuning the models on our minimal-change dataset also improves their performance video understanding tasks since video understanding tasks require identifying subtle frame-to-frame variations (see Fig. [o some examples). To test this, we evaluated the models on the EQBEN (3 benchmark, particularly the following splits: EQ- YouCook2, which focuses on cooking procedures from instructional videos; EQ-GEBC, which identifies event boundaries in kinetic videos; and EQ-AG, which captures changes between objects and their pairwise relationships during actions.
'Contrastive models with two-tower encodings are not suitable for these benchmarks, as these benchmarks require generating text outputs rather than just measuring similarity between encoded images and texts.
18

---

We would like to note that our fine-tuned models excel at discerning subtle semantic differences, but neighboring video frames often exhibit low-level rather than semantic changes. Abrupt changes between frames, like a bus turning into car, a shirt changing color, or objects swapping positions, are rare. Additionally, our dataset does not cover action changes, which are typical in videos. Therefore, we expect finetuning our dataset to result in limited improvements for nearby video frames.
This confirmed in our results (see Table 6 first block), where fine-tuning CLIP and Idefies2 showed significant improvements on EQ-Yout Yo-Yo22 (in group scores) due to its higher presence of object-based changes, while performance remained similar on EQ-GBEC and EQ-AG, which mostly feature action shifts. Thus, our dataset is most effective for enhancing video understanding in scenarios involving subtle semantic adjustments rather than dynamic actions.
EQ-YouCook2
squid rings
add ground pepper.. the same bow!
green orion
slices
size
EQ-AG
person holding bag which front of him/har.
contacting bag which on himyher.
holding clathes The the clathes
which on Nim/her. the person
Figure 6: Examples from different video splits from the EQBEN benchmark.
A.4
Annotation Platform and Benchmark Quality
A.4.1 Human Verification of Benchmark
The human verification process involves four main steps. Naturalness and Image-Texs Text Matching Verification: in this step, humans evaluate the data samples using the following three criteria: a) whether the image looks natural or not, b) whether the caption sounds sensical or not, and c) whether the image matches the caption not, addressing limitations of automatic filtering and maintaining robustness against manipulation X (see Figure 7p. Only 26% of synthetic images pass this step, highlighting the need for human verification. The low acceptance rate is mainly associated with the high rejection rate criterion a, where most counting and spatial relation synthetic images do not appear natural. Please refer Appendix Table for detailed acceptance rates for each criterion. Please note that we aimed for balanced benchmark. To address the higher rejection rates for the categories of counting and spatial relation, we began with larger number of samples in these categories. 2) Visual Edit Verification confirms that the edited images accurately reflect the specified edits without additional changes, ensuring visual minimal change, with an acceptance rate of 80% (see Figure (). 3) Edit Instruction Verification checks that LLM-generatee edit instructions are minimal and targeted, altering only one aspect of the object, attribute, counting, or spatial relation in captions and images, with acceptance rate of 84%. (see Figure 9). 4) Textual Edit Verification ensures the edited sentence accurately reflects the specified edit instruction without additional changes and classifies the change type, with an acceptance rate of 95% (see Figure 10).
A.4.2 Human baseline
We gather human annotations using Amazon Mechanical Turk (AMT) to determine human perfor- mance on our benchmark, replicating the exact tasks given to models as described in Section 5 Additionally, annotators can choose "none" or "both" options (please see Figure and Figure 12): we deliberately included these options to accurately estimate the best model performance when tasks are difficult for humans. We collect five annotations per sample for each of the four scenarios in the image-text matching task. We compute image score, text score, and group score described in
19

---

Table Acceptance for all criteria across different categories.
Criterion
Object
Attribute
Counting
Relation
Overall
Step 1-a: Naturalness of Image
64%
80%
41%
22%
37%
Step 1-b: Sensicality of Caption
84%
79%
75%
70%
74%
Step 1-c: Image-Tex Segge-Texttte Matching
83%
89%
52%
65%
65%
Step 2: Visual Edit Verification
81 31%
79%
81%
78%
80%
Step 3: Edit Instruction Verification
81%
86%
85%
84%
84%
Step 4: Textual Edit Verification
95%
94%
94%
95%
95%
Step 4: Automatic Categorization Verification
95%
91%
99%
100%
97%
Section 5 the answer "both" or "none" is chosen by three or more annotators, or if one annotator chooses "none" "both" while the other options each receive exactly votes, we randomly assign a 50% match to maintain consistency with the model evaluation setup.
20

---

IT seow and truuppre cocercl mayn war aooer
coenst netarble.
etservel the wor see secor photogent tn
Floe liariig
ressaer griees
Covs ree
ween reatyy werke when evaluding
un tonur
wnst
tiee oijecta
with menddr cvcopt
Please
proided
with cavarpore exparinng ohyy ouch nusigattvt masuus
Mexunal leeking
leeling
Seggen bonng cenring and
Pl
Merapticss
inoeeet euce condod ieees troiiatoon
dasciption
decription larga song and bar
Cevemt resebatee made
/bcami deurgpaa Sevead voeaan
Cereet deserative mean
Carve dese balee Severd neree
ferd
derai bocc plaas cvris.
Hease vnoer de Aidic toehe bast yourssas lnavaadg condeess work rejeced mali you caineal attenbi sndraa pour comilaci
Figure 7: Instructions for naturalness and image-text matching verification.
21

---

Indoutt
BAlnstreokes
money
pasiies sandy o Cabess wabu Euieedh
cwneec chease sdkct eatrpot
mestraleee Charpe
Figure 8: Instructions for visual edit verification after applying edit instructions.
22

---

Your
extrcdor naractio
dng poope nabo paddy
daaly
hooe "anciic
flowet
dhang Vinklee b changgnng lik hereerage shingging teeee
opense moshees only sasee sersence chieer Iicm the oeconnn congolee onl destwer waltore make sngirn wation
sheapt
Cound eq racibing caust catarin time doguto doges
cac))
cnu wen sbeees
dwn vali aude spmiis utp vere
Plosu
can mamertore Chamce mnsoee
selected pestion i sercdiin pleas welcct
cancapis
inooiee dha assscc sen div calgel es olpacs atriuss ardnng and wnoo. bas doer with spec cangor abact atricue counding specal
ancutiion valdr
case selecsel ner te above purtion, please speciry ty the wdd resuciio please select applst
diken mu escien concren
meve
Original serdance proup riding kicycles aring sients pash
esciss concapt
medifcation
spacieee obart
Ohiples
piciure at
insemmtieee Churne wiik brawe
aspect
redeiccaion tys does nor wite the our spac ded campones obact asio counting spedaa onahi.
monembbrre clarner people
far concapis
resat
mroolbe mnre tha one aipect ce 201 cargoo es oppcc atrouce craniin and unio.
bys dees withe fou spec camponas por.ert obact uci counting spedaal
thres
rntuution valdr
care you belereee har for abers scstion ofrace speccty te wike sepotio please select anple me benter eonepe
mroolee aspact
caagos countng wrio.
Original sertance Soverr vase or the sable and
soleoaed nai" redeaaeev sedies uffore spendy
juk whent werto
le concept
momber more the one stove servece carage ebbon, anir arding and
oboect
Origanaa aerannes bide puylutty dancing sugelier
inseeeniee Charge cleeninn
wirde uree. modi fosior tyse doee for tat withe for spor dest oarpaee obant aato coerinng spenda
Pleas vnever de quericn below besl www. vmaeadge Cardees wore rejccad make yor caneau ataenicc coraaa
Figure 9: Instructions for edit instruction verification.
23

---

tolruction
the iil le slo un vnstroooo thatspeliies peoiicc neriiio made fut deice culf todi aolket. You will be slewe editdt you tascie eveaakrre zorestress of editdd ceriooeeeeenc adbeet pertence sjatpes to cenecl it
axmunntly sot ace spactedd ade munuctive
other changes legonr
baing spacthed
ieflwiing
Objece redilynng siupect shung ny cuf,
Attibaai: realivig the urua of an oleon suuch eo calor, pettons ape. mato kul a- changhh roand loulo vota estagao xuldd sropt
Counting mediying the zount stjud johanging
Spartal Relatormiigg castying merapalal
changiing g cat deg tog
cac))
Roaemple i:
Original sertance A track and auitee in viding in tapen that in deet
insomuttion Chunge sak
baak and shtingg sablr
selested Ner lor above pestion, pleace specity the seneence notic glease
likes cocaati ralluct soll spus fed the mutruciion dot
changes beyorad sovctied tnrernaatios
Findlyy sulos the of the ouil boir apes devd it intruction
Ciject
white midiii
benals.
diess aocunat refies sgic des the wdil estruction
solect dedd in intruction:
O Utjest
Abrocce
O Spacial melk denelii
Example
insoumtion Change booka
Ediind ranmaace:: red clain vithout's digg rieeping on aithoatt anartty.
ta eduas surcu conoec
Ven Kn
case you for aboce purston, please spacity way serbence not glnase
sain stew arrng boy spadlied in uctions
of
Ciject
Dou
Peir
Ecample
Original sertonce A to lift decce
Cal momeeciroe Charge bike to left door" to the
bufitel centemee door the
coneect
Van
caie you wole deel het for sbove pretion dieece speckly senteca an ced gouass
aocunnny refire opee
sutro changes beggond spcci indin anrtutaoo
Rinelyy eieer the of the will beirg sper ded in instuntionn
Abtocc
Pleae votur the helic whthe hasd of Innssdddee Contecc repered Thase you www.//aeeeeeee atmeic vidra valade comiieli
Figure 10: Instructions for textual edit verification.

---

Instructions
desortatin along images. Your believe both images accuasely depict the descripton, please select both However, you find that nother image aocurasely depicts doscr poon, diooss none.
image that accuratety depicts desoristin you
Plenne san the ecampples below to underspand the tank betber:
Examplee
Description: A plaze that has three donuts it next to
Ric vmage
SPOCE roge
Which imagets) accunsely depicts the desol plio??
First imag
image
Bolt
Example
Description: ball lying of toy car.
Witch deccrption?
Seczord image
O None
Extungle
Description: kitchen refrigerator.
Pir inaage
Seroni moiD
Witchh image(s) arzuneedy depicisc pheeciptius?
O Fire image
image
Bol's
Exampple
Description Two giraftes standing by bartte wire fence.
image(s) secunntly depictic peoeciptiur??
image
Second image
decalled
answer questions ofyour knowledge repectdd you your voluutle
Figure 11: Instructions for choosing the best matching image.
25

---

Instructions
will shown with cleecripkars, Your select descriprion End acoursddy belleve both descriptinns accurately desorfoe select However, belleve nether desortation image,
Please soe extmples bolow
Description that laying down taptop.
Description
Whtch desziplienj) waznssaly desctiees the imrege?
Dlsuziption
Desoriptin
Both
Description sheep grass
Description One sheep by in the grass in sield.
pelon()) accuranty image?
Descliption 12
Nare
Ecample
young
Description 2: younng man drassed in jacket disccs ahirt and akinny tie.
desorittion)) accuratety
Desortatin
Desortation
None
Example
Description 1: skatmbnards
Description 2: surfboards lying on rug
Which descliptints) accuratcly describes the image?
Descriptio
Descriptin
Reth
Nome
For a decalled ed look at the imtoe,, please hover over it
Plesse answer the quastiona below best your krowledgee Curdless wark will be repedted Thank you carelul abent detal and velustde busicnl
Figure 12: Instructions for choosing the best matching description.
26

---

A.5 Random qualitative samples from training and VisMin benchmark
In this section, we present random qualitative samples from both the training set and the VisMin benchmark to illustrate the quality and diversity of data used in our experiments. Specifically, we provide examples across object, attribute, relation, and counting categories highlight the four strategic categories represented in our dataset.
Object
oiling mootrryglle
diee iegoretea
pasad
parkod notoryyleeen pereeeettt
ported
parads
whone
grwus tast.
apen tack
mow mnweneetton
can lece explnnd.
weodee tadima
meodle boand
Malrosed
sodatoond.
owralaptop.
Aboc drrwniigg intedidantta glaw.
Abos full of rasching uncott denato with withigane
Amonge with geen rpee saslong wasses.
gry nur uith opas snd leng whikkes
Counting
rwelage rechvoloo
wellagegion selneeks
contansngaak salire
coctaining rableee
soono watber
wicker
dr backit..
maci harkets.
VidMin Banchmark- Relation
K
Iysbur
A bad abows doe.
dogiaa abow tall.
dddiza boasd sight
A.Suze snda <kllig
ney
wata sragg Beitgg
with bood
Figure 13: Random qualitative samples from VisMin.
27

---

The enerrwen prace.
Theeothe
bont
The fedagtt
Jawe..
keat bates brable.
Triining Antibute
AR PROSGAA
APP Cop. glactille
Ae matty bund
Arma boncke
Ameareei wit dertes Laknng splezas
mase kandeles nitg spiten
cxtled martdd pren
Apernsd prpporet
perend regdeptns
Training:
Thee coobinng shal..
Four coeking shat.
bottlen
Acilectas colletes coanrggee
gits
Them www.welleeeeeeee.....oo
Two wwdicennennggggg......o......
Toupdder
Okeylla bolly
Training:
and ligep loo cottarg
Ateddy bed e sy
anddy abon
beconeed taviiat.
dooe zs da cure
AGrES Oe disence
Aiciiioee loaast to z9 stse
Ante locand to alcelss
gattr paaso
paaaoce.taa gafas
botton.
matersi.
Figure 14: Random qualitative samples from the training set.
28

---

Distribution Categories Subcategories Training
food 5066
animal
vehicle 2982
object 37,017
9310
idsor
1325
relation
1311
ttrtbute
10. 10.352
6973
Mbve Km
Color
122
Figure 15: Categories and subcategories distribution in the training split.
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions, For each question in the checklist:
You should answer [Yes], , [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers
integral part of your paper submission. They visible the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked While "[Yes] " is generally preferable to "[No] ", it perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars expensive" or "we were unable to find the license for the dataset we used"). ) In general, answering "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer is often more nuanced, so please just use your best, judgment and write justification elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
the checklist one of the factors in their evaluation.
reported because it would be too computationally
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist".
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
29

---

Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Yes, the abstract is summary of the content of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In the last paragraph of Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic upproximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: No theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
30

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide details of training and inference in section 6.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments. a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will release the data and code and benchmark once accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cd
for more details.
31

---

While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: yes, provided in section and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We simply benchmark existing available models, with greedy decoding for MLLMS e.g. Idefics2 and fixing seed for foundation models e.g. CLIP.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma emorr bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
32

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification:
We perform the task with one gpu, as it's simple benchmarking and finetuning.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, we do.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: No relevant.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymmett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
33

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No relevant.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We follow protocol.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide comprehensive analysis of our new benchmark and will release
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
34

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We provide in Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not relevant.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
35


Reviews:
[{"review_id": "a8F1OdRjqw", "reviewer": "Reviewer_Lkr8", "summary": "This paper introduces a new benchmark, VisMin, which mainly challenges models to detect semantic differences between visually similar but semantically different images. It uses an automated data curation pipeline and human verification to create dataset. The authors benchmark the dataset with current VLMs and MLLMs, showing that the proposed dataset can help improve image-text alignment and overall performance.", "questions": "1. I'd like to see evaluation results of more latest models, with more evaluation metrics.\n2. I am also interested in seeing the authors provide justification for the finetuning results on the CLIP/Idefics2 models, since they are not consistently strong.", "limitations": "The authors use editing models to modify original images. I suggest that the authors carefully review the curated dataset to ensure there are no ethical issues.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of the paper makes sense and the paper writing is clear.\n2. The curated dataset contains hard negative samples that will be beneficial to improve fine-grained understanding of current models. The dataset curation pipeline is well designed and the data quality looks quite good.\n3. It indicates huge performance on CLIP and Idefics2 after finetuning with the proposed dataset.", "weaknesses": "1. The paper only uses T, I, G to evaluate the performance of different models on the proposed dataset, which is insufficient to show the ability of understanding attribute, count and spatial relation. I think some basic metrics like accuracy are also needed. \n2. The models being benchmarked are not sufficient. It lacks some latest or popular models, like Flava, CogVLM, etc.\n3. Another concern is that the finetuned CLIP/Idefics2 models do not demonstrate improvements in all aspects, indicating that the benchmark may introduce some side effects. I am curious about the underlying reasons for this and recommend that the authors propose a new baseline method alongside the benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Some basic metrics like accuracy are also needed.**\n\nThe T (Text), I (Image), and G (Group) metrics we use *indeed measure accuracy*, i.e., the proportion of examples for which the model produces the correct output [1]. The correctness criteria is different for each of T, I, G, as stated in L244-255. To obtain the aggregate T, I, G scores for the entire test set, we average the per-sample T, I, G scores across all samples in the test set. Thus, the aggregate T, I, G scores represent the accuracies on the test set. We will clarify this in the camera-ready version.\n\nMoreover, these T, I, G metrics we use are standard metrics for evaluating fine-grained understanding, originally proposed in Winoground [2] and adopted in subsequent research efforts such as EQBEN [3] and SPEC [4].\n\n[1] Goodfellow et al. Deep Learning, Chapter 5. MIT Press 2016.\n\n[2] Thrush et al. Winoground. CVPR 2022.\n\n[3] Wang et al. Equivariant similarity for vision-language foundation models. ICCV 2023.\n\n[4] Peng et al. Synthesize, diagnose, and optimize. CVPR 2024.\n\n**The models being benchmarked are not sufficient.**\n\nWe disagree with this point. Compared to the models suggested by the reviewer, such as Flava (2022) and CogVLM (2023), our selection *already* includes contemporary or more up-to-date models models like LLaVa1.6 (2023), Idefics2 (2024), InternVL1.5 (2023), GPT4V Turbo (2023), and Gemini1.5 Pro (2024). We chose representative models from various families – foundation VLMs and MLLMs, open-source and closed-source, covering 9 different models: CLIP (2021), SigLip (2023), BLIP (2022), Coca (2022), LLaVa1.6 (2023), Idefics2 (2024), InternVL1.5 (2023), GPT4V Turbo (2023 - closed source), and Gemini1.5 Pro (2024 - closed source). We believe our evaluation is extensive and up-to-date.\n\nAs per R1’s request, we benchmarked Flava and CogVLM (results in Table 1 of the rebuttal PDF). CogVLM outperforms other MLLMs in understanding object and attribute changes, likely due to the incorporation of grounding tasks and high-quality human-annotated datasets like Flickr30K Entities [5], RefCOCO [6], and VisualGenome [7] in its pretraining. Flava performs similarly to other foundational VLMs, except for spatial relations where both FLAVA and BLIP outperform significantly. This is likely due to the incorporation of COCO [8]  in the pretraining datasets of these two models, which contains a significantly higher proportion of spatial reasoning data (10.74%), compared to web-scraped datasets such Conceptual Captions (5.72%) and LAION (2.32%) used to train other models benchmarked in the paper (see Figure 1 (right) in the PDF). To obtain the percentages, we compute the proportion of captions in each dataset that contain at least one of the following spatial relation words: left, right, top, bottom, below, above, under (these are all the spatial relation words in the VisMin spatial relation split).\n\n[5] Plummer et al. Flickr30K Entities. IJCV 2017.\n\n\n[6] Yu et al. Modeling Context in Referring Expressions. ECCV 2016.\n\n\n[7] Krishna et al. Visual Genome. IJCV 2017.\n\n\n[8] Lin et al. Microsoft COCO. ECCV 2014.\n\n\n**Reasons behind finetuned models not demonstrating improvements in all aspects.**\n\nWe believe the reviewer is referring to the cases (listed below) where our finetuned models do not outperform the baselines. We *have already provided* the underlying reasons for each of these cases in the review version (RV), but we explain them again below:\n*  For spatial relation tasks in the IID setting (Table 3 in RV), we noted a slight performance degradation for CLIP upon fine-tuning on our minimal change data (L308-310). This is likely due to CLIP's contrastive loss design which often overlooks spatial prepositions, as also discussed in [9].\n* For OOD settings (Table 4 in RV), VisMin-CLIP *consistently outperforms the base CLIP* model but does not outperform NegCLIP on Sugarcrepe, Valse and ImageCode. For Sugarcrepe and Valse, this is likely due to the NegCLIP’s method of generating hard negatives, which uses linguistic variations similar to these benchmarks, making the evaluation more favorable for NegCLIP (L317-318). For ImageCode, this is likely because ImageCode requires understanding various challenging phenomena (L325-326), such as temporal markers, actions, negation, and visibility/occlusion (source: Table 2 of the ImageCode paper [10]) which our minimal change dataset does not cover.\n\nAdditionally, for Idefics2, although finetuning on our data demonstrates overall improvements in both OOD and IID settings, there is a 2.8% decline in the performance on the VALSE benchmark. This is likely because VALSE tests for specific kinds of skills such as pronominal coreference resolution and understanding actions that our minimal change data does not cover.\n\nWe would appreciate it if the reviewer could share their thoughts on these justifications we provided above.\n\n[9] Kamath et al. What's \"up\" with vision-language models?. EMNLP 2023.\n\n\n[10] Krojer et al. Image Retrieval from Contextual Descriptions. ACL 2022.\n\n**“propose a new baseline method alongside the benchmark.”**\n\nOur paper aims to benchmark existing models and investigate if fine-tuning on minimally changed data alone can boost performance, *without introducing new methods or losses*. We propose a *model-agnostic* recipe to improve fine-grained understanding capabilities of both foundational VLMs and multimodal LLMs, which we believe would be of sufficient value to the community.\n\n**Potential ethical issues in the dataset**\n\nSince we only make minimal edits (object, attribute, counting, spatial relations) on top of published publicly available datasets such as COCO, we do not anticipate our edited images to have any data privacy, copyright, and consent issues following the definitions in the NeurIPS Ethics Guidelines. See more details in the comments below.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h7uSaa6SMc", "reviewer": "Reviewer_a22a", "summary": "This paper studies the fine-grained understanding of objects, attributes, and relationships between objects for VLMs and MLLMs. Specifically, it focuses on the capability of VLMs and MLLMs to distinguish between two very similar images given a caption. Firstly, by leveraging an image generation diffusion model and an LLM, this paper introduces a new benchmark termed Visual Minimal-Change Understanding, which requires models to predict the correct image-caption match given two images and two captions. Secondly, it performs evaluations on multiple VLMs and MLLMs and analyzes the performance in terms of different types of changes: object, attribute, count, and spatial relation. Thirdly, it fine-tunes existing VLMs and MLLMs on the generated datasets and shows improvement in fine-grained understanding.", "questions": "- What is the motivation behind designing text scores and image scores for MLLMs? In Table 2, there is a significant drop in performance from text scores to image scores, especially for MLLMs. This drop might be because most MLLMs do not undergo instruction fine-tuning with two images, potentially affecting their ability to perform this task correctly. Under such circumstances, if the performance on image scores is low (for example, in spatial relation tasks), it becomes unclear whether the issue is the model's inability to process information from two images or its inability to detect minor changes in spatial relationships. \n- Visual minimal changes occur frequently in videos. Do you have any experience in this direction, such as how this dataset and the fine-tuned model could benefit video understanding? (If it's discussed in the paper, could you point it out? I may have missed it.)\n- Will the model and dataset be released? Also, VisMin Instruct-IT dataset. \n- (Minor comment) GPT-4V has different versions; it would be great to specify which version was used in the paper.", "limitations": "- The authors discuss limitations and potential solutions in Section 7, such as the possibility of noisy data being included in the proposed dataset. The reviewer suggests conducting human evaluations on some samples from the dataset to provide statistics on the noisy data and to offer visualizations of these noisy data samples. \n- The social impact should be discussed in this paper. This paper propose fine-tuned VLMs and MLLMs which are able to detect visual minimal-change and can have potential social impact.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The motivation is clear, and the problem addressed in this paper is well-explained. \n- The proposed dataset is interesting and benefits the research of fine-grained understanding of foundational VLMs. \n- This paper is well-written and easy to follow. \n- The evaluation protocol and the methods for fine-tuning are technically sound. \n- The analysis of VLM and MLLM performance on the proposed benchmark is interesting.", "weaknesses": "- Some notations are confusing, and implementation details for the evaluation are missing: how are two captions chosen in text scores for VLMs? It appears from line 247 that captions chosen from a single image are paired instead of being randomly sampled. What's the difference between C_0 and C_1 in lines 247 and 248? The notation here is a bit confusing. Do the notations C in line 247 and T in line 294 both refer to the captions for the images? What's the difference? \n\n\n- Implementation details for fine-tuning are missing: How do the authors construct the VisMin Instruct-IT dataset? In line 344, some rule-based approaches are applied, but the reviewer cannot find details of these rule-based methods. It would be great to show some examples of how VisMin is converted to VisMin Instruct-IT. It also appears that the authors fine-tune the model using QALora and instruction fine-tuning techniques. How many instruction-image pairs are in this dataset? \n\n\n- Insufficient dataset analysis: The authors mention in this paper that after human filtering of the proposed dataset, there is still some noise, such as deformation and mismatches. Conducting a human evaluation on the proposed dataset to determine the percentage of these noise data would be great. This would provide a valuable reference for future works using this dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Some notations are confusing**\n\nWe apologize for this. The captions C_0 and C_1 are minimal-change pairs as described in Fig 1 of the review version. The C in L247 is exactly the same as T in L294. We will clarify these and make the notations consistent in the camera-ready version.\n\n**Details about Instruct-IT dataset**\n\nIn the VisMin dataset, a sample consists of a source image-caption pair (I0,C0) and its minimally edited version (I1, C1). To construct the VisMin Instruct-IT dataset, we create four instruction samples from each sample in VisMin: \n* two samples for the Image-Task (selecting the best matching image given a caption): 1) given C0, select between I0, I1, 2) given C1, select between I0, I1\n* two samples for the Text-Task (selecting the best matching caption given an image): 1) given I0, select between C0, C1, 2) given I1, select between C0, C1\n\nPlease refer to Table 3 in the rebuttal PDF for the exact set of instruction templates we used. We randomly sample one template for each task type when creating the instruction samples.\n\nIn total, we create 65k instruction samples for training. Additionally, we include 16k randomly selected image-caption pairs (this could be either I0,C0 or I1,C1) to retain the general image captioning ability of the model.\n\nWe will clarify these in the camera-ready version.\n\n**Possibility of noisy data being included in the proposed dataset, even after human filtering.**\n\nWe believe the reviewer is referring to L376-378. These lines refer to our training set, not the human-filtered benchmark set. We will clarify this in the camera-ready version.\n\n**Human evaluations and statistics on the noisy data.**\n\nWe did conduct a comprehensive four-stage human evaluation on the *entire* benchmark set as described in L174-188 of the review version (RV). The benchmark set is obtained after *filtering out the samples* that do not pass any of the four stages of human evaluation. We refer the reviewer to the RV for the evaluation criteria in each stage and the associated acceptance rates. All stages except the first stage have high acceptances rates (80-95%) indicating a low level of noise in our automatically generated data. For the first stage, the acceptance rate is 26%, due to a high rate of rejection (63%) of images that don’t look natural. We want to avoid unnatural looking images in the benchmark set to prevent models from easily recognizing such unnatural samples [1]. However, such noisy samples are acceptable in the training set as the goal is to teach minimal change understanding to the models. This is further validated by our experimental results where we see that finetuning on our noisy data improves model performance for both IID and OOD benchmarks.\n\n[1] Hsieh et al. SugarCrepe. NeurIPS Datasets and Benchmarks Track 2023.\n\n**Visualizations of these noisy data samples.**\n\nWe have provided random samples from both training and benchmark splits in Fig 16 and 17 in the supplementary material.\n\n**Motivation behind designing text scores and image scores for MLLMs**\n\nText and image scores are needed to disentangle a model’s capability of “distinguishing between two captions given an image” from “distinguishing between two images given a caption”. We use the same metric for both foundation VLMs and MLLMs to ensure **consistent** evaluation.\n\n**Drop from text scores to image scores in MLLMs due to lack of instruction fine-tuning with two images.**\n\nWe agree with the concern about image scores and for this very reason, we chose Idefics2 for our finetuning experiments as it is capable of processing multiple images (it’s trained with multiple images). GPT4, InternVL and Gemini are also capable of processing multiple images. Thus, low image scores for these models suggest an inability to detect minimal changes. Also, if the low image score for a model was solely due to its inability to process multiple images, the model should perform poorly in all categories (Object, Attribute, S.Relation, Count), however we observe particularly low scores on some categories such as S.Relation and Count.\n\n**How could VisMin benefit video understanding**\n\nOur finetuned models excel in discerning semantic differences, but neighboring video frames often exhibit *low-level* rather than semantic changes. Abrupt changes between consecutive frames are rare, such as a school bus changing to a car, a shirt changing color, the number of people changing or the objects swapping their positions. Moreover, our dataset does not cover action changes which are very common in videos. Therefore, we expect finetuning on our dataset to result in limited improvements for nearby video frames. We quantified the extent of improvement by reporting model performance on three video subsets of the EQBen dataset that were created by sampling nearby frames (see Fig 1 of the rebuttal PDF for some examples). Results are presented in Table 2 of the rebuttal PDF. For both CLIP and Idefics2, finetuning on our minimal change data leads to significant improvements on EQ-YouCook2 (group score), and similar performance on the other two splits (group score). From our qualitative inspection of the samples, we find that EQ-YouCook2 has more significant object changes compared to other splits (which mainly involve action changes). Thus our dataset can benefit video understanding when it has the kinds of changes that our dataset covers.\n\n\n**Will the model and dataset be released?**\n\nYes we will release all model weights, generated dataset as well as generation code.\n\n**GPT-4V version**\n\nWe use GPT-4-turbo-2024-04-09.\n\n**The social impact should be discussed in this paper**\n\nThanks, we will include this in the camera-ready version of the paper. We provide a brief discussion of this in the comments below.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7CYnmX4Rwt", "reviewer": "Reviewer_dNLV", "summary": "The paper introduces VisMin, a new benchmark for assessing fine-grained understanding in VLMs. VisMin evaluates the ability of models to distinguish between minimally different images given a caption, testing recognition of changes in objects, attributes, counts, and spatial relationships. The benchmark is created using an automated pipeline and human verification. Empirical results show that current VLMs have notable deficiencies in spatial relationships and counting. Fine-tuning models like CLIP and Idefics2 with a large-scale dataset generated by this pipeline leads to improvements in fine-grained understanding and image-text alignment.", "questions": "Please refer to Weaknesses. \nThere are also some typo like L76: can -> can, L169: he -> the, Duplicate citation [39], [40], Table 3 and 4: Vismin -> VisMin.", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.This paper introduces VisMin, a benchmark that challenges models to detect subtle semantic differences between similar images, revealing significant shortcomings in VLMs and MLLMs.\n2.This paper uses minimal-change image-text data to finetune VLM and MLLM models, improving the fine-grained understanding capabilities.\n3.Table 2 illustrates that current open-source MLLMs perform poorly in fine-grained image understanding. This paper suggests using minimal changes in images to enhance fine-grained image understanding, and I believe this idea is promising.", "weaknesses": "1.After fine-tuning on VisMin data, MLLMs showed improved performance on fine-grained understanding benchmarks. However, their performance on POPE and MMMU decreased, which contradicts the authors' claim of improved text-image alignment in MLLMs. The authors should provide results on additional benchmarks such as TextVQA, MathVista, and MMBench.\n2.The authors attribute the poor performance to the binary-choice task and limited computational resources preventing training of an 8B model. This explanation seems insufficient; given the computational constraints, the authors could reduce the model size, such as using Qwen-1.8B, to validate the method's effectiveness.\n3.For VLMs, the authors should also provide some zero-shot image classification results, such as on ImageNet. I am curious to know if fine-tuning on VisMin data affects zero-shot classification performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**“which contradicts the authors' claim of improved text-image alignment in MLLMs”**\n\nWe do not claim that we improve general image-text alignment *for MLLMs*. We claim that we improve *CLIP’s* general image-text alignment and substantiate that claim by showing improvements on the standard COCO image-text retrieval.\n\nAlso, we would like to note that improvement in fine-grained understanding tasks does not necessarily translate to an improvement in standard image-text understanding tasks as also observed in another recent work [1]. This is due to differences in data distributions of these tasks. For example, standard tasks have more diversity in object classes whereas fine-grained tasks have more diversity in relations, attributes etc. So the best we can hope for is that the performance on standard tasks does not drop much. And we show that this is the case for our method. \n\n[1] Bugliarello et al. Weakly-supervised learning of visual relations in multimodal pretraining. EMNLP 2023.\n\n**Results on additional benchmarks**\n\nFirst, we would like to clarify that the benchmarks suggested by the reviewer require certain specific skills (in addition to general image-text understanding) that our minimal change data does not cover. For instance, TextVQA requires Optical Character Recognition (OCR), MathVista requires math understanding, MMBench requires OCR, structured data understanding, and celebrity recognition. The base model (Idefics2) has been trained on datasets containing such specific skills (e.g., Idefics2’s pretraining data consists of numerous math understanding datasets [2, 3, 4]). However, our minimal change data focuses on general scene understanding. Thus, finetuning Idefics2 on our minimal change data is expected to cause catastrophic forgetting [5] of such specific skills leading to a degradation in model performance on such benchmarks. To mitigate this, one needs to continue training the base model on datasets containing such specific skills along with finetuning it on our dataset, however experimenting with that is out of the scope of this paper.\n\nAs per the reviewer’s request we evaluated the models on the suggested benchmarks. In addition, we also evaluated the models on the VQA v2 benchmark which tests for general scene understanding. See Table 2 of the rebuttal PDF for the results. All results are reported on the validation split (test splits are not publicly available or require submitting to evaluation servers), except for MathVista, whose test split is publicly available. As expected, we see that finetuning on our data leads to a drop in the performance on all benchmarks except VQAv2 where we see improvements in the performance of the base Idefics2 model, thus demonstrating that our data is indeed helpful for improving general multimodal scene understanding. \n\n[2] Kazemi et al. GeomVerse. arXiv 2023.\n\n[3] Lindström et al. CLEVR-Math. arXiv 2022.\n\n[4] Lu et al. Inter-GPS. ACL 2021.\n\n[5] Kirkpatrick et al. Overcoming catastrophic forgetting in neural networks. Proceedings \nof the National Academy of Sciences 2017. \n\n**“the authors could reduce the model size, such as using Qwen-1.8B, to validate the method's effectiveness”**\n\n\nWe’ve actively researched smaller models for our MLLM experiments. To the best of our knowledge, Idefics2 is the best-performing open-source model *capable of processing multiple images*. We’d like to clarify that the Qwen-1.8B suggested by the reviewer is a language model; not an MLLM. Indeed, there’s an MLLM that’s built on top of Qwen LLM called Qwen-VL [6]. However, Qwen-VL is a 7B parameter model and cannot process multiple images. We’ve again reviewed recent papers (during this rebuttal period) and we’ve not found any smaller model capable of processing multiple images. For instance, although MiniCPM [7] has models with 3B params; however none of them can process multiple images.\n\n[6] Bai et al. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv 2023.\n\n[7] Hu et al. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. arXiv 2024.\n\n**Zero-shot image classification results**\n\nWe’ve incorporated the zero-shot image classification results in the last column of table 2 of the rebuttal PDF. The results show that after fine-tuning with a hard negative dataset, both NegCLIP and VisMin-CLIP under-perform the base CLIP model on zero-shot image classification task. However, VisMin-CLIP shows a smaller performance drop compared to NegCLIP, thus demonstrating that our method learns more robust representations. \n\nLastly, as mentioned in our previous response, the drop in the performance on standard tasks is expected when finetuned on fine-grained tasks. We agree that an ideal model should work well for both types of tasks, however, given the large improvements of our method for fine-grained tasks and also for standard COCO image-text retrieval and VQA v2 tasks, we believe that the contribution of our method is still quite positive.\n\n**“There are also some typo”**\n\nWe apologize for these errors and thank the reviewer for pointing them out. We will fix them in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful feedback! We’re encouraged that they found the **motivation** of the paper to be **clear** (R1, R2), the **proposed idea** of using minimal change pairs to improve fine-grained understanding **promising** (R3), the **data curation pipeline** to be **well designed** (R1), and our **curated data** to be of **quite good quality** (R1), and **beneficial for improving fine-grained understanding** (R2). Further, the reviewers acknowledged that the proposed benchmark **reveals significant shortcomings** of current vision-language models (R3) and that fine-tuning these models on the proposed minimal change data results in **huge performance improvements** (R1). The reviewers also appreciated the **technically sound** evaluation protocol and fine-tuning methods (R2), and the **interesting analysis** of VLM and MLLM performance presented in the paper (R2). . Finally, we are elated that the reviewers found our **paper writing** to be **clear**, **well-written**, and **easy to follow** (R1, R2).

**We believe we have addressed all of the concerns of each reviewer. We hope they would consider increasing their respective scores.**

Meta-review: This paper received overall review scores, where most of the concerns of reviewers' concerns have been addressed after the rebuttal. Specifically, the reviewers acknowledge the clear motivation of explanation of the proposed approach, significance and impact of the proposed dataset, and the analysis of VLM and MLLM performance on the proposed benchmark.

The AC also found the authors' response to the ethical reviews convincing. So the AC recommends to accept this paper as a Poster.
Decision: Accept (poster)