Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning
Authors: Tianyi Bai
Yuxuan Fan
Qiu Jiantao
Fupeng Sun
Jiayi Song
Junlin Han
Zichen Liu
Conghui He
Wentao Zhang
Binhang Yuan
Venue: NeurIPS 2025 poster ()
Keywords: MLLMs
Primary area: deep_learning

Abstract: Multimodal large language models (MLLMs) have achieved strong performance on vision-language tasks but still struggle with fine-grained visual differences, leading to hallucinations or missed semantic shifts. We attribute this to limitations in both training data and learning objectives. To address these issues, we propose a controlled data generation pipeline that produces minimally edited image pairs with semantically aligned captions. Using this pipeline, we construct the Micro Edit Dataset (MED), containing over 50K image-text pairs spanning 11 fine-grained edit categories, including attribute, count, position, and object presence changes.
Building on MED, we introduce a supervised fine-tuning (SFT) framework with a feature-level consistency loss that promotes stable visual embeddings under small edits. We evaluate our approach on the Micro Edit Detection benchmark, which includes carefully balanced evaluation pairs designed to test sensitivity to subtle visual variations across the same edit categories.
Our method improves difference detection accuracy and reduces hallucinations compared to strong baselines, including GPT-4o. Moreover, it yields consistent gains on standard vision-language tasks such as image captioning and visual question answering. These results demonstrate the effectiveness of combining targeted data and alignment objectives for enhancing fine-grained visual reasoning in MLLMs. Code and datasets are publicly released at https://github.com/Relaxed-System-Lab/hallu_med.

Hallucination at a Glance: Controlled Visual Editing and Fine-Grained Multimodal Learning
Tianyi .2 Yuxuan Fan Qiu Jiantao2*1 Fupeng Sun', Jiayi Song", Junlin Han", Zichen Liu', Conghui He? Wentao Zhang BBinhang Yuan'
'The Hong Kong University of Science and Technology Shanghai Artificial Intelligence Laboratory
The Hong Kong University of Science and Technology (Guangzhou)
*Imperial College London, "Peking University, OOforr University heconghui@pl @pjaabuuuppppab... org. en, org wentao zhang0pku. edw.. edu.cn, en, biyuan@ust. hk
Abstract
Multimodal large language models (MLLMs) have achieved strong performance vision-language tasks but still struggle with fine-grained visual differences, leading to hallucinations or missed semantic shifts. We attribute this to limitations in both training data and learning objectives. To address these issues, we propose controlled data generation pipeline that produces minimally edited image pairs with semantically aligned captions. Using this pipeline, construct the Micro Edit Dataset (MED), containing over 50K image- text pairs spanning 11 fine- ane-grained edit categories, including attribute, count, position, and object presence changes. Building on MED, we introduce a supervised fine-tuning (SFT) framework with a feature-level consistency loss that promotes stable visual embeddings under small edits. We evaluate our approach on the Micro Edit Detection benchmark, which includes carefully balanced evaluation pairs designed to test sensitivity to subtle visual variations across the same edit categories. Our method improves difference detection accuracy and reduces hallucinations compared strong baselines, includ- ing GPT-4o. Moreover, it yields consistent gains on standard vision-languaa tasks such as image captioning and visual question answering. These results demonstrate the effectiveness of combining targeted data and alignment objectives for enhancing fine-gr grained visual reasoning MIL.Ms. Code and datasets are publicly released at https: //g thub. com/ Melaxxe--Ssstem Systtm-sttmaaaa Lab/ htophallp meddd_dd.d
Introduction
Multimodal large language models (MLLMs) have achieved impressive results on a wide range of vision-language tasks, including visual question answering, image captioning, and referring expression comprehension [8]. Despite this progress, such MLLMs sometimes become strikingly brittle when it comes to fine-grained visual understanding-tter ability detect and reason over small but semantically meaningful changes in images [46]. As shown in Figure 1, even state-of-the-ar models like GPT-4o frequently generate fluent but incorrect responses when faced with minimal edits involving object presence, count, spatial position, or attributes.
This limitation presents obstacle for the deployment of MILMss in real-world applications that demand high precision. Domains such as robotics [9], industrial quality control [16], medical
"Contribute equally.
Project leader.
"Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Type Ofjeet
Topu: Uximenaky
Tope Acdas
Typs Spatil
2 Do you soe the windoo the lift ohe imagi?
Whet's the ooler of the tiees Q
Q- How many sirplanes ar
Q la the white can looking
Whone is large, dark Q.
in the buckggorrd?
thore this imege?
srall bind convide the window?
forn in image?
steible sice the image It bahind the
Yo, theme windowe
rnd. The coloration viauily stiking
The taa in the backguund
Thare four airplanee viohic this irage All them Airlinss strenit.
American
the small hird that sarding
markings looking toward
Vos, the white with back
this image, the lange dack tem postioned Indil bathnab, Hance
window
Des Tark ING
Can metal hos
What doce rhe hankeyrordd imsge like!
Q: How dieco balls this
the dog
parillet!
Tox there metal bas the
The madkgraand Sciteres bright
There arz nine disco bull wiuble in the imagg, The ball are noflactive and add dynamic o clement in the actting
So, their wings anc not gurullld The two sircnafi luve their usings thall opponsts veriicll dirocluue
image. 21 mounted un the
orunge uiility amatteess a belgg
Yes. the day is holding
nor the whine von thone of
concente Iomer and red
bartlo clone to bodyy
the motencycle
Figure GPT-40 exhibits hallucination errors on fine-grained visual question answering. Instances are systematically selected to highlight failure cases in GPT-4o (Date accessed: May 08, 2025) visual question answering (VQA) performance across five hallucination-pronn categories [15, 31]: Object, Universality, Counting, Action, and Spatial. Each row shows two examples where GPT-4o provides fluent yet inaccurate answers due to subtle misinterpretation of visual details. Hallucinated content is shown in red bold text, indicating model-generated descriptions that not grounded in the image. All images are sourced from DOCCI and Visual Genome.
imaging [45], and assistive AI [63] all require reliable grounding in subtle visual cues. Crucially, the distinctions that current models fail to capture- such whether tool is present, how many objects are view, or the direction an object is facing- are trivial for humans, suggesting a fundamental mismatch between MILM representations and the demands of fine-grained reasoning [4, 15].
We attribute this deficiency to two intertwined factors: the lack of suitable training data and the limitations of current learning objectives. Large-scale web-crawled datasets rarely contain image pairs with tightly controlled, minimal semantic differences and aligned textual descriptions, making difficult for models to learn how small visual changes map to linguistic shifts [65]. Moreover, existing training paradigms do not explicitly enforce feature-level stability across such small perturbations, resulting in brittle visual-textual alignments that easily drift under fine- theeggriied edits [40].
To address these issues, we propose a framework that combines targeted data construction with a new fine-gn mme-grained aligmment objective. We introduce a semantically controlled image editing pipeline that generates minimally modified image pairs with precise, contrastive captions. Using this pipeline, we construct the Micro Edit Dataset (MED)a large-scale dataset comprising over 50K image-text pairs across 11 fine- -reiggr edit types, including attribute changes, object insertion/removal, and so on.
To leverage this data, we design supervised fine-tuning (SFT) strategy that incorporates a feature consistency regularization term, which encourages the image encoder produce stable embeddings across visually similar inputs. This objective aligns visual representations more closely with semantic granularity, helping reduce hallucinations and improve robustness.
Finally, we introduce the Micro Edit Detection benchmark, a new evaluation suite designed to directly assess model sensitivity subtle visual changes. Empirical results show that method significantly outperforms strong captioning tasks, demonstrating improved grounding and reduced hallucination.
GPT-4o-on both edit detection and standard VQA and
Our contributions are as follows:
We introduce a semantically controlled image editing pipeline to generate high-quality, minimally different image-caption pairs at scale.

---

We construct and release the Micro Edit Dataset (MED) and the Micro Edit Detection benchmark, targeting fine-grained vision- -lanuggg reasoning.
We propose a feature consistency regularization objective that improves representational stability under small semantic edits.
We demonstrate that our approach reduces hallucinations and improves fine-grained visual under- standing across multiple benchmarks and models.
2 Related Work
Image editing. Image editing involves modifying the visual appearance, structure, or elements of an existing image [23]. While GANs [18, 27, 44] pioneered realistic image manipulation, recent diffusion models [22, 48, 50] and flow-based models [35, 32] have further advanced visual generation and editing capabilities. To achieve more controllable and guided editing, various approaches have been developed, leveraging modalities such as textual instructions [66, 7, 6]. masks [24, 56], layouts [13, 37J, segmentation maps [39, 59], strokes [41, 58, 38], references [11, 12], and point- dragging interfaces [42, 44]. Recent offt--of-hhe-a image editing models enable precise localized modifications while preserving the source content and maintaining consistency with the original distribution [5, 34, 19]. Leveraging these advances, we use the Gemini Flash 2.0 model [19] to generate image pairs with subtle local differences, forming the foundation of our dataset.
Multimodal LLMs. Multimodal large language models (MILMs) integrate vision encoders with pretrained language models, enabling joint reasoning over text and images, seen in [55, 33). These typically combine visual encoders like CLIP [47] with adapter modules (MLPs, query-based transformers, or attention) to connect modalities. Discrete-token models such as Emu [51, 52, 57] and Chameleon [53] process all modalities as token sequences in a single transformer. Despite advancements, MLLMS still struggle with fine-gras tne-grained visual understanding, often hallucinating details or missing subtle differences, especially in minimal-differema tasks [29, 15]. These challenges arise from datasets lacking controlled variability and misaligned training objectives. Our work seeks to address these issues by improving data quality and training methods to enhance fine- tne-ggained reasoning in MI..Ms.
Evaluating Multimodal LLMS The evaluation of Multimodal Large Language Models (MILMs) has progressed from initial benchmarks like VQAv2 [20], GQA [25], and TextVQA [49] newer frameworks such as MM-Vet [61, 62). POPE [30]. MMBench [36], MMStaa[10) and MMVP [64], which focus on robustness, factual alignment, and hallucination. However, most benchmarks still emphasize coarse-grained tasks and overlook models' struggles with subtle, semantically important visual differences crucial for precision- sensitiie use cases [14]. Fine grained visual understanding remains a sigmificant challenge, with MLLMs often missing minor but meaningful changes. resulting in hallucinations or errors. To address this, we introduce the Micro Edit Detection benchmark, which evaluates MILMSs reasoning over minimally different image pairs, thereby complementing existing benchmarks and enabling more precise assessment of visual grounding and robustness.
The Micro Edit Dataset Construction
To advance fine-grained visual understanding in MILMs,, we construct dataset focused on subtle semantic differences, addressing gaps in existing benchmarks. Using DOCCI [43] and Visual Genome [28]. we develop a pipeline with filtering, semantic edit planning, and controlled image editing. In Section 3.2, we detail the complete pipeline used to generate the 50K-image Micro Edit Dataset, including the selection criteria, edit taxonomy, and caption alignment strategy. In Section 3.3, we describe how we construct the Micro Edit Detection (MED) benchmark from this dataset, which serves as targeted evaluation tool for assessing tne-grainee reasoning capabilities in MLLMs.
3.1 Data Generation
To construct dataset for evaluating fine- fine-grained visual understanding, we sample candidate image- text pairs from DOCCI and Visual Genome, which offer diverse, object-rich images and detailed captions- for producing semantically minimal image pairs.
MLLLMGuiided Data Filtering. Due to quality and editability variations in raw image-caption pairs from DOCCI and Visual Genome, we employ filtering pipeline using 2. 5-..--2... a robust pretrained MI..M, for automatic evaluation. The model evaluates image quality, caption clarity, and editability using structured template, considering factors such as sharpness, subject prominence,
3

---

Data Generation
stEp
sTep 2
step
STep 4
Filletig
Images
Gesetation
Ednting Prompt
Bilting
Fibering
Editied Image
E.9.
Stractured Evaluntion Templase
Image Quality
Composition Types
Object
Tis is picture an electric power line setip with muitiple hinds
Test Clirity
Negalint
percled on varsous parts of the Strccuee Can you namove all the hinds from the image and ourpet the image??
Potcutal
Spatial Relstion
SSoree
Interforrnce Factors
Origizal
Coption Alignment and Generation 17
SFT Dataset
Original Image
Otiginal Desenption
Editing Prempt
Complese Caption
Burast
MED Benchmark
Doseription
Imago
Complet: Cuption
Type Dessiption
Caption
Imsage
Pemplate
Wrong Answer
Figure 2: Overview of the Micro Edit Dataset (MED) construction pipeline. We begin with MLLM-guided filtering and controlled visual editing based on hallucination- paoonn change types. Caption aligmment is performed via hep-wise prompting to ensure consistency between original, edited images, and textual descriptions. The resulting data supports both supervised fine-tuning and benchmark evaluation for fine-gre mne-grained visual reasoning.
caption specificity, and whether minor edits would alter caption meaning. Only samples receiving a strict "Yes" are retained, ensuring alignment between visuals and text for semantic-present preeticc-reeeiinngg edits.
Controlled Visual Editing. After filtering, minimally different image pairs are generated using Gemini Flash 2. 0.0 for precise edits, guided by edit instructions created by Qwen-2.5- 5-VL-72B. These instructions are based on eleven compositional change categories linked to common MLLM hallucination sources [31, 15], such object manipulation, attribute changes, spatial reconfiguration, and complex edits like counting and comparison (see Figure 3). For each image, the most suitable category-favee
non-object edits is chosen, and fixed-format natural language prompt is generated. These instructions ensure minimal yet meaningful changes, resulting in contrastive image pairs with aligned original and updated captions reflecting the visual edits.
Data Filtering. To ensure the generated image pairs differ only in subtle ways, we intro- duce visual similarity filtering stage inspired by [54]. Using CLIP embeddings from the model, we calculate cosine similarity and discard pairs below 0.7 threshold following the MMVP [64]. This filtering retains pairs with subtle, meaningful edits, emphasizing minimal visual differences and supporting fine- tneeggraieed visual understanding evaluation.
3.2 Caption Alignment and Generation
To ensure image pairs have semantically precise and visually grounded captions, we design structured caption generation pipeline shown in 2. This pipeline both verifies caption correctness after editing and removes inconsistent image-text pairs. Although automatic fine-graine mne-grained alignment is difficult, we leverage state-of oofee-of-hhe of-the-art MILMs for captioning and text comparison. Specifically, we break the alignment check into four steps, each handled independently by Qwen2. Saee2.5-Vil 5VV--2...
Step I: Caption Completion for Original Image. Original captions in DOCCI and Visual Genome often brief and may omit elements relevant the edit prompt. To create faithful reference, prompt the model with the original image, its caption, and the edit prompt, asking to revise the caption only if the image visibly includes the specified elements. This ensures the original caption is comprehensive yet strictly grounded in the visible content.
Step 2: Caption Generation for Edited Image. With the edited image in hand, we generate new caption that accurately reflects the updated content. The model is given the complete original caption example, and description of the editing type (e.g., "change in spatial relation") to focus its attention on the most relevant aspects.
Step 3: Generating Difference Description. For contrastive supervision, we compare captions from Step and Step 2, then use Qwen3-32B. a strong text-only L.M,, to generate a concise description of their key difference. The model identifies the most salient change and classifies it into predefined

---

uge
Type Couttey
Type
Type Diffremtiittio
Type Neyetion
the sncedd pruge, wifler poo
An sie foos baage, tact opoi diapes diecee Ma offections
social image the fights jat bina bike warkings alanes the
In socmm rmags procecc of pegett
fereprouunl otaris the
apperesd foot
and rack are
matlagg
soord wigh wolle prolene
As the heuge, not spectss diane
socord kiuge fighte jer
image le pridow mil ie derkgrmand.
fess white recingate step
yugge to ciple an wiler' neakiw
suorlngss
Sype Object
Type
Benchmork
Questisne
types
VEGAN CUISINE
VEGAN CUISINE
lorke ascoed banga, sain
la uccud laage graas vidile mnclinaa
fo uccedd lrape, shiay and
GPT4 Anawar
rereeniing oppli
fest gives
Crroent Truch
(3)
12) lothe aooed boaps prese pleat
groed leve wil of albost aod and prics Aoltoe viteouan
trom
octlittica whene Li imaps
Waung Amrric
22
Right Arawe
Figure 3: Examples from the MED Benchmark covering all 11 composition types. These categories reflect common sources of hallucination in MILMs [15, 31] and include both basic changes (Object, Attribute, Scene, Spatial, Action, Part) and more complex transformations (Counting, Comparison, Differentiation, Negation, Universality). Each example shows an image pair, difference question, and multiple-choice answers including the correct response and plausible distractors.
semantic edit type. This difference sentence is later used during training or evaluation support fine-grained reasoning over minimal edits.
Step 4: Human Verification and Iterative Refinement. To validate data quality, we manually inspect 1,000 sampled image pairs across all eleven visual change types. Annotators evaluate whether captions and difference descriptions accurately align with the visual content. Based on issues
refine prompts and reprocess affected samples using Qwen- VL-MMx. This helps eliminate systematic errors and ensures high data quality. To enhance difference descriptions, Qwen3-32B and Qwen2. Twen2.5-VL-775 5-VL-72B rewrite each sample with richer vocabulary and improved clarity. Visual changes rephrased into QA format, forming MED, an instruction dataset with over 50K high-quality samples paired with the images.
under- speei--pe changes or inaccurate attributes
3.3
The Micro Edit Detection (MED) Benchmark
Using MED, we construct a benchmark for evaluating fine-grained visual reasoning in multimodal models. We select image pairs with high semantic similarity (CLIP cosine similarity 0.95), and for each pair, generate nine rephrased questions (see Appendix D). GPT-4o produces the correct answer and three subtle distractors for each question. All question-answw sets are manually verified and categorized into eleven edit types. To prevent contamination, benchmark samples are excluded from the fine-tuning tueitunnng dataset. The final benchmark contains 165 questions evenly distributed across all edit types, offering robust tool for fine-grained visual reasoning evaluation.
To address potential concerns about overfitting to synthetic data, as both the training set and benchmark stem from the same controlled editing pipeline, we augment the evaluation with real-work image pairs. The MED-Real Set is created by sampling 50 minimally different image pairs from the MMVP benchmark [54], selecting those with CLIP similarity scores above 0.95 to capture subtle yet meaningful visual variations. Question- items are generated for each pair using the same fine- fine-grained reasoning format as in the main benchmark. This expands the evaluation set to 215
5

---

items, combining 165 synthetic edit pairs and 50 real-world pairs, offering more comprehensive assessment of sensitivity to controlled differences and real-world generalization.
Fine-Grained Multimodal Learning for MLLMs
In this section, we present our approach for enhancing fine-grained visual difference understanding in MLLMs. Section 4.1 formalizes the problem setting. Section 4.2 analyzes the impact of semantic misalignment and objective mismatch on downstream performance. Finally, Section 4.3 introduces supervised fine-tuning (SFT) strategy that leverages semantically controlled data augmentation to improve the MLLM's capacity for fine-grained multimodal reasoning.
4.1 Problem Setup
We formalize the task of fine-grained visual difference detection using MILMs. Let X' denote the space of images, equipped with perceptual premetric dx (e.g., Learned Perceptual Image Patch Similarity (LPIPS)). Let T denote the space of natural language captions, equipped with a semantic premetric dt (c.g., edit distance BERT-score). Define the ground truth difference operator Ar (f1, 122 as the natural language description of the difference between two captions t and 12. A MLLM M₂ {Ig, To, Za} consists of image encoder Io, text encoder Ta, and an text decoder Za. with parameters We define two loss functions: the autoregressive captioning loss leap and a CLIP- style contrastive loss letip- Given a distribution ((Y') and ground-trun trutt captioning function
f: X T, MLLM training aims to minimize the population risk (equation (4) in [60]):
R(0) (XpP(()) (=)()) (Aap (zn (Zn [I((<r), + Aapp (Imp(I( To (o)),
(I)
where Apa and Aclip > 0 are penalty parameters.
The empirical risk* over a dataset D = { 1i)) is:
i((0)
Σ
Deap laap (Zo [o (x::)),, + Aetip Leiip To [t.)]]
(2)
D
(mi,A:)D
4.2 Downstream Image Difference Task
We consider downstream tasks where the goal is to describe differences between two visually similar images æ1, 22 E X, satisfying (x .22 S e. The performance of MILLM on this task can be evaluated using the generalization error:
G( = Egg P(x")) 4= f==f(x2) Meep (zo (Zo[Io(<)) - Io(x2)) A7 f2)),,
(3)
where (x - Is (x2) means that translating the feature-level image difference into a language- level description. Ideally, we would like the empirical minimizer E arg min R(()) to achieve low generalization error, i.e., G(Ò) is small. However, even state-of-the-art MLLMs (e.g., GPT-4o) still underperform on fine-grained difference description tasks. This raises a central question: why does this performance gap persist? There are two primary contributing factors. First, there may be semantic under-specification, where captions omit salient visual details, or caption incompleteness, where not all semantically meaningful aspects of an image caption t. More broadly, such cases fall under visual-textual misalignment, where the semantic content of the image and caption diverge. As these misalignments model them noisy captions i f(z), where f is arbitrary, potentially biased captioning function. Second, even if the MLLM is trained on a clean dataset, generalization gap can still arise when the model is trained to minimize R(0) but evaluated
described in the corresponding
range from mild to severe,
the downstream objective G(Ø). For example, models in prior works [3, 21, 33, 55) were trained on datasets with rewritten captions. However, our experiments show that these models still perform poorly on tasks that require sensitivity visual differences. The underlying intuition is that standard MI..Ms trained using the loss in equation (2) lack the architectural capacity to explicitly recognize and describe gra-gra differences between similar images.
"CII loss is introduced for background information only. It not used in method's SFT stage.

---

Model
Object
Attr.
Scene
Spalial
Action
Part
Count
Diller.
Compar
Neg.
Univ.
Avg
100
100
API-hased Models
GPPT-44-222-00-166
57.14
56.25
61.54
57.14
33.31
43.75
50.00
66.67
31.58
42.86
64.29
51.12
GPT-4. 1-2025-04-14
71.45
62.50
61.54
50,00
33.33
50.00
61.11
66.67
47.37
42.86
50.00
54.36
Clande3.2 Sonnet
57.14
43.75
35.71
46.67
25.00
38.89
50,00
13.79
42.86
42.86
VL
33.33
69.25
37.50
69.23
50.00
50.00
50.00
38.89
66.67
$2.63
42,86
42.86
51.81
Open-source
Quan2-V1-7B8
35.71
43.75
61.54
42.86
46.67
51.25
50.00
33.33
31.05
28.57
28.57
38.48
Qwen2 (ouns)
Quen2 5-VL-7B
50.00
38.46
42.86
12.50
18.73
50.00
26.32
42.85
Quen2: 5-V1-78 (omn))
57.15
56.25
53.84
42.86
46.67
43.75
55.56
50.00
47.37
50.00
64.29
51.61
LLaVA- V1.67B
64.29
37.50
30.77
21.43
33.33
25.00
27.78
25.00
26.32
21.43
28.57
31.01
LLaVA-V1 (ours)
11MMA-322-11B L1aMA-3.2-118 (curs)
43.75
50.00
50.00
22.22
33.33
15.79
38.46
62.50
46.15
35.71
37.50
$7.50
33.33
41.67
31.58
42.86
42.86
40.92
Table 1: Model performance the MED benchmark subtasks. Each column corresponds to a specific type of task in MED benchmark: Object, Attr. (Attribute), Scene, Spatial, Action, Part, Count, Differ. (Differentiation), Compar. (Comparison), Neg. (Negation), Univ. (Universality), and Avg (overall accuracy). Bold values indicate the best performance per column within each model category. We use colors to highlight whether model trained with our method increases or decreases performance compared the original model.
4.3 Supervised Fine-tuning
Motivated by the preceding two factors, we assume the MILM Mg is trained on a noisy dataser D,, where each image-caption pair (z, t) satisfies that = f()) with probability y,, and f(z) otherwise. That is, with probability 1). the caption faithfully describes the image; with probability 77, it reflects an imperfect or incomplete description.
Accordingly, the baseline MLLM Mg is trained on D,, by minimizing the empirical risk:
R, [D,l (E,,))),,
Σ
Deup leap [Io 14) + Xetip (Iop(I((x;) (t.)))
(4)
This objective corresponds to the population- risk:
(,) 17 R(0)) + (1-1)) -7) -R(0),
(5)
where
R()) E pp()), t= (x)) Deuu Teap (zo [I)())), + Acpi lenp - To (t))].
(6)
Let 6, and 8 denote the empirical and population risk minimizers under noisy supervision. Due to the presence of label noise and mismatch between the training and evaluation objectives, the downstream generalization
G(O,,) may remain large. As result, the model Mg, exhibits suboptimal performance on fine-graine mraigrai image difference tasks.
To mitigate this issue, we propose a supervised fine-tuning (SFT) approach using curated dataset focused image differences. Given a noisy dataset D,, for each pair (Ir, (2), construct augmented triplet as follows.
Create the clean caption t, f(x)):
Edit i, obtain f. and use an image generator (e.g., Gemini) produce visually similar image such that dx .2)) < and f f(x));
Use prompting engine Sp (e.g., an LLM) to generate a fine- fine-grained natural language difference description (t, fi), where Φ represents all parameters of the generator.
This process yields curated dataset Delit for post-training. We fine-tune the model Mg, this dataset by minimizing the following empirical SFT loss:
RSET (0) = DDedall
Σ
Meap [Zo[Io(>:)) - Io(e:)), (1,, 4))
(7)
(t,, D
7

---

Model
Pope
Coarse
Fine
Visual Sim
Visual Corr
Count
MMVP
Ave
MME
Qwen2-VL-7B
Qwen2-Ve VL-7B
92.30
71.21
48.24
51.11
30.23
55.83
31.33
54.35
1679.52
Owen2-VL-7B
73.92
51.85
59.17
56.25
Qwen2.5-Vi
96.29
73.95
$7.35
49.63
33.72
50.00
27.33
55,47
1685. 14
Qwen2.5. (Ours)
97.52
75.97
59.36
51.85
37.79
59.17
28,00
58.52
1701,87
LLaVA-V1.677B
95.56
58.28
31,93
51.11
21.31
45.83
28.67
47.56
1441,89
LLaVA-V1--1 (Ours)
97.39
56.74
35.13
48.14
24.42
49.17
30.00
48.71
1420.57
LLaMA-3.2-1BB
69.03
48.94
43.70
20.93
44.17
26.00
42.13
1421.71
LLaMA-3,2-1IB (Ours)
72,60
47.21
45.93
19.19
50,00
28.00
43.82
1430.67
Table 2:
Performance of models other benchmarks. Green indicates improved performance,
and red indicates decreased performance after fine-tuning. Coarse and Fine correspond to the Coarse Perception and Fine-grained Perception sub-tasks in the MMStar; Risual_Sim, Visual Corr, and Count correspond to the Visual Similarity, Visual_ Correspondence. and Counting sub-tasks in BLINK, respectively; Pope reports POPE precision, and MME is the MME perception score.
Pupe
Vanaa
Visua
Qum2-V1-78
02.50
71.21
48.24
$1.31
30.23
55.83
31.33
54.35
1679.52
Ow-72-VL-7B Vii
03.79
73.81
49.17
$1.31
31.40
$3.28
32.00
54.94
1668.18
Qwee2 VL. (Ous)
96.27
73.92
46.16
51.85
33.72
59.17
32.67
56.25
1681.27
Qwen2 S-V1-B
96.3
23.95
57.35
4063
13.22
50000
27.31
55.47
Qumn. S-V1-2 tmined-Wi
96-41
75.26
S6.35
50:37
15.14
55.00
11.33
57.18
1696 81
Qum225V1222 SVL-28 (Oun)
97.53
25.92
99.36
$1.85
32.70
59.17
26.00
58.52
1701
Table 3: Ablation results. Green cells denote an improvement over the corresponding base model,
red cells indicate a drop, and yellow marks change. All column abbreviations (Pope, Coarse, Fine, Visual Limm Visual Corr, Count, MMVP, Ave, MME) carry the same meanings as in Table 2.
By minimizing (O), the model parameter is updated from the noisy initialization O,, to a refined estimate SST.. The proposed loss function implicitly embeds an objective that forces the model to associate semantic image edits with structured changes in the feature space, and to maintain precise and consistent representations for unaltered image areas. Empirically, we find that the post-trained model M Mage achieves significantly better performance on visual difference description tasks than the baseline Mg.- Moreover, the enhanced ability to reason about fine-grained visual changes also improves performance on classical downstream tasks, as it fosters deeper visual understanding and architectural awareness of subtle semantic shifts between images.
Experiments
We evaluate our approach's effectiveness in enhancing fine-grained visual difference comprehension in MLLMs. In Section 5.1, we outline our experimental settings. Section 5.2 presents findings from the Micro Edit Detection (MED) benchmark, designed to assess sensitivity to minor yet semantically significant visual differences. Section 5.3 examines generalization by testing our models across multiple multimodal benchmarks. Section 5.4 offers an ablation study analyzing how our training strategy's s key design choices affect performance.
5.1 Experimental Settings
To assess our approach's 's effectiveness, we fine-tune several open-source vision-language models on the Micro Edit Dataset using our SFT loss (Fquation 7). Models include Qwen2-VL--7B [55], Qwen2.5-VL-7B [3], LLa VV-V-..... Vicuna-7B [33], and LLaMA-3.2 D.-.M---..-Viisin-nnnitiie Insioo-Instruut-111 [21]. All fine-tuning uses LLaMA-Factory
Factory [67] with consistent hyperparameters for fair comparison. We also evaluate recent commercial closed-source models as performance benchmarks: GPT-4o (2024-08- 06) [26], GPT-4.1 (2025-04-15 [2], Claude 3.7 Sonnet [1], Qwen-VL- VL-Plus (2025-03-18) [3], and Doubao-1.5.. Vision-Pro (2025-03-28) Implementation details are shown in Appendix C.
5.2 Experimental Results on MED Benchmark
We evaluate our method the Micro Edit Detection (MED) benchmark, which tests models' ability to detect and describe minimal yet semantically meaningful visual differences. Table presents results across eleven task categories, comparing human performance, API-based models, and open-source models with and without our fine-tuning.
flosed-source models still siruggle with fine-grained differences understanding compare with human. Human annotators achieve 95.21 accuracy, setting high bar that current models struggle to reach. Even top closed- models like GPT-4.1 (54.26%). Doubao-1.5-viiione --5viisio--rro (51.81%) and GPT-4o

---

(51.32%) fall short, particularly in relational tasks like Action, Comparison, and revealon persistent gaps in fine- ane-grained visual reasoning.
Our fine-tuning method significantly boosts open-source models. Our supervised fine-tuning consis- tently enhances all tested open-source backbones. Qwen2-VL--7B improves from 38.48% to 47.55% (+9.07), with major gains in Spatial (+14.28), Action (+6.67), Count (+5.56), and Differentiation (+25.00). Qwen2.5-VL-7B increases from 40.24% 51.61% (+11.37), excelling in Comparison (+21.06), Count (+22.23), and Universality (+14.29). LLaMA-3.2-1IB rises from 34.71% 1% to 40.92%
(+6.21), and LLaVA-V1.6778 from 40.44% 44.04% (+3.60). These results demonstrate our approach's 's broad applicabilityy across diverse architectures.
Our best model outperforms several closed models MED Benchmark. Among open-source models, Qwen2.5-VV.-7) (Ours) achieves the highest accuracy (51.61%) 61 rivaling commercial models like GPT-4o (51.32%) and Doubao-1.5- 1.0o--..--iiiio-Prrr (51.81%) ).). It also surpasses Claude 3.7 Sonnet (40.44%) and Qwen- VL-Plls (47.36%), despite its smaller scale and open-source nature. This demonstrates that with fine- fine-grained data and alignment objectives, open-source vision-language models can match closed-source systems in tasks requiring precise visual difference understanding.
Evaluation real-world image pairs confirms generalization. We evaluate all models the MED- Real Set (50 minimally different real- world image pairs; see Section 3.3 and Appendix B) to assess generalization beyond synthetic edits. Fine-tuned models outperform their base versions: Qwen2.55 VL-7B improves from 66.0% to 74.0%, ALaVA-V166-7B from 38.00 88.0% to 50.0%, and LLaMA-3.2-11B from 36.00% to 50.00%. Qwen2-VL--T improves from 78.0% to 80.0% accuracy, demonstrating strong intrinsic robustness. These results confirm effective generalization to real-world fine-grained visual differences.
5.3 Experimental Results on Other Benchmarks
To evaluate generalization beyond the MED benchmark, we test our fine-tuned models on several established multimodal benchmarks, including MMStar, BLINK, POPE, and MME shown in Table 2.
Fine-tuning yields consistent and generalizable performance gains. As shown in Table 2, across all models, supervised fine-tuning improves both perception and reasoning, Qwen2.5-VL-7B (Ours) achieves the highest average score (58.52) and MME perception score (1701.87), outperforming all open-source baselines. Notable gains include Count (+9.17), Visual Correspondence (+4.07), and POPE (+1.23), reflecting stronger numeracy, spatial grounding, and hallucination resistance. These improvements hold across model families, confirming the effectiveness of fine-graine supervision.
Robustness improvements reflected in Hallucination Benchmark. Sive2.55VLL-78 (Ours) achieves a POPE score of 97.52, ranking among the highest for open-source models, while I.LaVA-VI 11.6-BB (Ours) improves from 95.56 to 97.39. This demonstrates stronger resistance to hallucinated object predictions, even under challenging prompts. Although minor decreases in Fine-grained Perception are observed-likely due to increased sensitivity to small edits these are outweighed by overall improvements, confirming enhanced precision with minimal trade-offs.
5.4 Ablation Study
To analyze each component's 's contribution, we perform an ablation study with three settings: (1) base model without fine-tuning, (2) fine-tuning only the visual encoder (ViT) with the language model (LLM) frozen, and (3) joint fine-tuning of ViT and LLM (our full method). Table 3 presents the ablation results for Qwen2-VL-7B and Qwen2.5-VL--B.
Fine-tuning the vision encoder alone provides solid gains. Fine-tuning the vision encoder (ViT) alone yields consistent improvements in perception-heavy tasks. For Qwen2.5-VL-7B, this results in 1.31 gain in average score and nearly 10 on the MME perception score. Notable gains include Count (+5.00), Coarse Perception (+1.31), and MMVP (+4.00), highlighting that refining visual representations enhances low-level grounding and numeracy.
Joint fune-tuning further boosts multinodal reasoning. Joint fine-tuning of ViT and LLM achieves the best performance across metrics. For Qwen2.5-VLL77 it improves the average score by +1 +1.34 and MME by +7.04 compared to the vision-only model. Notable gains in Visual Correspondence (44.65) and POPE (+ (+1.09) underscore the importance of aligning the language head with updated visual features, which reduces hallucinations and enhances mrosl-modal integration.

---

6 Conclusions and Limitations
We propose framework to improve fine-grained visual difference understanding in MLLMs by tackling two key challenges: limited semantically controlled data and weak alignment objectives. Our contributions include: (1) scalable pipeline to generate minimally different image pairs, (2) the Micro Edit Dataset (MED) with 50K samples and a 215-item benchmark, and (3) a fine-tuning strategy with feature-level consistency to enhance robustness to small edits.
Limitations and future directions. Our framework focuses on binary edits between image pairs, with promising directions in extending it to multi-step transformations, temporal reasoning, and compositional edits. Full fine-tuning of vision and language components achieves strong performance, but exploring efficient adaptation methods could reduce computational costs. Applying our approach to unified-tole fokkn architectures like Chameleon and Emu3 may further enhance its applicability.
References
[1] Claude 3.7 sonnet system card. URL httpp:://pp....mmmttcccsbbb///be www.ssseeaannniicieieeee wwtp//orrps..:22776222222
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, ct al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392. 2025.
[4] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongho Han, 7heng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. CoRR, 2024.
[5] Black Forest Labs. Flux.1 fill [dev]. 1-Fill-dev, 2024. Accessed: 2025-04-19.
[6] Manuel Brack, Felix Friedrich, Katharia Kormeeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, and Apolinário Passos. Ledits+:: Limitless image editing using text-to-image models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8861-8870, 2024.
[7] Tim Brooks, Aleksander Holynski, and Alexci A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 18392-18402, 2023.
[8] Davide Caffagmi, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Comia, and Rita Cucchiara. The revolution of multimodal large language models: survey, arXiv preprint arXiv:2402. 12451, 2024.
[9] Guangyan Chen, Meiling Wang, Te Cui, Yao Mu, Haoyang La, Tianxing Zhou, Zicai Peng, Mengxiao Hu, Haizhou Li, Li Yuan, et al. Vlmimie: Vision language models are visual imitation learner for fine-grained actions. Advances Neural Information Processing Systems, 37:77860-77887, 2024.
[10] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang. Yuhang Zang. Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403. 200.3303.33 2024.
[11] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6593-6602, 2024.
[12] Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Ye Kim, Qing Liu, Yijun Li, Jianming Zhang Nanxuan Zhao, Yilin Wang. et al. Unireal: Universal image generation and editing via learning real-world dynamics. arXiv preprint 27X7422412.07774. 2024.
[13] Dave Epstein, Allan Jabri, Ben Poolc, Alexci Efros, and Aleksander Holynski. Diffusion self- self-guidance for controllable image generation. Advances in Neural Information Processing Systems, 11226222216299, 2023.
[14] Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang. Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, et al. Mme-survey: A comprehensive survey on evaluation of multimodal 1lms. arXiv preprint 1rxiv:241..559.. 2024.
[15] Deqing Fu, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang. Robin Jia, and Lawrence Chen. Tldr: Token-level detective reward model for large vision language models. arXiv preprint ar1i:22400.4473, 2024.
10

---

[16] Markus Michael Geipel. Towards a benchmark of multimodal large language models for industrial engineering. In 2024 JEEE 29th International Conference on Emerging Technologies and Factory Automation (ETFA), pp. 1-4. IEEE, 2024.
[17] Safouane EI Ghazouali, Umberto Michelucci, Yassin EI Hillali, and Hichem Nouira. Csim: A copula-based similarity index sensitive local changes for image quality assessment. arXiv preprint arXiv:2410.01411, 2024.
[18] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11)1139,14 2020.
[19] Google DeepMind. Gemini 2.5 flash. https: http://////pppo/////coo///////// 2025, Accessed: 2025-04-26.
[20] Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
[21] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 / 783, 2024.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[23] Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan. Jiaxi Lv. Jianzhuang Liu. Wei Xiong, He Zhang, Lian- gliang Cao, and Shifeng Chen. Diffusion model-based image editing: A survey, arXiv preprint arXiv:2402.17523. 2024.
[24] Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction basoo image editing with multimodal large language models. In Proceedings of the IEFF/CVP Conference Computer Vision and Pattern Recognition, pp. 8362-8371, 2024.
[25] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
[26] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint urXiv:2410.11276 2024.
[27] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401- 4410, 2019.
[28] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017
[29] Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. Evaluating and improving compositional text-to-visual generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5290-539000 2024.
[30] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.103:: 2023.
[31] Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pp. 366-384. Springer, 2024.
[32] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint 2rXiv:22/0.0274:4 2022.
[33] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
[34] Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang. Wei Cheng, Jiaqi Liao, Yingming Wang. Honghao Fu, Chunrui Han, et al. Steplx-edit: A practical framework for general image editing. arXiv preprint arXiv:2504.1 177611 2025.
11

---

[35] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint 220000200003003,, 2022.
[36] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2024.
[37] Zhiheng Liu, Yifei Zhang. Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng. Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Customizable image synthesis with multiple subjects. Advances in neural information processing systems, 36:57500-57519, 2023.
[38] Zichen Liu, Yue Yu, Hao Ouyang. Qiuyu Wang, Ka Leong Cheng, Wen Wang, Zhiheng Liu, Qifeng Chen, and Yujun Shen. Magicquill: An intelligent interactive image editing system. arXiv preprint urXiv2241..09703 2024.
[39] Naoki Matsunaga, Masato Ishii, Akio Hayakawa. Kenji Suzuki, and Takuya Narihira. Fine-grained image editing by pixel-wise guidance using diffusion models. arXiv preprint arXiv:2212.02022 2022.
[40] Brandon McKinzie, Zhe Gan, Jean Jean-Philipp Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng. Anton Belyi, et al. Mml: methods, analysis and insights from multimodal Ilm pre-training. In European Conference on Computer Vision, pp. 304-323. Springer, 2024.
[41] Chenlin Meng, Yutong Hc, Yang Song, Jiaming Song, Jiajun Wa, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2188.01073, 2021.
[42] Chong Mou, Xintao Wang. Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint (xXiv:2377.02221, 2023.
[43] Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh. Jordi Pont- Tuset, Garrett Tanzer, et al. Docci: Descriptions of connected and contrasting images. In Furppea Conference on Computer Vision, pp. 291-309. Springer, 2024.
[44] Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Mcka, and Christian Theobalt. Drag your gan: Interactive point-based manipulation on the generative image manifold. In ACM SIGGRAPH 2023 conference proceedings, pp. 1-11, 2023.
[45] Dimitrios P Panagoulias, Maria Virvou, and George A Tsihrintzis. Evaluating Ilm generaeed multimodal diagnosis from medical images and symptom analysis. arXiv preprint arXiv:2402.01730, 2024.
[46] Wujian Peng. Sicheng Xie. Zuyao You, Shiyi Lan, and Zuxuan Wu. Synthesize diagnose and optimize: Towards fine- -rained laigunla understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13288. 2024.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
[48] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
[49] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruy Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVE conference on computer vision and pattern recogmition, pp. 8317-8326, 2019.
[50] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02.500 2020.
[51] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222 2023.
[52] Quan Sun. Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang. Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14398-14409) 2024.
[53] Chameleon Team. Chameleon: Mixed-mode early-fusion foundation models.
arXiv preprint
PrXiv:2455008880 2024.
12

---

[54] Shengbang Tong. Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal Iss.. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568-9578, 2024.
[55] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu. Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception the world any resolution. arXiv preprint arXiv:240.121911, 2024.
[56] Tengfei Wang, Hao Ouyang, and Qifeng Chen. Image inpainting with extemal iiteeraal learning and monochromic bottleneck. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 5120-5129, 2021.
[57] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, ct al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409. 0000099.88800 18869, 2024.
[58] Shiyuan Yang, Xiaodong Chen, and Jing Liao. Uni-paint: A unified framework for multimodal image inpainting with pretrained diffusion model. In Proceedings of the 3 ACM International Conference on Multimedia, pp. 3190-3199, 2023.
[59] Yifan Yang, Houwen Peng, Yifei Shen, Yuqing Yang, Han Hu, Lili Qiu, Hideki Koike, et al. Imagebrush: Learning visual in-context instructions for exemplar-base based image manipulation. Advances Neural Information Processing Systems, 36:487233-8744 2023.
[60] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Young. Mojtaba Scyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 2022.
[61] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu. Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint ar3iv:2300..0222 2023.
[62] Weihao Yu, Zhengyuan Yang. Lingfeng Ren, Linjie Li, Jianfeng Wang. Kevin Lin, Chung- Lin, Zicheng Liu, Lijuan Wang, and Xinchao Wang. Mm-vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities. arXiv preprint 2rXii:2408. 00765, 2024.
[63] Zhiqiang Yuan, Ting Zhang, Ying Deng, Jiapei Zhang, Yoshuang Zhu, Zexi Jia, Jie Zhou, and Jinchao Zhang. Walkvlm: Aid visually impaired people walking by vision language model. arXiv preprint arXiv:24/2.2090. 20900, 2024.
[64] He Zhang, Shenghao Ren, Haolei Yuan, Jianhui Zhao, Fan Li, Shuangpeng Sun, Zhenghao Liang, Tao Yu, Qiu Shen, and Xun Cao. Mmvp: A multimodal mocap dataset with vision and pressure sensors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21842-21885555 2024.
[65] Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Mllms know where look: Training-free perception of small visual details with multimodal llms. arXiv preprint arXiv:2502. 17422, 2025.
[66] Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for ciatrructionnuudddee image editing. Advances in Neural Information Prucessing Systems, 36:31428-31449 2023.
[67] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Ilamafactory: Unified efficient fine-tuming of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL attt:://arzis arg/abs//42ee 13372.
13

---

Appendices
Examples of SFT Dataset
15
B Experiments on MED-Real Set
15
Training Details
16
D More Examples of MED Benchmark
16
Prompts
19
F Analysis on Text Diversity
24
Additional Analysis on CLIP-Score Threshold Selection
24
H Extended Evaluation BLIP Model Family
25
I Backbone Scaling Ablation Study
25
J Efficiency of LoRA Fine-tuning
26
Analysis of Image Quality Filtering and Metric Selection
26
Broader Impact
26
14

---

A Examples of SFT Dataset
In this section, we provide some examples of our constructed MED dataset for supervised fine-tuning in Fig 4. combines high-quality image pairs with corresponding questions and visually grounded answers, focusing on fine- finegrrineed reasoning tasks.
What's the most notable variation between these two pictures?
User
Assilant
Stoner Av
Street
second image shows background
second image shows bluc sky seatrered white clouds, while the first image
second image shows
with lush green trees, while the first image trees, highlighting visual courmst fallage densiry and color.
with black text and a metallie bracket, while in the image, sign blue white
displays clear sky with no clouds visihle.
metal bands securing it in place.
bugoba
bugoba
The second image shows man scated eyes closed phone while the finst image, man is holding phone and appears engaged in conversation.
The second image shows the mam wearing red polo shirt with while borizontal stripes, whole the inage, wearing chockkrred shirt with large black and white
The secund image shows additional prune compured to first image, with eight pranes present instend of seven.
spots.
SFT Dataset
Figure 4: Example samples from the Micro Edit Dataset (MED) used for supervised fine-tuning. Each sample consists minimally edited image pair, a question prompting for the most notable difference, and a visually grounded answer. Answers are refined to highlight concrete semantic changes based visual evidence. The full dataset contains 50K such high-quality QA pairs supporting fine-grained multimodal reasoning.
B Experiments on MED-Real Set
We provide the evaluation results the four models the MED-Real set of our MED Benchmark in Tab 4. We evaluate all models on the MED-Real Set, consisting of 50 minimally different real- world image pairs, to assess their generalization beyond synthetic edits. Fine-tuned models show consistent improvements over their base versions: Qwen2.5-VL-2 improves from 66.00% to 74.00%. LLaVA- V1.6-7B from 38.00% to 50.00%, and LLaMA-3.2-11 from 36.00% 50.00%. Qwen2-VL-7B from 78.00% to 80.00% These findings highlight effective generalization to real-world, fine-grained visual differences.
Model
Acc (%)
Qwen2-VI.- VI.-BB
78.00
Qwen2-VLL VL-7B (Ours)
80.00
Qwen2.5-VL-77
66.00
Qwen2.5-VL-7B (Ours)
74.00
...viuv..-vv
38.00
LLaVA-V LLaVA-VV.6-vicume- 16-vicuna-7BB (Ours)
50.00
Llama-3.22 1amma-3,2-1111 1B.--1BB-VV I1B-VisioonIInttuur
36.00
1----11--Viiiin-nnssee (Ours)
50.00
Table 4: Performance MED-Real Set.
Green cells denote improvement over the base
model.
15

---

C
Training Details
In this section, we present all the hyperparameters we used to training the three kinds models in Table 5, Table 6 and Table 7. All the training processes conducted using llamafactory [67].
Regarding image resolution and the number of image tokens, we adhere to the original settings specified by each model.
Table 5: Hyperaarameters for training Qwen2-VL Qwen2.5-VL. models
Hyperparameter
Value
LoRA Rank
LoRA
LoRA Dropout
0.1
LoRA Target
all
GPU
NVIDIA A800
Batch Size
16
Gradient Accumulation Steps
Warmup Ratio
0.1
Learning Rale
le-4
Learning Rate Scheduler
Cosine
Unfreeze Vision Tower
True
Table 6: Hyperparameters for training L.LVVAV1..--B model
Hyperparameter
Value
LoRA Rank
LoRA
16
LoRA Dropout
0.1
LoRA Target
all
GPU
x NVIDIA A800
Batch Size
Gradient Accumulation Steps
Warmup Ratio
Learning Rate
le-5
Learning Rate Scheduler
Cosine
Unfreeze Vision Tower
False
Table 7: Hyperparameters for training LLaMA-3.2-11B model
Hyperparameter
Value
LoRA Rank
LoRA
LoRA Dropout
0.1
LoRA Target
all
GPU
NVIDIA A800
Batch Size
Gradient Accumulation Steps
Warmup Ratio
0.1
Learning Rate
le-5
Learning Scheduler
Cosine
Unfreeze Vision Tower
D More Examples of MED Benchmark
In this section, we present additional examples from the MED Benchmark in Fig and Fig including their types, answer options, correct answers, and the answers provided by GPT-4.
16

---

Hore do there two images primarity differ from each othor?
Hom do these two images primarily differ from each other?
Type: ARDD wherese fins magg, licene plate her horders and
A
Type
C
licasse place
1123-
sccond imags, house with carks siding and nwo smalf window insted of tre trnk
trank
sharos pant
sxand image. first image, the horrse place blank Marc bonders.
limme plate readh 123-ABCD,"" wheress
B. la the secord image. there is aree trsi rbiiki, and the left side stows part of'a house witl light siingg and dioer festcel of sree mwnns.
the second image, the Elont licease plate rouds "123 123-ABCD ABCD." wheress the firs image, the licesse piaze readi 1w0-00000" black bossers.
C. In the second imase, dhere is no leve trust sirible, and the lefl side shows part
of house witl dask siding bus window destead of sree srank.
the sccond image the Boor licasse plane rouda M123-BBC: in white sout. whereas the fist inage, nends 1'12-ABCDI" but be black sexr withi no bender.
D. In the sccord imass, there is su ince tnnk visisi, and the lol vide shous peet
of's house with dark sidiny and winddow troicad of liee cank.
's must significant contrast between these nwo photos?
What's the notable variation between these
STOP
STOP
GROSS TRAFFIC
GROSS TRAFFIC
DOES NOT STOP
DOES NOT STOP!
Type: Negation
Type
P
A In the socord imags, the reffections and cranks the sto sign fostzee sann in the find image are compledly abari.
image, is looking a window, whrre image, wow booking ensight abead
noom.
II It the scond image. the for sit sickles and shadover the step sign thor are present is the first image not mentioned.
the second imago, the white car is fooking arwards e smulf hid cuzside the image, was looking tewards the floor right.
window, wheras
C. In the spcond image, ice sckles and reffections de the sign that are in the firs image not mentioned.
C. the second image, the while car is looking sewands o small bind outside the window, whereas the first image, was Anoking bekined msely rowards the left.
D. In the socord
ice ecklos and angles Aintin the thai in the find
the sccond image, the white cal aut looking tonoads a small bisd outidee the window, whevess the first image, i Anokinz bekind iself fowards the right.
daarnbal.
What is the fundamental difforence can see betweon these tro vimuls??
What distinguishes image other?
Type: Counting
D D
Type: Differentistion
D C
A. in second image, the refrigerator is large, Hdlie, and staanless sieel with solid dosa, whereas in the firsi insage the refrigeaatr smull, Black, and lacks glass doce.
the serond image, both the mumber of controllrs and their button configumtions iicerionl but rearnnged companed the first
socend image, the refrigeratee lage,, wheress the first imago, the refrigerator is small, white, and does not have a stainless steel tinish.
sacand inage, munder of controllis same, but buites configurations are reversed compared to the fist image.
the second configurations
number controllers difforour, but image.
C. in the sccond imaaee the reli refiigerator is Aurge, black, and stuinless steef solid doer, whereas the fiest image, refiigerator a
D. the sccond image, the number of contnollers and their bution configummions are different compared the first image.
D. in the second image, the refrigurator is large, black, and stainlees stuel with glass door, whereas the first image. the reftggeratc is small, white, and said
Figure 5: More examples in the MED benchmark
17

---

What divinnguishes one image from the other?
How do there two images primarily differffrom from each other?
Type:
P
S
B
Type
P
C
A. In the second mage, the roed appeary dty, whereas it the firs image, the road ceverol with water.
the second image the fern beside the daritted the flowr, wherees the fert imsage, the fern was on toy of the sink.
the wound image, the nead appoars to de daney, whanow fins imagg, the soad clessly flouded with water.
the scronn image. the ferm secr the sink on the counter, whanes in the fioe image, the fer was boods the buistb un the floer
C. In the second mage, the read sypears wer, whereas in the rinst imago, the roa
C. the socondl Iavege, the fimm is sip of the sink, wheres the fire image. the firm was bokiidd rise Aurituuh on the flawe.
cevared with muil
D. lo the formt image, the nead wherena in the sccomdd image, the neal be partally covered with water.
the fan boidd the Iuihtuh on small stoal, whenaae the Sext the fem was hanging the wall above the sink.
D. the soarn
How de thrie two inages primarily differ from each other?
most signifliant contrast between two photos?
Counting
Type Part
D
S A
A. the second imago, has greet place additional Powers, whenes image, table is adoomed white flosers no grven planr
the second image, the table has white flowers and ne green plant, whereas the first imagr, the table adommed with green plant and adtitional flowers.
the
C. the second imags, the seble adummea with flowers and gecce plam. wherress the flrst image, the rable green plant wilk white flovers and acdmonal flowers.
seuand imsage. planes.
D, saamd image three planes, wherraa first imags. there unly noo planss.
D. the second image, the table govosw plant with white flomers and additionll flowers, whereas the fist image, die sable is adorned with flowers and grwn plant.
distieguishes image from
What is the primary difference between the two images?
Type: Universality
P
Type Universaltty
second inage, imoge, dry sparso, colorral folinge
fillage figh--
strighht multiple parking
emppl,
greze.
parking
stright
In sxandd image, ribraur and deenne wheres tvny the foliage is spursc..
nough
and the jivllage image, the bark cenvvih lightrr
image, nuttiple parking
C. second imagn, the appears facied. but with intrvatte portors, tiv follage is sparse and grecwbil-yeeeee yellere w wereas is the first imagt, the back is smooth and light colored, and the follage dease und bright green.
soard imnage thene is lange parking lot wbwdy
imags, there are mulipsle parking fan filled cars
vehrides,
socand imagge there are aualtjil parking Aons flllied with winds road whterees in the fixs image. there single large parking lot with few visble velicles.
Dh. In the second imagn, the bark nectrred and lighe-oolord, and folloge Ie back and green, whence in the first imags, the fork is fadee and the jullege is sparne.
Figure 6: More examples in the MED benchmark
18

---

E
Prompts
In this section, we systematically present all the prompts used in the construction of the Micro Edit Dataset. These prompts are designed cover every stage of the dataset creation process, including filtering editable images, generating editing instructions, producing detailed descriptions for both original and edited images, describing the differences between image pairs, conducting evaluation, and generating training well evaluation data. Specifically:
Figure 7 shows the prompt for filtering editable images, ensuring that the selected images are suitable for subsequent editing tasks.
Figure 8 illustrates the prompt for generating image editing instructions, which enables the creation diverse editing scenarios.
Figures 9 and 10 demonstrate the prompts for generating comprehensive descriptions the original and the edited images, respectively, providing essential context for later analysis.
Figure 11 presents the prompt for describing the differences between two images, facilitating both quantitative and qualitative analysis of micro edits.
Figures 12 and 13 display the prompts used for evaluating the quality of the image edits, ensuring objective assessment process.
Figure 14 shows the prompt for generating Supervised Fine-Tuning (SFT) data, which supplies high-quality training samples for model fine-tuning.
Figure 15 provides the question template used for benchmark evaluations, enabling a systematic assessment of model performance micro editing tasks.
Figures 16 and 17 demonstrate the prompts for generating the correct and distractor answers,
respectively, enhancing the scientific rigor and difficulty of the benchmark.
All prompts have been carefully crafted and iteratively refined to ensure high quality and diversity in the dataset, laying solid foundation for subsequent research and algorithm development on miero editing tasks.
19

---

following sample-which inclades image accompanying (question-nnswrr pair desuription)-haeee the criletia listed below. Your response must STRICTLY follow the lormat of the provided csumple. For each critetion, provide hrief mnllysis, and conclude with clear "Yes" or "No" recormmendation.
Caption of the image: (caption)
#s **Faluutionn
**Final Recommendation:** **Yos** or N*N
**1 Image quality:**
**Clacit:"* [Evaluate the image sharp, free of blur, and low noise]
*Mainn Subject. 44 [Evalunte there clearly identifablle primary subject image]
*"Rackground Bambkgroudd_Cmpplciiee Complexity:** [Evaluate whether the background is simple and free of encessive clutter]
**2. Text Clarity (Descriptien):**
**Spxifity [Evalsate the deseription clear specifie, and unambignous]
UUniqueess: [Evalsate
provides staight/ooward. unsque information ambiguity]
**3 Editing Potental
*'dentifiable Editable Elements:** [Evaluate if image contains clearly identifiable objects attributes that be modified]
"Imppct on Test Accuracy:** [Bvalnure whether applying an odit would make the original text incorrect)
*4 Interfereuce Factors:**
VVvuul Distractins:** [Evellate whether there are distractung elements or complexs soenes that maght hander a clear edit]
*SSbjeectivity Text:* [Evallnte the contains subjective nepects that could interfere with evaluation]
**Exphamiin: ** [Proide expliasion supporiing your resommnendation]
IMPORIANT: Your answer MUST tollow this EXACT lormat, with bold section headers and bullet points shown above. Your Recommendation must clearly Yexter ree bold.
Figure 7: Prompt for Qwen-2.5-VL--7BB to filter editable images
20

---

protosssonal mmage promupt engunest specislizing tn umage edsting mstrrutiinn.
Analyze the provided umage and any accompanyıng text, then generate clear editing procupt based on what you sec.
Deseription of the image: [)
Your editing prompt must follow this exact structure:
'ih. This is photo of [tref deserppton of the current mmage]. Can you [spectlic ecating instructios] and outpui image""
Choose an appropeate editing insructionn from these composition categones:
BASIC COMPOSITIONS
Object: Adding cemoving entibes (people, mimals, food, vehicles, objects)
AAtribure Changing visual properties (coler
shape)
Scene: Modilyigg backgrounds settings
Spptiaal Relation: Altering physical amagements (right, left, top, helow, inside)
Actoom Relation: Changing interactooss between enbises
Part Relation: Modifying part-whole relationships (hody parts, clothing, accessoris)
ADVANCED COMPOSITIONS:
-Countingg Adjusting quantity
Dillerentiaion: Distingushing objects by altributes
Comparison Changing relative chimeteristies
Negatuon: Remocing
Univenatitty Applying changes to all members of
Examples:
- 'Hi, This a photo of dog with hiscaits on its head Can you suhtraet one hiseuit its head and outgut the image?"
add lwo vehicles 'Hi. This is photo of table Can you only move the phone out of this scene and output the image!"
picture ol' highway.
and oulput image?"
Based solely what you observe the provided image, first indicate which composition type using (Prioritize
COMPOSITIONS other than Ubiiet.. Only choose 'Object' other lypes camoot elfectively demonstrate the editing possibilities image.). ). Your response should be this fommat
appropriate editing prompt..
ell Type: [somposition type'aallii Prompt: [your editing promptfin
Figure 8: Prompt for Qwen-2.5-VL-72B to generate editing prompt
prompe tomplate
""GGyee en image, ats current caption, and an edal prompt. your lask is determine whether the captiea lislyy mnceporates inlormatson from edit prompt. If not, you need to update the caption to inclode any missing information.
Image:
[The image will be provided]
Current Capticn: coriginal description) Edit Prompt. (odil prompt)
IMPORTANT. complete eaption should ONLY deseribe ACTUALLY VISIBLE image. information the edit prompt if those elements don't actually exist in the image.
Follow these
Carefully esamine the image and understand the currntt caption
Identify all modified information requested in the edit prompt
3. Vardy if each element Eoen the edit prompx actually viscble in the image
Check whether
caption alredyy
visible
prompt
5. the caption FULLYY incorporates all relevant information from the edit prompt, respond with- "Caption is complete. No updates needed."
the caption MISSING relevars information that ACTUALLY VISIBLE the Preserving relevant parts of the original caption
updated capsion by:
Naturally integrating ONLY missing
actually
image
Maintaining colerent, fluent, and accurate description
Ensuring the updated caption uocurately dosnrbbs what is visible the image
EECCLUDING any information from the prompt that is actually present the image
Provide only the updated caption without explanations, confirm that no updates are needed (Ry Answer exactly "NO UPDATE")
Figure 9: Prompt for Qwen-2.5-V6 2.5-VL to generate complete original description for original images
21

---

is generating an accumat caption given image, using following context:
Reference deseription: (description)
Key sspect focus [type) (ftype deseciption))
task is
Carefilly examine the actual image content
2. Note any differences between the image and the reference description, especially regaeding the [typc] aspects
accuratc caption accurately deseribes ONLY Iruly image
IMPORIANT:
Your description must based solely what actually visible the
Do not include details from the reforeace doscription that arcn't present in the image
Pay special
to accurately describing (type] Be precise and objective your deseription
Please provide
caption English, beginning with "CAPTION:"
Figure 10: Prompt for Qwen-2.5-VLL777 to generate description for edited images
""Given image captions specific type
Captoo (complete caption)
Caption tnew__sgtiine
Type
Obbjoe: Adding or removing entities (poople, animels, food, vehicles, objects)
-Atrrbbut:: Chamging visual propetties (color, material, size, shape)
Scene: Modifying hackgroands settings
Spaiul Relation: Altering physocal ummgements (rigt, left, on lop, below, made))
Changing
between
Past Relstio:: Modalying part-whole relationstups (body parts, clothing, accessories)
Counting: Aéjusting quantity of entities
Difierentintion: Distinguishing objects aitributes
Comparison: Changing relative chamuteristties
Negation Removing elements
Universait: Applying changes all members ol group
Identily only
duférences
captions
'lype defference belong
Respond with just key differences, using one sentence one type
Please streily follow this fommat XX0
Figure 11: Prompt for Qwen-2.5-VIL-72B to generate difference description
""Deteminn if the desctibod difforence matches the edit
Edit Prompt (odit prompt)
Deseribed Difierence. (diffcrence description)
Assess whether the deseribed difference convectly reflects what requested in the edit prompt.
Answer with Only "Yes" mutch. explanation needed. 2. Only "No" ther'''s no obvious change. No explanation needed.
Figure 12: Prompt for Qwen-2.5-VLL-7BB to get judge
22

---

determine 'Deeribed Differenee" mitches Description
Second Deseription: (second edit prompt)
Doseribed Differenoc. (dilkrennce description)
IMPORTANT GUIDELINES EVALUATION:
"Caption I" refers to first image description, and "Caption 2" refers the second image description.
FOCUS MEANING, NOT EXACT WORDING: Synonyms, paraphrasing. altemative expressions convey
meaning should be considered equivalent.
ALLOW FOR
The difference description may use differeent plansing or perspective than description
stil being contect.
ACCEPT STRUCTURAL DIFFERENCES: Different sentence structures, word order, or tamative appronches that communicate the
valid.
RECOGNIZE CONCEPTUAL MATCHEES: When calegortes concepis, observations are semambcalll similar (even if usinn
terminology),
EVALUATE BASED ON CORE CHANGES, If buth descriptions identify the same fundumenttal clanges between images,
through different expressions, consider them aligned.
BB GENEROUS if details vary.
When doult,, toward accepting deseriptions that capture essential elements,
Answer with 1. Only "Yes" they match. No explanation uceded. 2. Only "No" if there's obvious change. No explmation needed.
Figure 13: Prompt for Qwen-2.5-VL-72B get second judge
need your help to rewrite a description of differences between two images.
Current description (needs improvement): "(diff" description."
Please rewrite description to focus the VISUAL DIFFERENCES between the two images, NOT just differences in captions.
Important guidelines:
Assume "Caption refers to the first image, and "Caption refers to the second image
describe what appears in second image compared to image Your rewritten description image shows.." or the first image, while in the second image..."
formats like
actual visual elements that are present or Be specific about what was added, removed, changed
image
Keep your description concise but informative
Your response follow
format:
IMPROVED DESCRIPTION: [Your rewritten description here]
includo any other text, explanations, formatting your response.
Figure 14: Prompt for Qwen3-328 to generate SFT data
# Question template variations
"What the primary difference between two images?".
"What the main distinction between these two images?".
"What's the key difference between these two pictures?",
"Can you identify the principal disparity between these two images?".
"How do these two images primurily differ from each other?"
"What's the significant contrast between these photos?",
"What distinguishes one image from the other?".
"What the fundamental difference you can see between these two visuals?".
"What's the most notable variation between these two pictures?"
Figure 15: Question Template of Benchmark
23

---

ten
You are analyzing the differnnce between two umnges deseribed by captioes. Your task is to gonerate close answer that describes how image differs first image.
Ingul: A dillerence description that compures elements between. two imgess.
Cutput statement that explains how second image (Caption differs from first image (Caption
output
ANSWER: (Your descrppioe how the secoed ditlers Eom the first umage)
Example:
Inpul dillerenze descriptiu:: "The deer on the lar right is grazing with his head ficing toward the grass in the lirst caption, while the caption, stands facing slightly
right"
Output:
ANSWER: In the second image, the deer on the far right is stinding alone facing slightly the right, whenens in the first image, the deer grazing
facing loward the grass.
Important requirements:
Always frume
describe the second image diflers from the lirst image
Use precise: languge to descrihe the spesific differences
clarity 4. Always follow the A ANSWER: fermat exactly shown
Begin second image"
Figure 16: Prompt for GPT-4o to generate right answer
""Plcese create oxaetly completaly incomrct stotements thout the diffcrence betwean two images thet are similar to the courect nasmer but woong subcle ways.
Contest:
The fist image (Caption and second image (Caption show the same scene with a specific ence
actual dillenence is: "chilerence descmption)"
difference moserr"
Your tock:
essctly stntements that deceptively
Each woung seawer chould closcy rebied lae actuol diTeacuce. but coutain crucill crres
Make the wrong answers simihe enought the right unswe that they could confisee someone
The statements should snard plsisiblee and ragiiee conafal atlention distingish from the cornat snswer
suggest specilo Use simils wording siruciuee level as excrect auswer
woong
image" "lhe unge?
arong answert: (Your dist wrong nnswer hare]
wrong inswer: (Your secoad wrong answer bere]
3: YYur thind
Remember statements should mbbly ways make them challenging distingnist from the anwer!"
Figure 17: Prompt for GPT-4o to generate wrong answer
Analysis on Text Diversity
To further validate the quality and diversity of the generated instructions in our dataset, we computed standard lexical diversity metrics. Specifically, we report the Moving Average Type-Token Ratio (MATTR) with a window size of 50 for both our proposed dataset and the LLaVA-Instruct- 150k dataset for comparison.
Our dataset achieves MATTR score of 0.7393 and LLaVA- Instrrttuut-150 scores 0.7834. These quantitative results demonstrate that our instructions exhibit rich lexical diversity and low redundancy in phrasing, which is a notable characteristic for a Visual Question Answering (VQA) dataset.
G Additional Analysis on CLIP-Score Threshold Selection
To determine the optimal CLIP-score threshold for constructing our MED dataset, we conducted an ablation study evaluating the trade-off between data quality and scale. This analysis complements approach for MED-Bench where we follow [54] in using 95% similarity threshold.

---

We trained identical Qwen2.5-VL- models on MED subsets filtered using different CLIP- simllaiily thresholds (6), with comprehensive evaluation across 12 reasoning categories. As shown in Table G, our analysis reveals several key findings:
Higher thresholds (Ó 0.8) produced better per-sample alignment, with á 0.8 achieving 78.6% accuracy on universal tasks. However, this came at the cost of significantly reduced data size (36k samples), leading performance degradation in compositional reasoning tasks such as spatial reasoning (-21.2% versus = 0.7) and part (-12.5%).
Conversely, thresholds below 8 < 0.7 introduced noisy image-text pairs that degraded performance across all categories.
The optimal threshold of 8 = 0.7 with 50k samples achieved the highest average accuracy (51.5%) by balancing two critical factors: (1) preserving sufficient diversity for complex reasoning tasks, evidenced by an 18.8% improvement in attribute binding compared to å = 0.8; and (2) maintaining scale advantages, as demonstrated by the 10.3% performance gap between the full 50k dataset and random 12k subset at the same threshold.
This systematic analysis confirms that å 0.7 optimally balances data quality and quantity for robust model training.
Table 8: Performance comparison across different CLIP- CLIP-similarity thresholds (d)
Model (6)
Data Sioe
Ave
Object
Altr:
Sceme
Spacial
Action
Pant
Count
Diller.
Compar.
Neg.
Univ.
0.7
ramlom- 12k
41.21
53.85
43.75
53.65
42.96
62.50
18.75
22.22
33.33
31.58
50.00
50.00
36k
66.67
53.84
37.50
46.15
21.43
13.75
31.25
61.11
50.00
42.11
50.00
78.57
0.9
12k
44.85
46.15
50.00
46.15
35.71
43.75
43.75
38.89
50.00
31.58
57.14
57.14
H Extended Evaluation on BLIP Model Family
To further demonstrate the broad applicability of our approach across different model architectures, extended our MED benchmark evaluation to include the BLIP series, specifically the newest BLIP-3o model. This evaluation covers all 11 semantic edit types and compares performance both before and after fine-tuning on our dataset.
As shown in Table After fine-tuning with our method, BLIP-30-8B demonstrates improved performance the MED benchmark, with average accuracy increasing from 44.24% to 50.90%. These results validate that our fine-tuning approach generalizes effectively beyond the Qwen and LLaVA families, providing consistent performance improvements across diverse model architectures and establishing the broad applicability of our methodology.
Table 9: Performance comparison of BLIP-30-8D on MED benchmark before and after fine-tuning
Model
Avg
Obj
Attr
Scene
Spatial
Action
Part
Count
Differ
Compar
Neg
Univ
BI.I--3-88
44.24
46.15
50.00
69.23
50.00
37.50)
31.25
44.44
41.67
31.58
50.00
42.86
BLIP-3o-8B
50.90
38.46
62.50
46.15
42.86
62.50
37.50
66.67
50.00
52.63
35.71
57.14
I Backbone Scaling Ablation Study
To evaluate the general applicability of our approach across different model scales, we conducted an ablation study by fine-tuning Qwen2.5-VE VL models of varying sizes (3B and 7B parameters) on the same MED dataset. The performance results, measured across 12 reasoning categories, are presented in Table 10.
Table 10: Performance comparison of Qwen2.5-VL. models different scales before and after fine-tuning on MED
Model
Obj
Scene
Spatial
Action
Part
Count
Diller
Compar
Univ
Owen2.5
33.94
38.46
37.50
33.85
21.41
25.00
43.75
27.78
16.67
16.32
35.71
50.00
Swen2.5-VL-33 (Ours)
38.79
46.15
43.75
38.46
35.71
43.75
43.75
38.89
16.67
42.11
28.57
42.86
Swen2.5-VL-72
39.74
53.85
50.00
38.46
42.86
12.50
18.75
44.44
50.00
26.32
42.86
57.14
Owen2.5 VL-7B (Ours)
51.61
56.25
43.75
47.37
64.29
25

---

Our experimental results demonstrate that both model variants benefit substantially from our super- vised tineetuiing strategy. The 3B parameter model shows an average accuracy improvement from 33.94% to 38.79%, while the 7B parameter model achieves more pronounced gain from 39.74% to 51.61% %. These consistent improvements across different model scales confirm that the effectiveness of our MED approach is robust and scalable, independent of the backbone model size. This ablation study validates our core design choice and highlights the general applicability of our method.
Efficiency of LoRA Fine-tuning
In our experiments, employed LoRA (Low-Rank Adaptation) for all models due its favorable trade-off. As summarized in Table 11, LoRA significantly reduces training time while achieving performance comparable to full-parameter fine-tuning across different model architectures.
Table 11: Comparison of LoRA and full-parameter fine-tuning
Model
Fine-tuning Method
Training Time (50k data)
LoRA
hours
Qwen2.5-VI--7B
Full
29 hours
LoRA
13 hours
LLaVA-1.6-7B
Full
57 hours
LoRA
hours
LLaMA322-1BB-L
Full
70+ hours
These results indicate that for fine-grained visual understanding tasks, the necessary model adaptations can be effectively achieved through low-rank updates without requiring full retraining of the model backbone. This suggests that such tasks primarily require adjustments in higher-layer representations rather than fundamental changes to the base model parameters.
K Analysis of Image Quality Filtering and Metric Selection
To ensure the quality and semantic consistency of edited images our MED dataset, we implemented comprehensive image filtering pipeline. Our primary approach adopts CL.P-based embedding similarity, following the methodology established by [54), to maintain high-level semantic alignment between edited images and their originals.
We complemented this CLIP-based analysis with Structural Similarity Index (SSIM) measurements to evaluate low-level visual consistency across our 50K image pairs. The mean SSIM score is 0.5011 (std: 0.2028, max: -0.0134, min 0.9839) confirm that most edits introduce structurally subtle modifications while preserving overall image integrity, aligning with objective of fine-grain fine-grained semantic editing.
Our selection of CLIP over SSIM as the primary filtering metric was based on several considerations. SSIM is known to struggle with localized edits on uniform backgrounds [17], whereas CLIP demon- strates superior robustness to superficial visual changes while focusing semantic-level alignment. This characteristic makes CL.IP particularly suitable for our task of ensuring conceptual consistency in fine-grained image editing.
For instruction-image alignment validation, we employed a dual approach: instructions were directly used as generation inputs to the Gemini model, followed by manual verification on over 1,000 samples. This comprehensive validation ensures strong semantic aligmment between textual instructions and their corresponding visual edits through both procedural enforcement and human evaluation.
L Broader Impact
This work aims to improve fine-grained visual understanding multimodal large language models (MILMs), which can positively impact applications requiring visual precision, such as robotics, assistive technologies, and industrial inspection. However, enhanced sensitivity to subtle visual differences could also increase risks of misuse, such generating more convincing disinformation
26

---

enabling deceptive visual narratives. While our dataset is based on synthetic edits over public images and does not introduce direct privacy concerns, models fine-tuned on it may still inherit underlying biases from pretraining. We recommend responsible deployment, especially in safety-critical settings, and see our benchmark as a tool to help identify and mitigate hallucination risks future systems.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our motivation, contribution, and scope are clearly described in both the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We describe our limitations in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper includes all necessary details for reproducing the main experimental results in Section5.1 and C. This includes thorough description of the model architecture, training procedures (e.g.. optimizer settings, batch size, learning rate schedules. number of training epochs), dataset splits, and evaluation metrics in We also provide hyperparameter settings and implementation details that influence performance. To further support repro- ducibility, we will release the source code and datasets for training and evaluation upon publication.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
29

---

reproducible, they some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The code and data are in supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.. cc/
oniioiccy)))))) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. httpppubuuuuuesssp/.//pppppppeep
onoolicy))) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We present our experimental setting and details in Appendix. C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Since our work focus on inference of language models, the results are provided evaluation scores on validation/test set. And we have explained how they are calculated in Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
30

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We show the experiments compute resources in C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: : hww.rpuuu.../// httpppbbii...iiiccniiieeeeelllle
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code of Ethics, and strictly followed it in this
work.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, talk about boarder impacts in Appendix L.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
31

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use two public datasets and strictly follow their usage license. Besides, we cite their original paper to express our gratitude for their work. The details of the dataset source used in this paper are described in 3.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
32

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We includes the anonymized URL as well as details about dataset and code in supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
33

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We utilize LLMs as one of the key components for generating the instruction data, and have provided ALL the prompts used in experiments.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components. http///ipp..c/oofeeeeeeeeeeeeeee wwt.cc////oooccoceeeeee ccc//Comferrences/222/LLLLL for what should or should not be described.
Please refer to our LLM policy (https humps://neune
34


Reviews:
[{"review_id": "RcYMMUFlcg", "reviewer": "Reviewer_WxCP", "summary": "This paper is targeting a core-issue in Large Vision-Language Models(LVLM) -- Fine-grained understanding of image content. First, the authors have shown that even in some most advanced LVLMs, such as GPT4o, there still exists issues with detail understanding. And they conclude these errors into a range of types. Then the authors construct a dataset for fine-grained image learning by their proposed pipeline. This pipeline uses image-editing to subtly modify two images and ask questions related to the original and edited images. Then the author argued that the models trained to describe the differences between two similar images are not doing well in the downstream description task. Inspired by this observation, the author proposed a new supervision training method to overcome this problem.  In this way, the model can learn to discreminate the fine-grained differences. The author also proposed a new loss function wich included the contrastive loss to train the LVLMs and show the performance superiority over other methods.", "questions": "1. In the data generation process, the pipeline uses a QwenVL model to generate instruction and filtering the images from the raw dataet. Then, the authors use the gemini model to generate the image pairs from the generated instructions. Since the instruction generation process is very important for the quality of generated images in the proposed MED dataset, do you consider any other quality control method for the instructions. From my understanding, QwenVL is indeed very powerful, but is far from perfect. A Qwen2.5VL-72B model can also have hallucination during generating instructions.\n\n2. The authors use another data filtering module powered by CLIP models to varify the generated images quality compared to the original images. However, from my experiences with many image generative models, only semantic filtering is not enough for generated image quality control. In fact, the author should consider introducing some basic image quality evaluation method to control the qulity(i.e. SSIM). Furthermore, the authors only do CLIP embedding similarity check between the original image and the edited image but not between the edited image and the instruction. Why is that?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Weaknesses:\n1. This data generation pipeline involved too many models: from open source to close source, from LVLM to LLM; acrossing different platform; including text generation and image generation models. This whole pipeline is quite complicate and thus hard to replicate your work because of the involving of too many models. \n2. Though I personally do not have any problem with leveraging synthetic data in training models, I still doubt the quality of the proposed datasets. I think the authors should release more quantitative results to evluate their prompt template and instruction quality.\n\n\nStrength:\n1. For the MED-Benchmarks, this work manually verified and categorize the question types. They also do the examination of data contamination for the splits.\n2. This work does not simply train a downstream models on the curated datasets but verified the generalizing problem and minimize the performance gap in downstream tasks by proposing a new training method. The authors suggest that the performance gap of the existence LVLMs might be caused by the noisy training data which has a binomial probability distribution for giving a good quality description of the paired images.  And thus this work proposing a clean supervision training pipeline to mitigate such noise. \n3. The curated dataset is quite valuable to the research community.", "was_revised": true, "final_justification": "I am satisfied with the response and recommend to accept this paper. Though I have considered about rasing my rating, I don't feel this paper is so good for a 5, accept. So, I will maintain my score as 4.", "rebuttal": "**Overall Response**\n\nThanks for the insightful feedback! Based on these comments, we clarified the modular design of our pipeline, addressed concerns about hallucination and instruction quality, and added quantitative metrics (e.g., MATTR, SSIM) to support our claims. We also highlighted the roles of Qwen3 and manual verification in ensuring data quality. All clarifications will be incorporated into the revised version. \n\n\n> **Response to W1: Complexity of the Data Generation Pipeline**\n> \n\nWe thank the reviewer for raising this important concern regarding the complexity and reproducibility of our data generation pipeline. While our framework may appear to involve a large number of models at first glance, we would like to clarify that the core pipeline relies primarily on just two key components:\n\n1. **Qwen2.5-VL-72B** — used consistently for instruction generation, caption refinement, and caption alignment.\n2. **Gemini Flash 2.0** — used for high-fidelity image editing with minimal semantic changes.\n\nTo ensure quality and robustness, we decompose the generation process into several *small, modular steps*, each focusing on a simple task well within the capability of the model employed. This modular design is intentional: rather than requiring complex, open-ended generation from a single model (which risks hallucination), each model performs only tightly-scoped operations, such as:\n\n- Revising captions only when elements are *visibly present* in the image.\n- Generating editing instructions based on predefined semantic categories (e.g., spatial, object, attribute).\n- Producing updated captions *conditioned on known differences*.\n\nThis design minimizes the cognitive load on the model and enhances output reliability.\n\nAdditionally, our dataset construction relies on publicly available APIs or open-source models, where all the scripts to leverage those toolkits are also released for reproducibility:\n\n- All prompt templates and model configurations have been released (in supplementary material).\n- Our dataset construction relies on publicly available APIs or open-source models, where all the scripts are provided.\n- Each pipeline stage is deterministic and documented in detail (see Section 3.2).\n\nWe believe this strategy offers a principled balance between *model efficiency*, *data quality*, and *reproducibility*. We will clarify this point further in the revision to dispel the impression of an overly entangled system.\n\n> **Response to Q1: Instruction Quality and Hallucination Control in Qwen2.5-VL**\n> \n\nWe appreciate the reviewer’s concern regarding potential hallucinations in instruction generation using Qwen2.5-VL-72B. While no vision-language model is entirely immune to hallucination, our pipeline is explicitly designed to *mitigate and control such risks* through two key strategies: task decomposition and multi-stage filtering.\n\n**1. Task decomposition: keeping model outputs focused and grounded**: Rather than asking the model to generate complex captions or instructions in one pass, we break the process into atomic and visually grounded subtasks, each scoped narrowly enough to reduce hallucination likelihood:\n\n- In Step 1, Qwen2.5-VL is only asked to *revise or extend* an existing caption, *conditioned on a clearly visible target object or attribute* and the original image. This minimizes hallucination by anchoring generation on visual evidence.\n- In Step 2, when generating the edited caption, the model receives the full original caption, the edit type (e.g., “change in spatial relation”), and the edited image. This instruction framing ensures that only relevant semantic changes are included.\n- In Step 3, difference descriptions are generated by a *separate text-only LLM* (Qwen3-32B), which compares the original and edited captions in isolation from visual noise, further reducing compounding hallucinations.\n\n**2. Multi-stage filtering: catching imperfect instructions or captions**: Beyond carefully-scoped prompts, we apply multiple steps of filtering to ensure quality and remove any hallucinated or inconsistent outputs:\n\n- In the initial image filtering stage, Qwen2.5-VL-72B evaluates each image-caption pair using a *structured scoring rubric* that checks for caption clarity, visual relevance, and editability. Only high-quality samples are retained.\n- After image editing and caption generation, we apply CLIP-based similarity filtering between the original and edited images to ensure that visual changes are minimal and semantically plausible.\n- For the final dataset, we manually verify 1,000 randomly sampled image pairs (spanning all 11 edit types). This human check helps catch rare but systematic hallucinations and refine our instruction prompt templates accordingly.\n\nThrough this design, we ensure that Qwen2.5-VL is always solving a *constrained*, visually grounded task rather than performing free-form generation. We will clarify this design in the revised version and include additional metrics (e.g., instruction acceptance rates, manual error rates) to support our filtering effectiveness.\n\n> **Response to W2: Evaluating Prompt Template and Instruction Quality**\n> \n\nWe appreciate the reviewer’s concern regarding the quality of the proposed synthetic dataset, especially the prompt templates and instructions.\n\n1. **Quantitative metrics for text diversity:** To further validate textual quality, we calculated standard diversity metrics on our instructions:\n    \n    - Moving Average TTR (MATTR, window size=50, ours):  0.7393\n    - Moving Average TTR (MATTR, window size=50, llava-instruct-150k): 0.7834\n    \n    These scores indicate rich lexical diversity and low redundancy in instruction and prompting phrasing for a VQA dataset.\n    \n2. **Instruction quality control via Qwen3 and manual verification:** In our original data generation pipeline (Section 3.2), we explicitly employed Qwen3-32B, a strong text-only LLM, to refine and optimize edit instructions and difference descriptions. These instructions were rewritten for clarity, specificity, and semantic correctness. Furthermore, we conducted manual verification on 1,000 samples spanning all 11 edit categories to ensure alignment between images, captions, and difference descriptions, reducing systematic errors.\n    \n3. **Strong downstream performance:** The consistent improvements achieved across multiple model families (Qwen, LLaVA, LLaMA) and benchmarks demonstrate the practical value of the curated data.\n\nWe will include these quantitative metrics and clarify the role of Qwen3 and manual validation in the revised version to strengthen transparency around instruction quality.\n\n> **Response to Q2: On Image Quality Filtering and Use of CLIP vs. SSIM**\n> \n\nWe appreciate the reviewer’s insightful suggestions regarding image quality control.\n\nOur current pipeline adopts CLIP-based embedding similarity—a practice inspired by [1]—to ensure that edited images remain semantically close to the originals. CLIP excels at capturing high-level conceptual consistency, which aligns well with our goal of fine-grained semantic editing.\n\nTo complement this, we computed SSIM on our 50K image pairs to assess low-level visual similarity:\n\n**SSIM Statistics (on MED Dataset):**\nMean: 0.5011 | Std: 0.2028 | Min: -0.0134 | Max: 0.9839\n\nThese results confirm that most edits are structurally subtle, as intended.\n\nWe chose CLIP over SSIM because:\n\n- SSIM is known to struggle with localized edits on uniform backgrounds [2].\n- CLIP is more robust to superficial visual changes and focuses on semantic-level alignment.\n\nAs for instruction-image alignment, the instructions are already used as generation inputs to Gemini and further validated via manual checks on 1,000+ samples. While CLIP-text similarity could be added, our current design already enforces semantic alignment both procedurally and manually.\n\nWe will add the SSIM results to the appendix and clarify this in the revision.\n\n[1] Tong S, Liu Z, Zhai Y, et al. Eyes wide shut? exploring the visual shortcomings of multimodal llms[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 9568-9578.\n\n[2] Ghazouali S E, Michelucci U, Hillali Y E, et al. CSIM: A Copula-based similarity index sensitive to local changes for Image quality assessment[J]. arXiv preprint arXiv:2410.01411, 2024"}, {"review_id": "RtjObyp6fd", "reviewer": "Reviewer_driM", "summary": "This paper works on fine-grained visual reasoning in Multimodal Large Language Models (MLLMs), particularly their tendency to hallucinate or miss small but meaningful visual differences (e.g., object presence, count, spatial changes). The authors identify two key limitations: insufficient training data and weak alignment objectives.\n\nTo address these, they propose:\n1. Micro Edit Dataset (MED): A large-scale dataset (~50K image-text pairs) of minimally edited image pairs and aligned captions spanning 11 fine-grained semantic edit types.\n2. Controlled Visual Editing Pipeline: Generates image pairs using editing models like Gemini Flash 2.0, and captions via Qwen-VL and Qwen3-32B.\n3. Micro Edit Detection (MED) Benchmark: A 165-question benchmark across the same 11 edit types, designed to evaluate sensitivity to small differences.\n4. Feature Consistency Regularization: A fine-tuning objective that encourages stable visual embeddings for minor edits, thereby improving semantic alignment.\n\nThey fine-tune open-source models (e.g., Qwen2-VL-7B, LLaVA) and show significant improvements on the MED benchmark and generalization to other tasks (e.g., POPE, MMStar, BLINK). Their best model (Qwen2.5-VL-7B) matches or surpasses GPT-4o and Claude 3.7 Sonnet on fine-grained reasoning tasks.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Well-motivated and timely problem: Hallucination in MLLMs is a known but under-addressed issue, and the paper identifies a specific failure mode and addresses it directly.\n2. High-quality dataset creation: the MED dataset is carefully constructed using various tools for editing, filtering, and alignment. The dataset and benchmark can be a solid contribution. \n\nWeaknesses:\n1. Finetuning with specific learning objective on a specific dataset are expected to gain improvements, but it is not clear how the fine-tuned models perform on general benchmarks. \n2. the MED-Real set only has 35 image pairs, which might to small to verify real-world applicability.\n3. Other families of open-source models to be evaluated, e.g., BLIP-2 / BLIP-3.", "was_revised": true, "final_justification": "Authors addressed some of my concerns in the rebuttal.", "rebuttal": "**Overall Response**\n\nThanks for the insightful feedback! Based on these comments, we have added a new experiment on BLIP-3o, expanded the MED-Real set to 50 samples, and included 4 general benchmarks (POPE, MMStar, BLINK, MMVP) to demonstrate the generalizability and robustness of our method. We also clarified the concerns regarding overfitting, dataset size, and model diversity. We will include all these discussions and results in the final version if the paper is accepted.\n\n\n> **Response to W1: Generalizability Beyond MED**\n> \n\nWe thank the reviewer for raising the concern about generalization. As shown in Table 2 (Page 8) and the table below, our fine-tuned models consistently improve across diverse general benchmarks such as POPE, MMStar, BLINK, and MMVP.\n\nFor instance, Qwen2.5-VL-7B improves in average score from 55.47 to 58.52, and MME from 1685.14 to 1701.87, demonstrating clear gains beyond MED. These results indicate that our fine-grained tuning approach enhances not only task-specific performance but also general multimodal reasoning, including coarse/fine perception, hallucination mitigation, and counting.\n\n| Model | Pope | Coarse | Fine | Visual_Sim | Visual_Corr | Count | MMVP | Ave | MME |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Qwen2-VL-7B | 92.50 | 71.21 | 48.24 | 51.11 | 30.23 | 55.83 | 31.33 | 54.35 | 1679.52 |\n| Qwen2-VL-7B (Fine-tuned) | 96.27 | 73.92 | 46.16 | 51.85 | 33.72 | 59.17 | 32.67 | 56.25 | 1681.27 |\n| Qwen2.5-VL-7B | 96.29 | 73.95 | 57.35 | 49.63 | 33.72 | 50.00 | 27.33 | 55.47 | 1685.14 |\n| Qwen2.5-VL-7B (Fine-tuned) | 97.52 | 75.97 | 59.36 | 51.85 | 37.79 | 59.17 | 28.00 | 58.52 | 1701.87 |\n| LLaVA-V1.6-7B | 95.56 | 58.28 | 31.93 | 51.11 | 21.51 | 45.83 | 28.67 | 47.56 | 1441.89 |\n| LLaVA-V1.6-7B (Fine-tuned) | 97.39 | 56.74 | 35.13 | 48.14 | 24.42 | 49.17 | 30.00 | 48.71 | 1420.57 |\n| LLaMA-3.2-11B | – | 69.03 | 48.94 | 43.70 | 20.93 | 44.17 | 26.00 | 42.13 | 1421.71 |\n| LLaMA-3.2-11B (Fine-tuned) | – | 72.60 | 47.21 | 45.93 | 19.19 | 50.00 | 28.00 | 43.82 | 1430.67 |\n\n> **Response to W2: MED-Real Set Size and Generalization Validation**\n> \n\nWe thank the reviewer for pointing out the limited size of the original MED-Real set (35 image pairs). While this subset already shows meaningful performance differences across models, we agree that further validation would be very helpful.\n\nTo address this, we augmented the real-world evaluation with an additional 15 minimally edited image pairs, bringing the total to 50 samples. This allows us to test real-world generalization at multiple scales (15, 35, 50). The results are as follows:\n\n| Model | acc@15 | acc@35 | acc@50 |\n| --- | --- | --- | --- |\n| Qwen2 | 66.67 | 82.86 | 78.00 |\n| Qwen2 (Ours) | 73.33 | 82.86 | 80.00 |\n| Qwen2.5-3B | 46.67 | 54.29 | 52.00 |\n| Qwen2.5-3B (Ours) | 53.33 | 57.14 | 56.00 |\n| Qwen2.5-7B | 60.00 | 68.57 | 66.00 |\n| Qwen2.5-7B (Ours) | 73.33 | 74.29 | 74.00 |\n| LLaVA | 46.67 | 34.29 | 38.00 |\n| LLaVA (Ours) | 60.00 | 45.71 | 50.00 |\n| LLaMA3.2 | 26.67 | 40.00 | 36.00 |\n| LLaMA3.2 (Ours) | 46.67 | 51.43 | 50.00 |\n\nThese results indicate:\n\n- Consistent performance boost from our fine-tuning across all sizes and models.\n- No overfitting to synthetic edits — the models trained on MED also generalize well to real-world minimal differences as Response to W1 explained.\n\nWe will include the expanded 50-sample MED-Real set and analysis in the revised appendix to strengthen our empirical claims.\n\n> **Response to W3: Inclusion of Other Open-Source Models (e.g., BLIP-2 / BLIP-3)**\n> \n\nWe appreciate the reviewer’s suggestion to evaluate additional model families beyond Qwen, LLaVA, and LLaMA.\n\nTo address this, we include the newest model from BLIP series, **BLIP-3o** in our MED benchmark evaluation. The table below shows its performance across all 11 semantic edit types, both before and after fine-tuning on our dataset:\n\n| Model | Avg | Obj | Attr | Scene | Spatial | Action | Part | Count | Differ | Compar | Neg | Univ |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| BLIP-3o-8B | 44.24 | 46.15 | 50.00 | 69.23 | 50.00 | 37.50 | 31.25 | 44.44 | 41.67 | 31.58 | 50.00 | 42.86 |\n| BLIP-3o-8B (Ours) | **50.90** | 38.46 | **62.50** | 46.15 | 42.86 | **62.50** | **37.50** | **66.67** | **50.00** | **52.63** | 35.71 | **57.14** |\n\nThese results show:\n\n- Consistent performance gains from our fine-tuning approach.\n- Improved reasoning on complex types such as comparison, counting, and action.\n\nWe will add these results to the revised paper and appendix to further demonstrate the broad applicability of our method across model families."}, {"review_id": "CcquHnYDx9", "reviewer": "Reviewer_ysLX", "summary": "This paper tackles hallucinations in large VLMs, a problem of high interest in LM research. The authors hypothesize the root of hallucinations of large VLMs is inadequate training data requiring fine-grained visual understanding. The authors propose the Micro Edit Dataset, which provides both a training and benchmarking portion for improving and evaluating performance on fine-grained visual differences. The authors perform extensive fine-tuning on their dataset based on risk minimization over hallucination. They show that the SFT procedure results in higher performance across a number of standard VLM benchmarks.", "questions": "Can the authors comment on why performance seems to be a large margin higher on the MED-Real set vs the main MED benchmark? Are there any particular reasons stemming from the candidate data distribution, or is this an artifact of the model training domain? In this case, how can it be verified that the performance gains on fine-tuning on the MED were not primarily due to bridging the domain gap between real and synthetic?\n\nCan the authors comment on the quite frequent artifacts of the texts referring to samples as texts? Will this not introduce false negatives in benchmarked models?\n\nCan the authors demonstrate qualitative or quantitative ablations over the data curation pipeline, showing why specific similarity thresholding, etc. necessary to curate the best dataset?\n\nOverall I have a positive impression of this work, but addressing my questions would encourage me to improve my rating. Thanks!", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe problem is clearly of high importance, and the visual quality of paired images seems high. While not totally realistic visually, the resulting performance gains seem clear to validate the pipeline to be built upon as parts of larger VLM fine-tuning pipelines. The formulation of SFT as empirical risk minimization for hallucination makes sense, and results in nice performance gains.\n\nWeaknesses:\nThere are artifacts of the text-based difference caption generation in multiple of the presented examples. For instance, the texts include phrases like \"are not mentioned,\" \"is described,\" \"in the second caption.\" This raises questions about potential false negatives during benchmarking due to selecting the next best choice that describes an image instead of a caption.\n\nAs the dataset is a main contribution of the paper, more extensive qualification behind design choices (e.g. similarity thresholding, failure cases, ablations on backbone LLMs) would help strengthen the usefulness of the MED.\n\nSome of the images are qualitatively obviously synthetic. The performance on the MED-Real set is much higher than analogous performance on the synthetic set (admittedly the real set is small).", "was_revised": true, "final_justification": "I maintain my positive review. I maintain concern about some of my initially raised points, as there were many text artifacts presented in the paper, and the revised data was not shared for qualitative evaluation. Additionally, while the expansion of real-image evaluation is appreciated, 50 images is still not a very large sample size. I recognize these concerns are difficult to address during a rebuttal period, and I remain positive overall about the contribution and core idea of the work.", "rebuttal": "**Overall Response**\n\nThanks for the insightful feedback! Based on thse comments, we conducted new ablation studies (on similarity threshold and model scale), expanded the MED-Real set, and clarified benchmark construction procedures to ensure robustness and generalization. We address all concerns in detail below and will incorporate these updates into the final version if the paper is accepted.\n\n> **Response to W1: Text Artifacts in Difference Descriptions and Benchmark Reliability**\n> \n\nWe thank the reviewer for pointing out the phrasing artifacts such as *“is described”*, *“are not mentioned”*, or *“in the second caption”*, which appear in some of our difference descriptions.\n\nWe want to clarify that these phrases originally appeared in intermediate contrastive descriptions produced during Step 3 of our data construction pipeline (Section 3.2). However, before finalizing the benchmark question-answer format, we apply a systematic transformation of these descriptions into visually grounded natural language questions. Specifically:\n\n- We rewrite contrastive captions into QA-style prompts, phrased purely in terms of *visual content*, not textual description.\n- All questions in the MED benchmark are then presented with four answer choices, and we manually verify every question-answer set (Section 3.3) to ensure that:\n    - Exactly one option is visually correct.\n    - The other options are plausible but visually incorrect.\n    - No options rely on meta-knowledge of captions or textual artifacts.\n\nThis human verification step is essential to prevent the kind of false negatives the reviewer is concerned about — e.g., cases where a model might “correctly” describe the image but get penalized due to linguistic mismatches in the caption text.\n\nWe will add clarification and examples in the final version showing how we resolve these caption artifacts during the transition from raw difference sentences to benchmark QA pairs, ensuring evaluation is image-grounded and artifact-free.\n\n> **Response to W2: Design Choices and Ablations in MED Construction and Q3: Impact of Similarity Threshold and Backbone Scaling**\n> \n\nWe thank the reviewers for emphasizing the importance of justifying our dataset design, especially regarding similarity thresholding and backbone model selection. We have now included **two additional sets of ablation studies** to strengthen this point:\n\n**1. Similarity threshold (δ) ablation**: We compare model performance when training on subsets of MED with different CLIP-based similarity thresholds (δ). Results show a clear trend: stricter thresholds (e.g., δ = 0.8 or 0.9) yield better per-sample quality, but smaller data sizes limit overall performance. Meanwhile, δ = 0.7 with full 50k samples yields the best average accuracy, balancing quality and data scale.\n\n| Model with different δ | Data size | avg | obj | attr | scene | spatial | action | part | count | differ | compar | neg | univ |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| δ=0.7 | 50k | **51.51** | 57.14 | 56.25 | 53.84 | 42.86 | 46.67 | 43.75 | 55.56 | 50.00 | 47.37 | 50.00 | 64.29 |\n| δ=0.7 | random-12k | 41.21 | 53.85 | 43.75 | 53.85 | 42.86 | 62.50 | 18.75 | 22.22 | 33.33 | 31.58 | 50.00 | 50.00 |\n| δ=0.8 | 36k | 46.67 | 53.84 | 37.50 | 46.15 | 21.43 | 43.75 | 31.25 | 61.11 | 50.00 | 42.11 | 50.00 | 78.57 |\n| δ=0.8 | random-12k | 42.42 | 61.54 | 31.25 | 53.85 | 28.57 | 37.50 | 37.50 | 33.33 | 41.67 | 36.84 | 64.86 | 50.00 |\n| δ=0.9 | 12k | 44.85 | 46.15 | 50.00 | 46.15 | 35.71 | 43.75 | 43.75 | 38.89 | 50.00 | 31.58 | 57.14 | 57.14 |\n\n**2. Backbone scaling ablation**: To test the general applicability across model scales, we fine-tuned Qwen2.5-VL-3B and 7B on the same data. As shown below, both 3B and 7B variants benefit significantly from our SFT strategy, demonstrating that MED’s effectiveness is robust across different backbone sizes.\n\n| **Model** | **Avg** | **Obj** | **Attr** | **Scene** | **Spatial** | **Action** | **Part** | **Count** | **Differ** | **Compar** | **Neg** | **Univ** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Qwen2.5-VL-3B | 33.94 | 38.46 | 37.50 | 53.85 | 21.43 | 25.00 | 43.75 | 27.78 | 16.67 | 26.32 | 35.71 | 50.00 |\n| Qwen2.5-VL-3B (Ours) | **38.79** | 46.15 | 43.75 | 38.46 | 35.71 | 43.75 | 43.75 | 38.89 | 16.67 | 42.11 | 28.57 | 42.86 |\n| Qwen2.5-VL-7B | 39.74 | 53.85 | 50.00 | 38.46 | 42.86 | 12.50 | 18.75 | 44.44 | 50.00 | 26.32 | 42.86 | 57.14 |\n| Qwen2.5-VL-7B (Ours) | **51.61** | 57.14 | 56.25 | 53.84 | 42.86 | 46.67 | 43.75 | 55.56 | 50.00 | 47.37 | 50.00 | 64.29 |\n\nTogether, these ablations validate the core design choices behind MED—highlighting how both threshold calibration and model scalability are accounted for. We will include these new results and discussion in the revised version.\n\n> **Response to W3: Synthetic Visual Quality and Real Set Performance Gap**\n> \n\nWe thank the reviewer for pointing out the visual quality difference between synthetic images and the MED-Real subset. While some synthetic samples may appear artificial upon close inspection, we ensured semantic fidelity through CLIP-based filtering with a cosine similarity threshold > 0.95. The average CLIP similarity gap between synthetic and real pairs is < 5%, suggesting the visual differences are minimal in embedding space.\n\nAs shown below, the performance gap between MED and MED-Real is not uniformly large, especially for LLaVA and LLaMA models:\n\n| Model | MED (Avg) | MED-Real (Acc %) |\n| --- | --- | --- |\n| Qwen2-VL-7B | 38.48 | 82.86 |\n| Qwen2-VL-7B (Ours) | 47.55 | 82.86 |\n| Qwen2.5-VL-7B | 39.74 | 68.57 |\n| Qwen2.5-VL-7B (Ours) | 51.61 | 74.29 |\n| LLaVA-V1.6-vicuna-7B | 31.04 | 34.29 |\n| LLaVA-V1.6-vicuna-7B (Ours) | 40.44 | 45.71 |\n| LLaMA-3.2-11B-Vision-Instruct | 34.71 | 40.00 |\n| LLaMA-3.2-11B (Ours) | 40.92 | 51.43 |\n\nThese results suggest that our synthetic dataset generalizes well to real-world data. The high real-set performance of Qwen models may be partially due to pretraining exposure to Visual Genome–style content, which is consistent with the source of MED-Real. However, for other models like LLaVA and LLaMA, the real/synthetic performance is comparable, indicating no evidence of domain overfitting\n\n> **Response to Q1: MED vs. MED-Real Performance and Domain Concerns**\n> \n\nWe appreciate the reviewer’s question on the observed performance gap between the synthetic MED benchmark and the MED-Real set. We address this from three perspectives:\n\n1. **Performance gap is not uniformly large**: As we noted in our response to W3, while Qwen models show higher performance on MED-Real (possibly due to pretraining exposure), models like LLaVA and LLaMA exhibit only minor differences between MED and MED-Real. This suggests that the synthetic nature of MED is not the primary limitation for those models.\n    \n2. **Improvement is not just from bridging synthetic–real gap**: Our fine-tuned models improve not only on MED, but also on **unrelated, real-world general benchmarks** that do not involve synthetic data. As shown in **Table 2 (Page 8) for full results**: \n    \n    | Model | Ave | MME |\n    | --- | --- | --- |\n    | Qwen2-VL-7B | 54.35 | 1679.52 |\n    | Qwen2-VL-7B (Ours) | 56.25 | 1681.27 |\n    | Qwen2.5-VL-7B | 55.47 | 1685.14 |\n    | Qwen2.5-VL-7B (Ours) | 58.52 | 1701.87 |\n    | LLaVA-V1.6-7B | 47.56 | 1441.89 |\n    | LLaVA-V1.6-7B (Ours) | 48.71 | 1420.57 |\n    | LLaMA-3.2-11B | 42.13 | 1421.71 |\n    | LLaMA-3.2-11B (Ours) | 43.82 | 1430.67 |\n    \n    These results confirm that our method improves fine-grained perception capabilities broadly, rather than simply adapting models to synthetic artifacts.\n    \n3. **Careful design to avoid domain artifacts**: To further avoid domain leakage:\n    \n    - The MED-Real set is strictly held out from any fine-tuning.\n    - Our instruction pipeline includes both open-domain instructions and domain-agnostic edit types.\n    - We apply manual verification to ensure semantic correctness and diversity in both real and synthetic settings.\n\nTogether, these points support that our performance gains reflect true improvements in fine-grained reasoning, not merely domain adaptation. We will clarify this in the final version.\n\n> **Response to Q2: Benchmark Robustness to Text-Based Artifacts**\n> \n\nWe understand the reviewer’s concern that textual phrasing in the dataset might cause false negatives—for example, if a model selects the most accurate visual description, but the correct answer is tied to wording about the captions themselves.\n\nWe emphasize that such risks are fully mitigated through two key measures:\n\n1. All benchmark questions are rewritten to remove references to text or caption structure.\n2. Manual verification ensures that each question is answerable by observing the image alone, without needing to reason about how something was “described” or “mentioned.”\n    \n    In short, although textual artifacts may appear in internal steps of our pipeline, they are completely eliminated from the final evaluation format. We will clarify this in the revision and include an example in the appendix illustrating this transformation."}, {"review_id": "jZKpW2Fv5J", "reviewer": "Reviewer_PV47", "summary": "The paper points out that current MLLMs struggles at fine-grained visual differences, which may further lead to inaccurate predictions or hallucinations. To address this, the authors designed an automated data generation pipeline, yielding the Micro Edit Dataset (MED). This dataset comprises over 50K image-text pairs across 11 fine-grained edit categories, specifically designed to highlight subtle visual changes.  Then, the author validate the effectiveness of the dataset with different MLLMs like Qwen 2.5-VL. The results demonstrate that the dataset not only enhances MLLM performance on the dedicated MED benchmark but also improves performance on other VL tasks.", "questions": "1. I am curious about the scalability of the Micro Edit Dataset (MED). If the dataset size were continuously scaled up, would performance improvements persist?\n2. The authors employed LoRA for fine-tuning the model. What would be the impact of not using LoRA? Additionally, the hyper-parameters vary significantly across different base models. What might explain this discrepancy?\n3.  I noticed that CLIP loss is incorporated into all presented equations. However, most existing MLLMs do not typically integrate CLIP loss during SFT. Is CLIP loss indeed utilized in the paper's SFT process?\n4. The paper claims to use a feature-level consistency loss. How is this implemented within the proposed method? If I understand correctly, the SFT loss only requires the model to distinguish differences between two input images.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The identification of existing MLLMs' weakness in recognizing subtle visual changes is significant. Current SOTA MLLMs remain considerably behind human-level recognition abilities (e.g., GPT-4.1-2025-04-14 at 54% versus human performance at 95%), highlighting a crucial direction for future MLLM optimization.\n2. The dataset generation pipeline is well-designed, incorporating various filtering and editing techniques to ensure the production of a high-quality dataset.\n3. The authors rigorously validated the MED across diverse MLLMs, utilizing both the MED benchmark and other VL benchmarks. This extensive validation confirms the dataset's utility in enhancing MLLM recognition capabilities.\n\n# Weakness\n1. Some details are not clear. See Questions.", "was_revised": true, "final_justification": "After careful consideration, I will maintain my score as \"borderline accept\".", "rebuttal": "**Overall Response**\n\nThanks so much for the time and thoughtful feedback! Based on these comments, we conducted detailed scalability experiments and efficiency comparisons between LoRA and full fine-tuning. We also clarified the role of CLIP loss, hyperparameter settings, and the implicit feature-level consistency enforced by our SFT objective. These updates confirm the scalability, efficiency, and soundness of our method, and we will incorporate them into the final version if the paper is accepted.\n\n> **Response to Q1: Dataset Scalability of MED**\n> \n\nWe thank the reviewer for the valuable question regarding the scalability of the Micro Edit Dataset (MED).\n\nTo assess this, we trained models on progressively larger subsets of MED, ranging from 10k to 50k samples. The results are shown in the table below:\n\n| Model with sft scalability | avg | obj | attr | scene | spatial | action | part | count | differ | compar | neg | univ |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| 10k | 40.00 | 53.85 | 25.00 | 46.15 | 57.14 | 18.75 | 31.25 | 44.44 | 58.33 | 36.84 | 50.00 | 28.57 |\n| 20k | 42.42 | 53.85 | 25.00 | 53.85 | 42.86 | 37.50 | 50.00 | 50.00 | 50.00 | 31.58 | 35.71 | 42.86 |\n| 30k | 43.64 | 46.15 | 56.25 | 61.54 | 61.54 | 37.50 | 31.25 | 27.78 | 33.33 | 33.33 | 35.71 | 50.00 |\n| 40k | 47.88 | 53.85 | 56.25 | 38.46 | 50.00 | 50.00 | 43.75 | 50.00 | 50.00 | 36.84 | 42.86 | 57.14 |\n| 50k | **51.51** | 57.14 | 56.25 | 53.84 | 42.86 | 46.67 | 43.75 | 55.56 | 50.00 | 47.37 | 50.00 | 64.29 |\n\nThe results show a consistent improvement in performance as the dataset size increases. The model trained with the full 50k MED samples achieves the highest average accuracy and shows stronger capability in nearly all categories. This demonstrates that MED supports scalable learning and that its fine-grained design continues to provide useful supervision as more data is added.\n\n> **Response to Q2: Impact of Using LoRA and Hyperparameter Differences**\n> \n\nWe thank the reviewer for the thoughtful question regarding the use of LoRA and the variation in hyperparameters across base models.\n\n**1. LoRA vs. full-parameter fine-tuning**: In our experiments, we chose to apply LoRA for all models due to its strong efficiency-performance trade-off. As shown below, LoRA not only significantly reduces training time, but also achieves comparable—or in most cases, nearly identical—performance to full fine-tuning:\n\n| **Model** | **Fine-tuning Method** | **Time (50k data)** | **Performance Impact** |\n| --- | --- | --- | --- |\n| Qwen2.5-VL-7B | LoRA | 6 hours | Similar |\n|  | Full | 29 hours | – |\n| LLaVA-1.6-7B | LoRA | 13 hours | Similar |\n|  | Full | 57 hours | – |\n| LLaMA3.2-11B-VL | LoRA | 8 hours | Similar |\n|  | Full | 70+ hours | – |\n\nThese results suggest that for fine-grained visual understanding, most of the necessary capability can be adapted through low-rank adaptation. This may be because such tasks primarily require adjustments in higher-layer representations, rather than full re-training of the model backbone.\n\n**2. Hyperparameter variations across models**: Regarding the hyperparameter differences across base models, the variations largely reflect architecture-specific tuning practices. Our implementation follows the LLaMA-Factory framework, and the learning rate, batch size, and warmup ratios we use are based on commonly adopted settings within that community for each model family. The variation arises from differences in:\n\n- Model scale (e.g., 3B vs 7B vs 11B),\n- Tokenization behavior and sequence lengths,\n- Pretraining objectives and convergence speeds.\n\nWe will clarify this in the appendix and include the exact configurations for each model.\n\n> **Response to Q3: Role of CLIP Loss vs. SFT Loss**\n> \n\nWe thank the reviewer for the close reading and for raising this important point.\n\nTo clarify: although our paper discusses CLIP loss (Equations 2 and 4) as part of the general pretraining framework for MLLMs, we do not use the CLIP loss during the supervised fine-tuning (SFT) stage proposed in our method.\n\nThe presentation of the CLIP-style loss is meant to formally contextualize the limitations of existing training objectives (e.g., Eq. 2, 4, and 6) and to motivate the need for a task-aligned loss for difference reasoning. As explained in Section 4.2 and 4.3, models trained under these conventional objectives struggle with downstream fine-grained difference detection, due to both noise in supervision and objective mismatch.\n\nOur proposed SFT method addresses this by replacing the standard objective with a targeted difference-aware training loss. The actual SFT objective used in our method is shown in Equation (7):\n\n$$\\hat{R}\\_{\\text{SFT}}(\\theta) = \\frac{1}{|\\mathcal{D}\\_{\\text{edit}}|} \\sum\\_{(x\\_i, \\hat{x}\\_i, \\texttt{S}\\_{\\phi}(t\\_i, \\hat{t}\\_i)) \\in \\mathcal{D}\\_{\\text{edit}}} \\left[\nl\\_{\\text{cap}}\\left(\\texttt{Z}\\\\_{\\theta}[\\texttt{I}\\_{\\theta}(x\\_i) - \\texttt{I}\\_{\\theta}(\\hat{x}\\_i)],\\ \\texttt{S}\\_{\\phi}(t\\_i, \\hat{t}\\_i)\\right)\n\\right].$$\n\nThis loss does not include any CLIP-style component, and is purely designed to supervise the model to describe visual differences between two images in a contrastive, semantically grounded way.\n\nWe will revise the text to make it more explicit that CLIP loss is discussed only to frame prior training regimes and is not used in our final supervised fine-tuning pipeline.\n\n> **Response to Q4: Feature-Level Consistency Loss Implementation**\n> \n\nWe thank the reviewer for the careful reading of our manuscript and for raising this insightful question. We appreciate the opportunity to clarify the role of the *feature-level consistency* in our proposed method.\n\nYour understanding is correct—the supervised fine-tuning (SFT) loss requires the model to distinguish and describe the differences between two input images. Importantly, the so-called *feature-level consistency loss* is not introduced as an explicit auxiliary loss term, but rather is implicitly embedded in the design of the SFT loss itself.\n\nAs shown in Equation (7) of our paper, the loss is defined as:\n\n$$\\hat{R}\\_{\\text{SFT}}(\\theta) = \\frac{1}{|\\mathcal{D}\\_{\\text{edit}}|} \\sum\\_{(x\\_i, \\hat{x}\\_i, \\texttt{S}\\_{\\phi}(t\\_i, \\hat{t}\\_i)) \\in \\mathcal{D}\\_{\\text{edit}}} \\left[\nl\\_{\\text{cap}}\\left(\\texttt{Z}\\_{\\theta}[\\texttt{I}\\_{\\theta}(x\\_i) - \\texttt{I}\\_{\\theta}(\\hat{x}\\_i)],\\ \\texttt{S}\\_{\\phi}(t\\_i, \\hat{t}\\_i)\\right)\n\\right].$$\n\nHere:\n- $I\\_{\\theta}(x\\_i)$ and $I\\_{\\theta}(\\hat{x}\\_i)$ denote the feature-level embeddings of the original and edited images, respectively, produced by the image encoder $I\\_{\\theta}$.\n- Their difference, $I_{\\theta}(x_i) - I_{\\theta}(\\hat{x}_i)$, is passed to the text decoder $Z\\_{\\theta}$, which is trained to generate the correct description $y\\_i$ of the visual change.\n\nThis design implicitly enforces feature-level consistency: for the model to succeed in generating a correct difference description, the encoder must produce embeddings where fine-grained, semantically meaningful edits in the image space correspond to predictable and structured changes in feature space.\n\nThus, the model is encouraged to:\n\n- Produce *stable visual representations* for unedited content.\n- Encode *fine-grained differences* in a way that can be directly mapped to textual explanations.\n\nThis implicit regularization improves the model’s fine-grained reasoning ability and reduces hallucination, as shown in our experimental results across both synthetic and real benchmarks. We will clarify this point more explicitly in the revised version."}, {"review_id": "mWBvnh2NHl", "reviewer": "Reviewer_K4PJ", "summary": "The paper introduces a lightweight, training-free framework for detecting hallucinations in multimodal large language models (MLLMs), particularly vision-language models. The proposed method, Single-shot Hallucination Detection (SHD), leverages spatial attention maps and token-level entropy to determine whether the generated text is grounded in the input image. Rather than relying on costly retraining or dataset-specific tuning, SHD performs hallucination detection in a single forward pass using only internal model signals. The authors evaluate SHD across six hallucination types and multiple state-of-the-art MLLMs, showing competitive or superior performance compared to more complex baselines. The work contributes a novel, interpretable, and efficient technique for improving the reliability of multimodal LLM outputs in real-world settings.", "questions": "1.  How well does SHD generalize to hallucination detection tasks beyond the POPE benchmark and short-form VQA-style prompts? For instance, how would it perform in document-grounded generation, image captioning, or complex reasoning scenarios involving multiple modalities?\n2.  How does SHD handle borderline or ambiguous cases? Are there examples where the model flags correct outputs as hallucinations (false positives)?\n3. Even a qualitative comparison or brief discussion in related work would help position SHD among broader hallucination mitigation strategies.", "limitations": "Partially.\n\nThe authors include a limitations section that briefly acknowledges the dataset scope (POPE) and the reliance on internal model signals (token entropy, cross-modal attention). However, several critical aspects are under-addressed and would benefit from deeper reflection:\n\nSuggestions for Improvement\nBias Amplification & Model Dependence:\nSince SHD depends entirely on internal attention and entropy signals, which are products of the underlying model's learned biases, the method may inherit or even reinforce spurious correlations and visual/textual biases. A discussion on how this might affect fairness or performance across diverse demographic attributes, image styles, or non-English content would be valuable.\n\nDeployment Misuse or Overtrust:\nSHD is lightweight and attractive for production settings, but it may create overconfidence in model safety if not properly calibrated. The authors should caution against using SHD as a standalone truth evaluator, especially in sensitive domains (e.g., healthcare, legal AI), and emphasize the need for human oversight.\n\nScope of Applicability:\nThe method has only been evaluated on static images and short-form QA. Authors should note that applying SHD to other modalities (e.g., video, audio, document OCR) may not yield reliable results without adaptation or retraining, even if SHD is training-free in its current form.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality\nThe proposed method, SHD (Single-shot Hallucination Detection), is a well-engineered, training-free solution that leverages intrinsic model signals—namely token entropy and cross-modal attention—to detect hallucinations.\n\nThe experiments are extensive, covering six different hallucination categories and multiple state-of-the-art MLLMs (e.g., MiniGPT4, LLaVA, Otter). The authors use a recently proposed benchmark (POPE) and ablation studies to validate individual components.\n\nThe paper reports competitive results against several strong baselines, including detect-then-rank models, supervised discriminators, and gradient-based techniques—despite SHD requiring no additional tuning or training.\n\nClarity\nThe paper is generally well-organized and clearly written. The motivation for a lightweight, interpretable hallucination detector is well-argued in the introduction.\n\nThe authors provide intuitive visualizations of the entropy maps and cross-attention distributions to support interpretability.\n\nEach of the six hallucination categories is explained with clarity, and examples are included for illustration.\n\nSignificance\nHallucination detection in multimodal LLMs is an increasingly important area, particularly as vision-language models see growing deployment in safety-critical domains.\n\nThe proposed method has practical significance: being lightweight, interpretable, and model-agnostic, it can be applied in real-world applications without the need for costly retraining.\n\nThe ability to detect hallucinations in a single forward pass makes SHD particularly relevant for latency-sensitive systems and scalable inference.\n\nOriginality\nThe work is original in its framing of hallucination detection as a signal derived from token entropy and cross-attention, without the need for ground-truth labels or external discriminators.\n\nUnlike many prior works that rely on retraining or multiple forward passes (e.g., question answering consistency checks, multi-step verification), SHD provides a conceptually novel and computationally efficient approach.", "was_revised": false, "final_justification": "", "rebuttal": "**Overall Response**\n\nThanks for the thoughtful and constructive feedback! In response, we expanded the MED-Real set to 50 real-world samples, and included results on BLIP-3o to demonstrate architectural generality. We also clarified our hallucination mitigation strategy, feature-level supervision design, and benchmark construction process. We will include all clarifications and results in the revised version.\n\n> **Response to W1: Scale of MED-Real and Generalization to Real-World Benchmarks**\n> \n\nWe appreciate the reviewer’s suggestion to strengthen claims of real-world generalizability with more robust evaluation.\n\n**1. Expanded MED-Real evaluation**: While the original MED-Real set contained 35 real-world image pairs, we have now expanded it to 50 examples by adding 15 newly curated pairs. The table below shows the accuracy progression with increasing real-image samples:\n\n| Model with sft scalability |    acc（15） |  acc（35） | acc（50） |\n| --- | --- | --- | --- |\n| Qwen2-VL-7B | 66.67 | 82.86 | 78.00 |\n| Qwen2-VL-7B(ours) | 73.33 | 82.86 | 80.00 |\n| Qwen2.5-VL-3B | 46.67 | 54.29 | 52.00 |\n| Qwen2.5-VL-3B(ours) | 53.33 | 57.14 | 56.00 |\n| Qwen2.5-VL-7B | 60.00 | 68.57 | 66.00 |\n| Qwen2.5-VL-7B(ours) | 73.33 | 74.29 | 74.00 |\n| LLaVA-1.6-7B | 46.67 | 34.29 | 38.00 |\n| LLaVA-1.6-7B(ours) | 60.00 | 45.71 | 50.00 |\n| LLaMA3.2-11B-VL | 26.67 | 40.00 | 36.00 |\n| LLaMA3.2-11B-VL(ours) | 46.67 | 51.43 | 50.00 |\n\nAcross all model families, MED-finetuning results in clear performance gains. The improvements are consistent as the real-world sample set grows, which suggests that the model is not simply overfitting to synthetic distribution patterns.\n\n**2. General benchmark validation (Table 3, Page 8)**: To further validate generalizability, we report results on a range of diverse real-image benchmarks, including POPE, MMStar, BLINK, and MME. These datasets contain non-synthetic, real-world image distributions and diverse tasks. For example:\n\n- Qwen2.5-VL-7B (Ours) improves:\n    - POPE: 96.29 → 97.52\n    - MME: 1685.14 → 1701.87\n    - BLINK Visual Corr: 33.72 → 37.79\n\nThese results show that MED improves real-world performance even on unrelated tasks, indicating robust, transferable fine-grained reasoning capabilities — not simply domain overfitting.\n\nWe will include this clarification and the expanded table in the revision to emphasize MED’s impact on generalizable multimodal understanding.\n\n> **Response to W2: Hallucination Risk from Qwen-VL and Gemini**\n> \n\nWe thank the reviewer for raising the important concern about potential hallucinations from Qwen2.5-VL-72B and Gemini in our data generation pipeline. We fully agree that vision-language models are not immune to hallucination, which is why our pipeline is explicitly designed to minimize and isolate such errors through two key strategies: task decomposition and multi-stage filtering.\n\n**1. Task decomposition: minimizing hallucination through controlled prompts**: Instead of relying on the model to generate free-form captions or edits, we break the data generation into small, grounded subtasks:\n\n- Qwen2.5-VL is only asked to revise or extend existing captions, based on concrete visual prompts (e.g., “add a red ball”).\n- Edited captions are generated with strong contextual guidance, including the original caption and the specified edit type.\n- Difference descriptions are produced by a separate text-only model (Qwen3-32B) to avoid compounding visual noise.\n\nThis decomposition ensures that each model step is *tightly scoped and grounded in the image*, significantly reducing hallucination risks.\n\n**2. Multi-stage filtering: detecting and removing faulty samples**: To further control quality, we apply multiple filtering layers:\n\n- Qwen2.5-VL first evaluates raw image-caption pairs via a structured quality rubric, discarding low-alignment or low-editability samples.\n- After editing, we use CLIP-based similarity filtering to ensure visual differences are minimal and semantically plausible.\n- Finally, we perform manual verification on 1,000 image pairs, covering all 11 edit types, to identify and eliminate rare hallucination patterns and refine prompt templates. According to the manual verification results, we designed heuristic rules to filter out low-quality data.\n\nThrough this process, we ensure that both Qwen-VL and Gemini are used only within narrow, controllable contexts, with human- and CLIP-based safeguards in place. We will add clarification and provide statistics (e.g., instruction rejection rates) in the revised version to further support the robustness of our pipeline.\n\n> **Response to W3: Meta-Text Artifacts and Benchmarking Noise**\n> \n\nWe thank the reviewer for raising the concern about meta-text phrases such as *“in the second caption”* or *“is described”*, which could introduce subtle benchmarking noise or false negatives.\n\nWe clarify that such phrasing appears only in intermediate textual difference descriptions generated during Step 3 of our data construction pipeline (see Section 3.2).\n\nBefore creating the final MED benchmark, we perform a systematic transformation to ensure questions are fully visually grounded:\n\n- All contrastive difference sentences are rewritten into QA-style prompts that refer exclusively to visual content, not to how things are described in text.\n- Each QA item includes four answer choices, and we perform manual verification to ensure:\n    - Only one answer is visually correct.\n    - Distractors are visually plausible but wrong.\n    - No options rely on or reference the structure of captions or linguistic cues like “caption” or “mention”.\n\nThis transformation ensures that no meta-textual language appears in the final benchmark, and that model evaluation remains image-grounded. We will make this clarification explicit in the final version and include an illustrative before/after example to demonstrate how these artifacts are resolved.\n\n> **Response to W4: Broader Evaluation Across Model Architectures**\n> \n\nWe thank the reviewer for highlighting the importance of evaluating across diverse model families.\n\nWhile our primary evaluations focused on Qwen, LLaVA, and LLaMA-style vision-language models, we emphasize that our method is model-agnostic and readily applicable to other architectures. To demonstrate this, we additionally applied our fine-grained supervision method to BLIP-3o, a strong open-source model architecturally distinct from the LLaVA/Qwen family.\n\n| Model | avg | obj | attr | scene | spatial | action | part | count | differ | compar | neg | univ |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| BLIP-3o-8B | 44.24 | 46.15 | 50.00 | 69.23 | 50.00 | 37.50 | 31.25 | 44.44 | 41.67 | 31.58 | 50.00 | 42.86 |\n| BLIP-3o-8B(ours) | 50.90 | 38.46 | 62.50 | 46.15 | 42.86 | 62.50 | 37.50 | 66.67 | 50.00 | 52.63 | 35.71 | 57.14 |\n\nThe finetuned BLIP-3o model shows significant improvements across many dimensions, especially in *attribute*, *action*, *counting*, and *difference* reasoning. This supports our claim that the proposed fine-grained visual edit supervision enhances a model’s reasoning ability regardless of its backbone architecture.\n\nIn total, we evaluate and improve models from four diverse model families:\n\n- Qwen series (VL-7B / VL-2.5-7B / 3B)\n- LLaVA series (1.6-Vicuna)\n- LLaMA-Vision (3.2-11B)\n- BLIP-3o (Transformer with Querying Transformer head)\n\nWe will include this extended evaluation in the revised version to reinforce the generality and adaptability of our approach across architectures.\n\n> **Response to W5: Image Quality and Clarity**\n> \n\nThank you for the valuable feedback. In the revised version, we will provide higher-resolution images and add reference captions to enhance clarity and interpretability.\n\n> **Response to Q1: Feature Consistency Loss and Hallucination Reduction**\n> \n\nThank you for the insightful question. Our feature consistency loss reduces hallucinations by improving visual representations prior to decoding, not by modifying the autoregressive decoding itself.\n\nSpecifically, the loss encourages stable and semantically aligned embeddings for minimally edited image pairs, which enhances the fidelity of the input to the language decoder. This grounding helps reduce hallucinations during text generation by ensuring that the decoder is conditioned on more reliable visual features.\n\nAs shown in Table 3, even updating only the vision encoder leads to improved performance on perception-heavy tasks (e.g., Count, Coarse), highlighting that better visual representations alone significantly reduce errors. Full gains are achieved with joint tuning, but the primary hallucination mitigation stems from the representation level, not decoding.\n\nIn summary, the feature consistency loss acts at the representation level, ensuring the visual embeddings are robust and well-grounded. This improved representation serves as a more reliable foundation for the standard autoregressive decoder, which then produces more accurate and less hallucinatory text as a downstream effect. We hope this clarifies the interaction. We thank the reviewer again for the valuable feedback."}]

Author rebuttal:

Meta-review: Summary: Introduces MED (50K minimally edited pairs) + MED-Bench, and an SFT feature-consistency objective to improve fine-grained difference detection and reduce hallucinations.  Also gives gains on POPE/MMStar/BLINK/MME vs strong baselines.

Strengths: timely focus on fine-grained hallucination, carefully filtered dataset/benchmark with manual checks, and empirical gains with released code/prompts/configs.

Weaknesses: complex multi-model pipeline with concurrent reproducibility burden.  Small real-image set and initially narrow model coverage.

(d) clear dataset+benchmark contribution and an effective lightweight training signal that measurably reduces hallucination across tasks/models.

(e) Discussion/rebuttal summary: reviewers pressed on pipeline complexity, instruction/annotation quality, model diversity, and caption artifacts; authors responded to address many concerns.
Decision: Accept (poster)