VaPR - Vision-language Preference alignment for Reasoning
Authors: Rohan Wadhawan
Fabrice Y Harel-Canada
Zi-Yi Dou
Suhaila Shakiah
Robinson Piramuthu
Nanyun Peng
Venue: COLM 2025 (2025)
Keywords: Vision Language Models
Preference Optimization
DPO
Data Generation
Reasoning
TL;DR: VaPR, a hard-negative preference dataset that mitigates stylistic and length biases in AI feedback, enabling improved reasoning and robustness in preference finetuned (DPO) vision-language models across ten benchmarks.

Abstract: Preference finetuning methods like Direct Preference Optimization (DPO) with AI-generated feedback have shown promise in aligning Large Vision-Language Models (LVLMs) with human preferences. However, existing techniques overlook the prevalence of noise in synthetic preference annotations in the form of stylistic and length biases. To this end, we introduce a hard-negative response generation framework based on LLM-guided response editing, that produces rejected responses with targeted errors, maintaining stylistic and length similarity to the accepted ones. Using this framework, we develop the VaPR dataset, comprising 30K high-quality samples, to finetune three LVLM families: LLaVA-V1.5, Qwen2VL \& Qwen2.5VL (2B-13B sizes). Our VaPR models deliver significant performance improvements across ten benchmarks, achieving average gains of 6.5% (LLaVA), 4.0% (Qwen2VL), and 1.5% (Qwen2.5VL), with notable improvements on reasoning tasks. A scaling analysis shows that performance consistently improves with data size, with LLaVA models benefiting even at smaller scales. Moreover, VaPR reduces the tendency to answer "Yes" in binary questions - addressing a common failure mode in LVLMs like LLaVA. Lastly, we show that the framework generalizes to open-source LLMs as editors, with models trained on VaPR-OS achieving ~99% of the performance of models trained on VaPR, which is synthesized using GPT-4o. Our data, models, and code can be found on the project page https://vap-r.github.io/vap-r/

Published as a conference paper at COLM 2025
VAPR - Vision-language Preference alignment for Reasoning
Rohan Wadhawan's Fabrice Harel-Canada', Zi-Yi Dou', Suhaila Shakiah?, Robinson Piramuthum², Nanyun Peng'
!Dpparmeet of Computer Science, University of California Los Angeles, USA 2 EAmazon.com, Inc., USA
Abstract
Preference finetuning methods like Direct Preference Optimization (DPO) with AT-generated feedback have shown promise in aligning Large Vision- Language Models (LVLMs) with human preferences. However, existing techniques overlook the prevalence of noise in synthetic preference annota- tions the form of stylistic and length biases. To this end, we introduce a hard-negative response generation framework based on LLM-guided response editing, that produces rejected responses with targeted errors, maintaining stylistic and length similarity to the accepted ones. Using this framework, we develop the VAPR dataset, comprising 30K high-quality sam- ples, to finetune three LVLM families: VLAVA- Qwen2VL. & Qwen2.5VL (2B-13B sizes). Our VAPR models deliver significant performance improve- ments across ten benchmarks, achieving average gains of 6.5% (LLaVA), 4.0% (Qwen2VL), and 1.5% (Qwen2.5VL), with notable improvements on reasoning tasks. A scaling analysis shows that performance consistently improves with data size, with LLaVA models benefiting even at smaller scales. Moreover, VAPR reduces the tendency questions add common failure mode in I.LMMs like LA. Lastly, we show that the framework generalizes to open-source LLMs as editors, with models trained on VaPR-OS achieving 99% of the performance of models trained on VAPR, which is synthesized using GPT-4o. Our data, models, and code can be found on the project page https://vap-tae /vtp.b..tt
answer "Yes" in binary
Introduction
Recent advances in Large Vision-Language Models (LVLMs) have greatly enhanced their ability to perceive, reason, and generate open-ended responses to instructions (Liu et al., b023c;;; 2024; Ye al., 2024; Dai et al., 2023; Awadalla al., 2023; Tong et al., 2024; Chen et al., 2024b; Team, 2024a; Chen et al., 2023; Zhu et al., 2023; Wang et al., 2024a). Despite the development, LVLMs often face challenges in vision-language alignment and reasoning, resulting in linguistically plausible texts that either contradict the visual context or lack logical reasoning. Several studies (Sun et al., 2023; Yu et al., 2024a;b; Zhou et al., 2024a;b; Deng et al., 2024; Li et al., 2023c; Zhao et al., 2023; Lee et al., 2023b; Sarkar et al., 2024) have focused on improving the modality alignment capabilities of instruction-tunett LVLMs through preference fine-tuning methods, including reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Schulman et al., 2017), AT feedback (RLAIF) (Lee et al., 2023a), and direct preference optimization (DPO) (Rafailov al., 2024).
Early preference finetuning methods for LVLMs primarily incorporated human feedback (Sun et al., 2023; Yu et al., 2024a), but the high costs and limited scalability of this approach have led to growing use of Al-generated feedback and synthetic preference dataset creation (Zhou al., 2024a; Deng et al., 2024; Yu al., 2024b; Zhou et al., 2024b; Lee al., 2023b; Zhao et al., 2023). These synthetic preference datasets have demonstrated advantages over human-annotated versions, particularly in maintaining truthfulness (Ivison et al., 2024b) and enhancing modality alignment (Zhou et al., 2024a; Deng et al., 2024; Yu et al., 2024b;
' *oorresponaance to: rohanwaddhavar/@gmaal.com

---

Published as a conference paper at COLM 2025
Zhou et al., 2024b; Lee et al., 2023b; Zhao et al., 2023). However, our analysis reveals that these methods often overlook inadvertently injected "noises", including biases in length and style (Zhao et al., 2023) (see Table 1). As DPO has been found to exploit both length (Park et al., 2024; Lu et al., 2024) and stylistic biases (Yu et al., 2024a; Zhao et al., 2023; Hong et al., 2024; Yu et al., 2024b), this noise in the dataset can consequently undermine the alignment process.
To improve LVLMs' alignment and reasoning, we propose hard-negative preference data generation framework and create VaPR, dataset of 30K high-quality samples derived from the LLaVA-665K SFT dataset (Liu al., 2023b). Each preference sample in VAPR pairs ground truth response with generated hard-negative response that preserves style and length while introducing targeted perturbations to ensure deliberate misalignment (see Fig. 1). While existing approaches (Zhou et al., 2024a; Wang et al., 2024b; Zhou et al., 2024b; Yu et al., 2024b) rely on VLMs (the same or a large oracle VLM) to generate and/ score sampled responses, we frame hard-negative response generation as ground truth response editing using an LLM -implemented via constrained data generation. This editing approach leverages LLMs' semantic understanding of text, which is typically more reliable than the image-text comprehension of VLMs (Guan et al., 2024). We provide the LLM with the instruction, ground truth response, and VL task-specific information (e.g., spatial reasoning, object attributes), and prompt it to perturb specific spans of the ground truth response. Unlike prior works, we use task-specific information guide the editor in minimally editing the ground-truth response, injecting semantic errors that make the rejected response incorrect for the task while preserving style and length (see Table 1). This prevents models from exploiting superficial cues during preference optimization (64.2, & 54.3.1). We also ensure that the VAPR corpus covers diverse instructions perception (e.g., object existence, attribute, size, color, environment), reasoning (e.g., counting, spatial, comparison), and captioning- to help models learn generalizable preferences. For our main results, we use GPT-4o (OpenAI, 2024) as the LLM editor and refer to the approach as VAPR. Lastly, to evaluate the generalizability of the pipeline across different LLMs, we also conduct an ablation experiment using Qwen3-32b (Yang et al., 2025) as the editor, referred to VxPR-OS (open-source), to distinguish it from the GPT-4o based VAPR dataset.
Instruction: How many planes are visible the image?
Instruction: Where is the man standing in relation to the baby clephant?
Chosen Response: There are four planes
The man is standing next to the baby elephant the water.
Chosen Response:
visible in the image.
VAPR Response: There are six planes visible in the image.
VAPR Response: The
standing far away from the
baby elephant, near the edge the water.
Response with length bias: There are seven planes visible in the sky in the image, each leaving bright white trail behind.
Response with length bias: The man is standing far from the baby elephant, attempting to throw water using bucket while standing at the edge of the water. Response with style bias: The elephant is standing in the water, away from the man, who on the shore.
Response with style bias: Five planes can be seen in image.
Figure 1: Examples from the VAPR hard-negative generation framework show instruction, image, chosen response, and three rejected variants. (a) fine-grained perception and counting capability, and (b) spatial reasoning. VAPR introduces targeted error modifying only task-relevant spans while length-biased rejections add verbose description, and style-biased ones alter style and content. Blue highlights relevant spans in chosen response, green shows VAPR perturbations, and red indicates stylistic or length-biased edits.
To assess the effectiveness of the proposed framework, fine-tune models from the LLa V1.5-Instruct(7B, 13B), Qwen2VL-Instre (2B, 7B), and Qwen2.5VL-Instrruc (3B, 7B) families on the VAPR dataset and evaluate them across ten diverse benchmarks. These span open-
2

---

Published as a conference paper at COLM 2025
ended instruction following, vision-centric reasoning, academic and mathematical tasks, hallucination detection, and adversarial robustness. VaPR models outperform baselines on 8 out of 10 benchmarks, with average gains of 6.5% for LLaVA-V1.5, 4% for Qwen2VL, and 1.5% for Qwen2.5VL. Notably, VaPR yields consistent improvements on comprehensive benchmarks like SEED (Li et al., 2023a) and MMStar (Chen et al., 2024a), vision-centric reasoning benchmarks such as CV-Bench (Tong et al., 2024) (counting and spatial reasoning), and adversarial reasoning benchmarks like NaturalBench (Li et al., 2024a), which test compositional Visio-linguistic reasoning. Interestingly, improvements are also observed in textual and math reasoning despite no explicit training on such data. scaling analysis reveals that performance scales with dataset size, with LLaVA-v1.5 models benefiting even with small preference tuning data budget, and Qwen2VL Qwen2.5VI. variants improving at larger scales.
We further compare VAPR to other preference datasets and empirically analyze how they function under DPO optimization. Our findings highlight the influence of stylistic and length differences in preference data, which can lead to spurious learning signals during training. VAPR mitigates this issue by generating hard-negative rejected responses, which are stylistically and length-wise similar to the chosen ones. We also observe that VaPR reduces the tendency answer "Yes" in binary questions, addressing a common issue in LVLMs like LLaVA (Li et al., 2023d; Liu et al., 2023a; Guan et al., 2024; Li et al., 2024a). Lastly, our ablation using Qwen3-32b to generate VaPR-OS show that open-source models can follow the same prompting strategy, produce targeted perturbations, and yield models that outperform the base instruct model and achieve 99% performance of the VaPR models.
The contributions of this work are fourfold:
We propose VAPR, a hard-negative generation framework based on LLM-guided response editing that constructs synthetic preference data with reduced stylistic and length biases. Using this framework, we create a 30K sample high-quality dataset.
We fine-tune LLaVA-V1.5, Qwen2VL, and Qwen2.5VL. on VAPR and evaluate them across ten benchmarks. VAPR outperforms SFT and preference-ture out of 10 benchmarks, with average gains 6.5% (LLaVA), 4.0% (Qwen2VL)) and 1.5% (Qwen2.5VL.). Scaling analysis shows LLa VA benefits from smaller-scale data, while Qwen models require larger-scale data to improve.
baselines on 8
Our analysis shows that VAPR enhances performance on reasoning tasks like adversarial, spatial, counting, & textual, reduces the overuse of "Yes" in binary questions, and avoids spurious learning signals by mitigating stylistic and length biases present in prior preference datasets.
Our ablation with Qwen3-32b as the LLM-editor, demonstrates that the framework generalizes well to open-source LLMs, producing similarly effective hard negatives (VaPR-OS) and preference-tunee models achieving 99% of the performance of models trained on VAPR. Our data, models, and code can be found on the project page https:/ / rvtth..uu//...ttuu.//o
2 Preliminaries
Large Vision Language Models (LVLMs) enhance the capabilities of Large Language Models (LLMs) by adapting them to multimodal tasks. This enables the model to predict the probability distribution for the next token a sequence given multimodal inputs. Specifically, given an input pair x (x), comprising image Xy and instruction text Xt, the LVLM generates a text response y. Preference optimization has emerged as a promising technique for fine-tuning language models to align their outputs with desired outcomes, which we briefly overview in this section.
2.1 Direct Preference Optimization
Given a prompt x (X,, large vision language model governed by policy ny can yield a conditional distribution To (y x),, where y is the generated text response. To train using
3

---

Published as a conference paper at COLM 2025
preference data, we define the dataset {(x11, V/w Vi (0 )) where yw and yi represent the more and less preferred responses, in our case, chosen and hard-negative rejected response, respectively, for a given input x(1) Preference optimization leverages this dataset to fine-tune models effectively.
Objective Direct Preference Optimization (DPO) calculates the probability of preferring Vw over y1 as:
pyy y1) ((((x, rxxx,y))),
where o() is the sigmoid function. The loss function for DPO can be written follows:
Tolyw x)) [x)
Te (yn x) Tee (1/ x) ).
(T Teet) (x
/D MD
logor (olog
a log
(1)
Thet
where (y represents the reference policy, typically the model after SFT.
Preference Dataset There are different ways to construct the preference dataset. For example, Sun al. (2023) employs human annotations, which are of high quality but is difficult to scale. On the other hand, AT annotation with LVLMs like GPT-4V (OpenAI, 2023) scales better but risks inconsistencies and hallucinations (Zhao al., 2023; Hong et al., 2024). Our focus is to construct scalable framework for preference dataset construction while ensuring its quality.
3 VAPR Dataset Construction
Categorize Sample
Remove data with undesirable properties
Categorize tasks keywords containa ("abare")
negatives lask-spacific prompts
MuttileeChoice Short Response
Enstrsctions] f0cound Truch]
OD0
[Pens Mett] loget COnNTS
Object
Colors
Counting
Caption
D+D+
Back-
Referential
Size
Spotial
General
Figure 2: VAPR: three-stage pipeline that generates 30K hard-negative preference pairs from LLaVA-v1.5-6655 SFT (Liu et al., 2023b). Stage 1: Filter out irrelevant samples MCQs). Stage 2: Categorize remaining samples based on task. Stage 3: Use task-specific prompts (with optional penalty lists) to produce stylistically and length-wise similar but content-distinct negative responses.
We detail the design principles and construction processes of our preference dataset. Our main idea is to construct hard negatives that are stylistically and length-wise similar to the ground-truth samples.
3.1 Design Principles
We adhered two key principles for constructing the VAPR dataset:
Task Diversity The dataset ensures near-balanced distribution across wide range of tasks (see Fig. 3). It consists of foundational vision-language capabilities, including perceptual (e.g., object recognition, object attribute recognition color, size, etc., and

---

Published as a conference paper at COLM 2025
background understanding) and reasoning (e.g., spatial, counting, comparative). We also incorporate tasks combining perception and reasoning like captioning, world knowledge reasoning, and referential VQA (eg. region-level perception reasoning). This diversity is designed to improve the model's comprehensive capabilities.
Hard Negative Rejected Responses Inspired by the role of hard negatives in enhancing vision-language compositionality in image-text pretraining (Radford etal., 2021; Yuksekgonul et al., 2022; Hsieh et al., 2023), we generate hard-negative responses. These are synthesized by introducing targeted perturbations to high-quality SFT ground truth responses while maintaining stylistic and structural similarity (e.g,, length), thereby resulting in incorrect responses given task.
3.2 Sourcing & Processing
Sourcing We build preference samples using the high-quality LLaVA-665K SFT dataset (Liu et al., 2023b), which has broad task coverage.
Filtering The data undergoes careful processing to filter tasks unsuitable for hard-negative generation. We exclude tasks that lack sufficient training signals to address vision-language misalignments, such as text-only tasks and simple response types (e.g., MCQ or bounding box prediction). However, we retain "Yes/No" response instruction on the existence of objects, attributes, and reasoning (e.g., count, spatial, comparative), as they correspond to the key vision-language capabilities, and convert them to extended responses to better suit preference optimization. We also filter out OCR instructions, prior works emphasize the need for larger input image resolutions and fine-grained visual perception as driving factors in improving OCR performance (Wadhawan et al., 2024; Li et al., 2024b; Yu et al., 2024c).
Categorization We categorize the filtered corpus into ten task categories using task-specific keywords (see Fig. 2). We subsample portion of the SFT dataset from each category for hard-negative response generation. Notably, for binary response instructions ("Yes") /"No""NN we enforce an equal distribution of "Yes" and "No" responses mitigate bias towards "Yes," which arises from the predominance of affirmative instructions in the original SFT dataset (Li et al., 2023d; Guan al., 2024; Liu al., 2023a). See Appendix $B.1 for details.
3.3 Generation Pipeline
The VaPR framework generates hard-negative responses by editing ground truth (instruction, response) pairs from our filtered & categorized SFT subset using GPT-4o. This process is guided by two key components: conditioning information and perturbation diversity.
Conditioning Information Task-specific prompts (e.g., attributes, spatial reasoning, count- ing) guide edits to ensure responses remain fluent yet incorrect (see Fig. 2). These were preferred over general prompts, which often introduced irrelevant changes given the task or added verbosity or style changes, contributing to noise in the dataset.
Diversity of Perturbations Most categories (e.g., object type, size, existence) produced diverse outputs with zero-shot prompting, while for others (e.g., color, counting, captioning), introduced a penalty list, periodically updating with commonly used perturbation values to prevent reuse. In captioning, we first extracted dimensions (e.g., color, spatial relation) and applied random perturbations across them, using a penalty list to encourage variety of dimensions and perturbation values. See Appendix $B.2 for details.
3.4 Dataset Analysis
We perform statistical analysis of the VAPR dataset, with task-wise distribution shown in Fig. 3. To assess annotation quality, conducted human evaluation on 500 stratified samples across task categories, finding 97% alignment with hard-negative criteria and 86% inter-annolator agreement (IAA) using Fleiss' kappa (see Appendix $B.4).
5

---

Published as a conference paper at COLM 2025
General Reasoning Referentia dereena VOA
Caption
3.3%
13.3%
Existence
Table 1: Comparison of the VAPR Preference Dataset with Related Works. Lower LD and smaller length differences indicate higher stylistic Weshow the percentage of samples where chosen responses are longer (chosen >> rejected) or shorter (rejected > chosen).
6.7%
object
23.3%
13.3%
Counting
Ours HA-DPO FOVDD REAIEV
CSR
Calor
3.33%
Overall Samples (%)
(21
(41, 59) (24, (45,55)
(38,
Linguistic Similanity
89
97
Avg. Token Length Diffevence
Length Difference
() (15, 20) (27. (17,14) (23,29)
Background
Spatial
Figure 3: VAPR task distribution
To quantify stylistic and length similarity, we compare VAPR against prior datasets HA- DPO (Zhao et al., 2023), POVID (Zhou et al., 2024a), RLAIF-V (Yu et al., 2024b), and CSR (Zhou et al., 2024b). Linguistic similarity is measured via word-level Levenshtein distance, where lower values indicate targeted content edits; higher values reflect broader stylistic variation. Length similarity is computed as token-level sequence length differences. We also report the proportion of samples where chosen or rejected responses are longer, for both short-form (e.g., VQA) and long-form (e.g,, captioning) tasks (see Appendix $B.3). Table shows that VAPR has the lowest Levenshtein distance and token length difference, reinforcing the hard-negative nature of its rejections. In Section 54.3.1, we show that larger stylistic and length discrepancies in other datasets lead to premature saturation in reward accuracy, suggesting reward hacking behavior that is mitigated with VAPR.
Experiments
4.1 Experimental Setup
4.1.1 Models & Baselines
We preference-tune LLaVA-1.5 (Liu et al., 2023b) (7B and 13B), Qwen2VL-Instruct (Wang et al., 2024a) (2B and 7B), and Qwen2.5VL-Instruue (3B and 7B) using Direct Preference Optimization (DPO) (Rafailov et al., 2024). LoRA finetuning (Hu et al., 2021) is employed during the preference learning phase. Models preference-tuned on the VAPR dataset, which comprises 30K samples, are referred to LLaVA-VAPR, Qwen2VL-VAPR, and Qwen2.5VL. VAPR, respectively. Additional details on the training setup are provided in Appendix SC.1.
We compare the performance of the VAPR models with two baselines: base instruct models and Supervised Finetuned (SFT) variants of base instruct models on the VAPR dataset. We also compare the VAPR models with prior works involving different preference dataset generation techniques Human (LLaVA-RLHE (Sun et al., 2023)), AI annotations using a large closed-source model (HA-DPO (Zhao al., 2023), POVID (Zhou et al., 2024a)),, self-generd celf-generated AI feedback (SIMA (Wang et al., 2024b), RLAIF-V (Yu et al., 2024b), and CSR (Zhou et al., 2024b)) - using models made publicly available by the respective works.
4.1.2 Evaluation Benchmarks
We evaluate VAPR models across ten benchmarks covering diverse skills: Open-ended QA -LLaVA-intthe-will (LLaVAW) W) (Liu et al., 2023c), ConTextual (Con') (Wadhawan et al., 2024), & MM-VET (MMV) (Yu et al., 2023); Comprehensive vision benchmarks (reasoning & perception) SEED Bench (SEED image split) (Li et al., 2023a) & MMStar (MMS) (Chen

---

Published as a conference paper at COLM 2025
Table 2: Performance comparison of
Qwen2VL-Instruct,
SFT and DPO models finetuned on VaPR, and other preference datasets across 2B, 3B, 7B, and 13B parameter sizes ten benchmarks, Higher scores indicate better performance across all benchmarks, with the highest score for each benchmark highlighted in bold. All models are evaluated using publicly available checkpoints, adhering to evaluation parameters prescribed by the benchmarks. represents results that show statistically significant improvement via bootstrap resampling (95% CT).
Row METROD
LLAVA"
CoNT
MMV
MV
MMMU
LLaVA-155-B
6488
16.8
30.9
66.2
62.1
30.1
35.4
32.6
85.9
12.7
Fact-RLHI
10.5
55.7
26.4
28.0
30.7
HA-DPO
69.8
14.2
30.9
64.6
62.0
29.7
35.5
32.9
83.5
13.5
POVID
70.1
18.6
31.8
66.3
62.3
30.6
35.6
33.3
13.1
SIMA
66.4
18.0
31.4
66.1
60.1
30.1
35.6
32.5
85.8
12.6
RLAIEY
73.1
14.1
29.3
65.8
39.8
30.3
34.2
33.7
78.9
12.9
CSR
69.5
15.8
65.9
61.1
30.5
35.6
32.2
13.4
VAPR SET (ours)
64.0
16.0
30,0
65.6
39.5
29.5
34.9
31.7
83.1
12.2
VAPR DPO (ours)
76.2
20.6'
32.9
66.7!
62.9'
30.8
35.7
34.7!
85.4
14.5
10 11AVA-L5--13B
72.3
18.6
36.7
68.2
62.5
30.7
36.1
33.8
86.0
14.9
11
Fact-RLHF
70.4
15.6
61.1
31.2
28.0
32.1
12
SIMA
70.7
19.2
36.0
68.0
62.6
30.8
35.4
34.0
86.0
14.6
13
CSR
74.2
17.8
35.6
68.2
62.4
31.3
35.9
33.9
86.8*
14.9
(ours)
17.4
67.2
15
VAPR DPO (ours)
80.51
21.2
37.3
68.77
64.6"
32.3'
35.8
35.6'
86.3
18.2
16
Owen2VL-2B
83.2
27.7
73.6
66.5
38.7
17
VAPR SFT (ours)
69.1
22.1
43.3
70.3
54.5
46.4
36.6
41.5
85.5
15.1
18
VAPR DPO (ours)
88.1
34.8
54.1
74.0"
69.0*
50.9
39.2
43.7
88.3!
25.7!
19
Qwen2VL-7B
92.5
39.7
62.1
76.4
75.7
57.5
30.7
56.7
87.3
30.8
20
VAPR SFT (ours)
79.6
36.6
52.6
73.9
67.4
56.3
47.7
52.9
86.2
23.7
21
VAPR DPO (ours)
96.2
43.9*
76.8"
76.3'
58.2
50.0
57.8'
32.5'
22
Qwen2.5VL-3B
98.1
37.2
67.3
75.0
71.5
52.5
45.7
54.7
25.4
23
VAPR SFT (ours)
79,8
33.4
50.1
70.9
60.5
48.6
43.1
48.5
84.8
21.2
24
VAPR DPO (ours)
97.1
40.3
67.4
75.5*
72.7'
53.4
44.9
56.1!
86.4
26.3*
25
Qwen2.5VL-77
101.4
53.3
71.0
77.7
80.1
58.6
50,9
61.9
86.3
32.0
26
VAPR SFT (ours)
80.2
38.9
39.0
74.9
69
56.8
48.1
36.1
82.8
23.8
(ours)
77.8
50.6
et al., 2024a);, Vision-centric reasoning (spatial reasoning, counting) CV Bench (CV) (Tong et al., 2024); Hallucination & Adversarial reasoning Pope (Li et al., 2023d), NaturalBench (NB) (Li et al., 2024a); Academic & Math Reasoning- MathVista Vista (MV) (Lu et al., 2023) & MMMU (Yue al., 2024) (benchmark details provided in Appendix SC.2).
4.2 Results
Table 2 presents the core experimental findings. VAPR models outperform baseline and prior preference-tunest models on out benchmarks and remain competitive on the rest. iLAVA-VAPR 7B and 13B achieve average gains of 7% and 6%, respectively, while Qwen2VI--VARR 2B and 7B improve by 5% and 3%. Despite Qwen2.5VL being strong VAPR yields improvements of 2% (3B) and 1% (7B), with
baseline, preference finetuning gains concentrated in vision-centric benchmarks (CV-Bench, MMStar, BeecBench) and the adversarial benchmark (NaturalBench).
These improvements in vision-centric and adversarial benchmarks are consistent across all VAPR models, indicating that preference finetuning on VAPR strengthens visio-linguistic compositionality- particularly in perception (e.g,, fine-grained object-attribute identification) and reasoning tasks such as spatial relationships and counting (see Appendix $D.2). Notably, improvements also extend to textual and mathematical reasoning benchmarks (ConTextual and MathVistal despite not being explicitly trained on OCR, textual reasoning, or math tasks, VaPR models achieve strong performance in these areas. We attribute this to their enhanced fine-grained perception, spatial reasoning, and counting capabilities, which is consistent with prior work (Fu et al., 2024). In contrast, all models show limited or improvement on MMMU, aligning with prior findings (Ivison et al., 2024a) that preference optimization primarily enhances alignment and truthfulness, rather than factuality.
7

---

Published as a conference paper at COLM 2025
Lastly, we observe that While VaPR demonstrates strong effectiveness under DPO optimiza- tion, supervised finetuning (SFT) on the same dataset does not yield comparable gains. Specifically, SFT results in gentle performance degradation for LLaVA, and more for the Qwen2VI. and Qwen2.5VI. families. This outcome suggests that SFT on a relatively small dataset like VAPR- particularly in comparison to the original large-scale SFT corpora may lead to overfitting, especially in models with strong pretrained priors like Qwen2.5VL, thereby limiting their generalization capabilities. In contrast, preference optimization with carefully constructed, length and stylistically similar hard-negative pairs enables more robust and generalizable representation learning. These findings indicate that the gains observed in VAPR models stem not from the selection of samples, but from the preference signal induced by VaPR preference pairs.
4.3 Analysis
4.3.1
Preference Dataset Comparison
We compare VAPR with two alternatives: (1) POVID, which uses GPT-4V to generate rejected responses given SFT sample, and (2) SIMA, a
method that composes preference pairs from greedy and sampled outputs generated and critiqued by the same LVLM to be preference tuned. VaPR consistently outperforms both LLaVA and Qwen2VL families, showing 4-6% average gains over POVID and SIMA.
To analyze how the different datasets affect the DPO optimization process, let us rewrite the loss:<pro = log 0 (a (A Artt), where A = log 7&p (Vw x) log (V x) and Aref = log nef x) - log [yt x). We observe that POVID exhibits higher Aref than VAPR despite similar chosen log-probabilities (see Fig. 4a), indicating that its rejected responses are less likely under the reference model most likely due to greater stylistic and length differences (see Table 1). We further observe that the POVID model rapidly attains reward accuracy of (see Fig. 4b), suggesting that the model may be overfitting based on preference signals derived from length and stylistic differences, unlike VAPR models, which improve more gradually without saturating, indicating reduced overfitting due to exposure to more challenging preference pairs.
SIMA, on the other hand, shows Are 2 0 (see Fig. 4a), with chosen and rejected responses often near-identical - including 20% duplicates. In such cases, the loss depends entirely on Ag, removing reference- siicceeggieedd regularization and amplifying noisy or weak signals. This leads to poor reward accuracy 50%) and degraded generalization (see Fig. 4). While self-preference methods like CSR and RLAIF-V can mitigate some issues via multi-step scoring, like POVID, they can still reward hack due to length and stylistic differences (see Table 1). Detailed methodology, results & analysis provided in Appendix SD.3.
VAPR addresses both issues by explicitly generating hard negatives that are stylistically and length-wise similar to positives, ensuring DPO learns from content differences. In low-resource settings where high-quality SFT data is scarce, VAPR can complement self- preference methods by using confident responses from approaches like CSR or RLAIF-V as positives and generating hard negatives via the VaPR pipeline, allowing the curation of high-quality preference data in an unsupervised manner a promising future direction
4.3.2 Scaling analysis
We investigate the impact of dataset size on model performance by conducting data scaling analysis using three training budgets: 3K, 10K, and 30K samples from the VAPR dataset. Detailed results are provided in Appendix $D.1. As shown in Fig. 5, all VAPR models exhibit improved performance with increasing data. Interestingly, LLaVA- V-PP models achieve substantial gains even at the lowest data budget (3K), with diminishing returns at higher scales. In contrast, Qwen2VL and Qwen2.5VL models- show more modest improvements at 3K but benefit more dataset size increases. This trend aligns with their stronger pretrained priors compared to LLaVA-11.5, which may require more supervision to meaningfully shift under preference tuning.
models being stronger base instruct models-
8

---

Published as a conference paper at COLM 2025
Nverage Log Probebilities under Reference Model
Reward Accuracy (EMA Smeothed)
0.8
-50
-60
0.2
Chosen
VolR
Rejected
0.0
SIMA
POVID
VaPR
200
600
800
1200
1400
Datases
Training Steps
(a) Reference model log-probabillties
(b) Reward accuracy trends
Figure 4: Comparison of preference datasets. (a) Average reference model log-probabilities for chosen vs, rejected responses across VAPR, SIMA, and POVID lower values indicate lower reference
likelihood. (b) Reward accuracy trends over training steps show that SIMA improves gradually while POVID saturates quickly.
4.3.3 VAPR models do not overconfidently say "Yes" " to "Yes/No" questions
In this section, we examine the response patterns large vision-language models (LVLMs), which tend to answer "Yes" more frequently than "No" for binary "Yes/ No" questions (Li et el., 2023d; 2024a; Liu al., 2023a; Guan al., 2024). Specifically, we analyze model outputs NaturalBench, an adversarial benchmark comprising paired instructions and paired images. As shown in Fig. 6, base SFT variants of LLa VA-v1.5, Qwen2VL, and Qwen2.5VL exhibit clear tendency to favor "Yes" responses, even for questions where the correct answer is "No" (highlighted in red). However, the VAPR models demonstrate a notable reduction in this bias, with emergent shift towards answering "No" more frequently than "Yes." This behavior is most pronounced in ALaVA-VAPR 13B. We attribute these improvements to enhanced perception and reasoning capabilities, which manifest improved visio-linguistic compositionality and reduced bias towards "Yes" responses.
4.3.4 VaPR-OS: Ablation Study using open-source LLM Editor
We conduct an ablation using Qwen3-32B as the open-source LLM editor evaluate the generalizability of the VaPR pipeline beyond closed models like GPT-4o (see Appendix (Dnddix$D.4) $D.4). The resulting dataset, VAPR-OS, is constructed from the same subset as the 10K version of VAPR. Analyzing the data, we find that VAPR-OS exhibits comparable hard-negative characteristics to VaPR, with a token length gap of (vs. 3 in VaPR) and a Levenshtein distance of 10 (vs. 6). We fine-tuned Qwen2.5VV-InstruuttB3 on VAPR-OS and evaluated their performance against models trained the GPT-4o-based VAPR (10K subset) across benchmarks (SD.4), Results show that models trained on VAPR-OS achieve 99% of the performance of their GPT-4o-based counterparts (VAPR), with both consistently outperforming base instruct models. These findings demonstrate that the VaPR pipeline generalizes well to open-source editors, allowing researchers to apply the VAPR framework to their own SFT datasets without relying closed-source APIs.
1.-A-vil.--nsstuutt-7. Qwen2VI--nsstuut--25 and
Related Work
Large Vision Language Models (LVLMs) LVLMs combine visual inputs from pre-trained vision encoders (Radford et al., 2021; Zhai et al., 2023) with large language models (LLMs) (Team, 2024b; Touvron et al., 2023; Dubey et al., 2024; Cai et al., 2024) via projection modules (Liu et al., 2023b; Ye et al., 2024; Li et al., 2023b; Tong et al., 2024). They typically trained in two stages: pretraining on large-scale image-text data to align modalities and instruction tuning or supervised finetuning vision- languaela datasets for open-ended tasks.

---

Published as a conference paper at COLM 2025
Distrbbttoon of Model Model Predictions win Ground
Yes Correct
Correct
Cornect)
Correct)
Geecdel Gver2 511-67575
0.6
uave 5-78
0.0
LLAVA-PPR-
uave--15- 1-13.
uLavA VAPP- 36
GwenZVi-22
Owen VaPn
(wen)VL
OwenzVL MannVV-VAPP:.
Owen2 98
Qwen2 svt Va-aa
QwenZ.5V VL-7
owenz Varn 78
Figure 5: Performance scaling of VAPR models with 3K, 10K, and 30K samples, shown % improvement over base instruct models. Note: X-axis spacing between 3K, 10K, and 30K not uniform.
Figure 6: Comparison of ("No" responses on Natural- Bench - base vs. VaPR models. Dotted rectangles highlight incorrect "Yes" (red) and "No" (yellow) predictions,, illustrating reduced "Yes" bias after preference finetuning.
Preference Optimization in LVLMs Preference optimization, typically the third training stage for LVLMs, improves modality alignment and reduces hallucinations. It relies on reference datasets sourced via human annotation (Yu al., 2024a; Sun al., 2023), Al annotation (Zhou et al., 2024a; Li et al., 2023c; Zhao et al., 2023), or self-curation (Yu et al., 2024b; Zhou et al., 2024b; Deng et al., 2024; Wang et al., 2024b). While human labels are high-quality but expensive, Al annotations using models like GPT-4V (OpenAT, 2023) are scalable but may introduce stylistic or length inconsistencies and distill hallucinations. Self-curated approaches generate multiple responses, rank them heuristically or with LV LVLMs, and select preference pairs, but often fail to control for stylistic or length biases leading to noise, as both chosen and rejected responses may seem equally plausible (Yan et al., 2024). Our method builds on Direct Preference Optimization (DPO) (Rafailov et al., 2024) and introduces post-SFT data construction pipeline that addresses these challenges. Unlike prior works (Zhou et al., 2024a; Yu et al., 2024b; Zhou et al., 2024b; Deng et al., 2024; Wang scoring, employ LLM-guided response
et al., 2024b) that VLMs for generation editing to inject task-aware semantic errors into rejected responses while explicitly preserving style and length. This reduces the risk of DPO exploiting superficial cues. In contrast to Chen et al. (2025), which uses LLMs to generate hard negatives for multilingual embedding training, our approach targets Post-SFT preference alignment English, where fine-grained semantic and stylistic control is crucial.
Conclusion
In this work, we address challenges in the alignment and reasoning capabilities of LVLMs by introducing VaPR, a hard-negative preference data generation framework. By constructing high-quality preference dataset of 30K samples, we mitigate length and stylistic biases prevalent in existing synthetic datasets. We demonstrate significant improvements across benchmarks, particularly excelling in reasoning tasks and adversarial scenarios. Our analysis highlights the effectiveness of VAPR in improving vision- reducing binary question bias, paving the way for more reliable and generalizable I.I.Ms..
compositionality and
Future work includes extending the framework to larger and more diverse datasets, applying broader reasoning tasks, and refining hard-negative generation to better target nuanced reasoning errors. Another promising direction is combining VAPR with self-preference methods in low-resource settings using confident responses as positives and generating hard negatives via the VAPR pipeline to enable unsupervised preference data creation. We also plan to explore integrating VAPR with online preference optimization methods such as PPO and GRPO, particularly for complex reasoning tasks like math.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
This research is supported in part by the ECOLE program under Cooperative Agreement HR00112390060 with the US Defense Advanced Research Projects Agency (DARPA), UCLA- Amazon Science Hub for Humanity and Artificial Intelligence, and OpenAl Researcher Access Program (Grant No. 0000004868).
References
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arviv preprint a'Xiv:2308.01390 2023.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. InternIm2 technical report, 2024. URL https:: httppp///aaai..or/a////////5 org/abs//4033 17297.
Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. mme5: Improving multimodal multilingual embeddings via high-quality synthetic data. Xiv preprint ar Xrio:2502208666 2025.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegptáv: Improving large multi-modal models with better captions. Xrv preprint ar Xrx::22311.12799 2023.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, al. Are we on the right way for evaluating large vision-language models? arXiv preprint ar arXio:2403.20330 2024a.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far are we gpt-4v? closing the gap commercial multimodal models with open-source suites, 2024b. URL https: //wwxiaaiivv////a////a//aaaaaia org/ abss/224 16821.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. URL https: //arxiv http//awvvv.g/.aa.a////oo org/abs/
Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang. Enhancing large vision language models with self- tolf-training on image comprehension. ar Xiv preprint ar arXiv:2055.197 19716, 2024.
11

---

Published as a conference paper at COLM 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. Xiv preprint ar Xio::4407..1788 2024.
Ling Fu, Zhebin Kuang, Jiajun Song, Mingxin Huang, Biao Yang, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, et al. Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning. arXiv preprint arXiv:2501.0000 003211, 2024.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallu- sionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models, 2024. URL https: /ttps:////xvvvv/// orgg abs//3//./////////// wwwp://..rrrgaa//....e.o 14566 14566.
Jiwoo Hong, Noah Lee, and James Thore. Orpo: Monolithic preference optimization without reference model, 2024. URL httpss///aaxvvvo///aaaa/ 2403.06691.
Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality, 2023. URL https://arxiv.orr http://www..gg////////////// 14610.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. 09685.
URL https://arxiv.org//bbaa
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hanna Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. Advances in neural information processing systems, 37:36602-36633, 2024a.
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentan- gling best practices for learning from preference feedback. arXie preprint xrXiv22406.09299, 2024b.
Philipp Koehn. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 conference on empirical methods in natural language processing, pp. 388-395, 2004.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step- dpo: Step-wise preference optimization for long-chain reasoning of llms. arviv preprint arXiv:2406.18629 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif: Scaling reinforce- ment learning from human feedback with ai feedback. Xiv preprint arXio:2309.00267, 2023a.
Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo. Volcano: mitigating multimodal hallucination through self-feedback guided revision. arv preprint ar arXi::2311.00362, 2023b.
Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. Naturalbench: Evaluating vision-language models on natural adversarial samples. arXiv preprint ar XxXo:24100.14669 2024a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal Ilms with generative comprehension. ar arXiv preprint ar arXit:2307.16125, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. ar arv preprint ar 7Xit:2301.12590 2023b.
12

---

Published as a conference paper at COLM 2025
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint ar arXiv:2512.1066,, 2023c.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023d. URL https://arxiv.cr org/ abs/2305. 10355.
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26763-26773, 2024b.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023c.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oct, and world knowledge, January 2024. URL https://llava-vl. gi thub. wtt.lo///llo/oo 0220-010-222200-0000000000011100 -10---011----aa----------------- llavaanexva...
Junru Lu, Jiazheng Li, Siyu An, Meng Zhao, Yulan He, Di Yin, and Xing Sun. Eliminating biased length reliance of direct preference optimization via down-sampled kl divergence. ar preprint ar Xix::2206.1000 10957, 2024.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arviv preprint ar arxto=3300002255, 2023.
OpenAL. Gpt-4v(ision) system card, 2023b. 2023. 2023.
https://openai..co/rresss
gpt-4v-systemn-cae
OpenAL. Gpt-4o system card. https:
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization, 2024. URL https://arxii. org/abs/2403..99155 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https: pattps://arxiin wttpa/awwww.../////////a/////// 00020.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. URL
18290.
Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ö. Ank,, and Tomas Pfister. Mitigating object hallucination via data augmented contrastive tuning, 2024. URL http://awxxiv.oo/rbbbb///s
18654.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https:://rxxv.oor///aea ot70/ 17777006777
13

---

Published as a conference paper at COLM 2025
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arviv preprint ar Xrii:2209.11425, 2023.
OpenGVLab Team. Internv12: Better than the best best-exppndiigg performance boundaries of open-source multimodal models with the progressive scaling strategy. https://internvl. github. ht/blog//////oo//oo
I2-InneenVL 0000/ 2024a.
The Vicuna Team. Vicuna: An open-source chatbot impressing gpt-4 with 90 https:
2024b.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya lyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. ar preprint ar arXi::2406.1688 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arviv preprint ar Xiv:23307 Xii:23377009288, 2023.
Rohan Wadhawan, Hritik Bansal, Kai-Wei Chang, and Nanyun Peng. Contextual: Evaluating context-sensitive text-rich visual reasoning in large multimodal models. arv preprint arXiv:2401.13311, 2024.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint vrXic:2409.12111, 2024a.
Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, et al. Enhancing visual-language modality alignment in large vision language models via relf-improvement. arXiv preprint :Xir:2405.15973, 2024b.
Siming Yan, Min Bai, Weifeng Chen, Xiong Zhou, Qixing Huang, and Li Erran Li. Vigor: Improving visual grounding of large vision language models with fine-grained reward modeling, 2024. URL
/rg/ 0014422..6166 18.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. ar arXiv preprint (rXiv:2505.09388 2025.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Zhang, and Fei Huang. mplug-owi2, Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13040-13051, 2024.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807-13816, 2024a.
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. ar Xiv preprint ar Xie:2245..172D0 2024b.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint ar Xtt:2308.02290 2023.
Ya-Qi Yu, Minghui Liao, Jiwen Zhang, and Jihao Wu. Texthawk2: A large vision-language model excels bilingual ocr and grounding with 16x fewer tokens. arXiv preprint XrXi::2410.05 05261, 2024c.
14

---

Published as a conference paper at COLM 2025
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-955555 2024.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? arxiv preprint ar xfXiv:2200001936, 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975-11986, 2023.
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing Ivlms through hallucination-aware direct preference optimization. Xiv preprint ar Xf1:23311.16889 2023.
Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning, arv preprint axXt:2402..11411, 2024a. 2024a.
Yiyang Zhou, Zhiyuan Fan, Dongje Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. Xiv preprint 2rXi::2405.1422 2024b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ar Xiv preprint arXix:2304. 10592, 2023.
15

---

Published as a conference paper at COLM 2025
A Appendix Overview
The appendix is organized into the following sections and subsections:
Detailed Methodology of VAPR Data Generation ($B):
Task Categories ($B.1): Definition of the task categories included in the VAPR dataset.
Prompt Design Examples (5B.2): Specific prompts used for data generation for each task category and corresponding qualitative examples illustrating the generated data.
Fine-grained length & Stylistic analysis ($B.3)
Human Study (6B.4): Annotation and evaluation setup.
Experimental Setup (SC):
Training Setup (5C.1): Details of the training setup for our models are provided here.
Evaluation Setup (5C.2): Implementation of the evaluation benchmarks.
Extended Results (SD):
Data Scaling (6D.1): Detailed results for VAPR model with increasing training dataset size.
-Benchmark detailed results ($D.2): Fine-grained evaluation across benchmark categories, for CVBench and MMStar.
-Prfference Dataset Comparison (SD.3): Detailed results for the comparison of base models trained on VAPR and other preference datasets.
16

---

Published as a conference paper at COLM 2025
B Detailed Methodology of VAPR Data Generation
In this section, we provide the breakdown of VaPR dataset into task categories, prompt design examples for each category, setup for human study and linguistic length analysis study.
B.1 Task Categories
An essential component of our data generation pipeline is the categorization of filtered samples (~270K) from the VA-6655 SFT set into distinct task categories. This ensures comprehensive coverage of perception, reasoning, and composite tasks (perception & reasoning) in the final VaPR dataset. The VaPR dataset comprises ten task categories: object (like type, material, action), color, size, background (like weather, time of day, surrounding lighting), counting, spatial reasoning, existence, referential VQA (like color of an object on the left, etc), general reasoning (like abstract & knowledge-basse outlined Table The categorization process is carried out in the following three steps:
and image captioning, as
Task-specific keywords (see Table 3) are applied to the instructions in the filtered SFT set to assign a task category to each sample containing those keywords.
Categorizatioo follows defined order:
Samples are initially tagged into the categories: color, size, counting, spatial reasoning, background, existence, captioning, referential VQA (comparative reasoning) and general reasoning (in no specific order). Specifically for the existence category, we check the first word in the instruction.
-Remaining samples are assigned to the object category. This approach is necessary because object-related tasks, encompassing types and knowledge, require a broad, non-generalizable keyword list.
Resolving Multiple Task Categories for sample: Samples tagged with multiple task categories are categorized as referential VQA (qualitative examples shown in $B.2.8). Thus, referential VQA consists of samples tagged as comparative reasoning and the above samples.
Task
Task Type
Keyword
Color
Perception
color(s)
Size
Perception
size(8)
Background
Perception
environment, time of, day, year, weather, lighting, many, count(s), instance(s), counting
Counting
Reasoning
Spatial Reasoning
Reasoning
where, located, placed, positioned, left, right, in front of, down, above, below
Existence
Perception & Reasoning
are, is, can, do, does, would, will
General Reasoning
Perception & Reasoning
could, would, might, purpose, reason, based, should
Referential VQA
Perception & Reasoning Samples with keywords from more than one task
Comparative Reasoning: comparison, difference, closer, nearer, bigger
Image Captioning
Perception & Reasoning
analyze, describe, write, elaborate, description, snapshot
Table 3: Overview of tasks, task types (perception & reasoning), and task-specific keywords used for VaPR dataset categorization. Keyword list is shortened for clarity. Note untagged samples are grouped together into the object category.
17

---

Published as a conference paper at COLM 2025
B.2 Prompt Design & Examples
In this section, we provide a detailed walkthrough of hard-negative generation for each task category, accompanied by additional examples from each category object (SB.2.1), color ($B.2.2), size ($B.2.3), background ($B.2.4), counting ($B.2.5), spatial reasoning ($B.2.6), existence ($B.2.7), referential VQA ($B.2.8), image captioning (SB.2.9), general reasoning ($B.2.10). In the prompt figures, spans of the chosen responses are highlighted in blue, while the corresponding perturbed spans in the generated hard-negative responses are highlighted in green. Lastly, we also provide the algorithm and additional examples showing generation steps for task categories with penalty list (e.g. color) SB.2.11.
B.2.1 Object
Prompt:
You are ResponseEditorGPT who is given instruction and response generated by a vision-language model. The instruction asks about the objects, or actions. Your task is follows:
1. Modify the "objects" "action" to make the "Original Response" about "objects" or "actions" incorrect.
2. "New Response" must be linguistically very similar to "Original Response" and must be incorrect.
3. You must ensure changes must be realistic given world knowledge.
4. You can minimally change other spans of the sentence to grammatical correctness and fluency.
5. The output format should be "New Response:"
Instruction:
What type of flooring is in the room?
Original Response: The room has hard wood floors.
Your Turn
New Response:
Chosen Response:
The room has hard wood floors.
Hard-negative Rejected Response: The room has carpeted floors.
Figure 7: Example prompt and generated hard-negative for VAPR sample of object task category.
18

---

Published as a conference paper at COLM 2025
Instruction: What type of airplane displayed in the image?
Instruction:
What is the woman doing in the
image?
Chosen Response: The image displays a World War era airplane with propeller hanging in museum.
Chosen Response: The woman is sitting at a desk table, working laptop computer.
Rejected Response: The image displays World War airplane with jet engines hang- ing in museum.
Rejected Response: The woman standing next to desk table, working laptop computer.
(a) Task requiring perception of object's com- ponents
(b) Task requiring perception of an object's action
Instruction:
What type of cake is on the plate?
Instruction: What is the cat doing in the image?
Chosen Response: There slice of chocolate cake on plate.
The cat is sitting on the edge of bathroom sink.
Chosen Response:
Rejected Response: There is slice of straw- berry cake on the plate.
Rejected Response:
The cat is sleeping the edge of a bathroom sink.
(c) Task requiring perception of object's type
(d) Task requiring perception of an object's action
Figure 8: Qualitative examples of the object task category, like object perception, its type, components, and associated actions.
19

---

Published as a conference paper at COLM 2025
B.2.2 Color
EoNnk BUCK
Prompt:
You are a ResponseEditorGPT who is given an instruction and a response generated by a vision-language model. The instruction asks about the colors of objects, environments, or themes. Your task is as follows:
1. Modify the "colors" of objects, environments, or themes in the response to make "Original Response" about "colors" incorrect.
2. You must only change the "colors", so that "New Response" is linguistically very similar to "Original Response" and is incorrect.
3. The "New colors" you use to replace original colors must be unique and not be too descriptive.
4. The "New colors" must be realistically possible, considering the object they describe.
5. You cannot use colors in the penalty list.
6. You can minimally change other spans the sentence to grammatical correctness and fluency.
7. List the "New colors" you replace within the response.
Penalty list: [white, blue]
Instruction: What color is the kitchen counter where the vegetables are placed?
Original Response: The kitchen counter where the vegetables placed is green.
Your Turn
New Response:
New Colors:
Chosen Response: The kitchen counter where the vegetables are placed is green.
Hard-negative Rejected Response: The kitchen counter where the vegetables are placed yellow.
New Colors: [yellow]
Example prompt and generated hard-negative for VAPR sample of color task category. Here, the penalty list is simulated for illustration. New colors represent additional information generated alongside the hard-negative rejected response to maintain the penalty list. We maintain list of most perturbed color values (size 2), updated after every 10 hard-negative rejected response generations. Note, if more than one color is perturbed, "New Colors" will have more elements the list.
Figure 9:
20

---

Published as a conference paper at COLM 2025
What are the colors of the flowers in the vase?
Instruction:
Instruction: What color are the couches in the living room?
Chosen Response: The colors of the flowers in the vase are red, green, and purple.
Chosen Response: The couches in the living room are black.
Rejected Response: The colors of the flowers in the vase are yellow, blue, and orange.
Rejected Response: The couches in the living
room orange.
(b) Task requiring color perception of mulitple objects.
(a) Task requiring color perception of an object.
Instruction:
What is the overall color theme of the living room?
The overall color theme the living room is predominantly white, with white furniture and white elements on the brick wall.
Chosen Response:
Instruction:
What is the color of the surfboard
the dog sitting on?
The color of the surfboard the dog sitting is blue.
Chosen Response:
Rejected Response: The overall color theme of the living room is predominantly turquoise with turquoise furniture and turquoise ele- ments on the brick wall.
Rejected Response:
The color of the surfboard
the dog sitting is orange.
(d) Task requiring color perception of the theme of background setting.
(c) Task requiring color perception of object.
Figure 10: Qualitative examples of the color task category, like perception of colors of single object, multiple objects and background theme.
21

---

Published as a conference paper at COLM 2025
B.2.3
Size
TEXA Y
GETCH
ORIGIN
Prompt:
You ResponseEditorGPT who is given instruction and response generated by vision-language model. The instruction asks about the "size" of objects or themes. Your task as follows:
ablunnse""bbutttii 1. Modify the "iiz"s of objects themes to the "Original Response" about the "size" incorrect.
2. "New Response" must be linguistically very similar to "Original Response" and must be incorrect.
3. You must ensure changes must be realistic given world knowledge.
4. You can minimally change other spans of the sentence to grammatical correctness and fluency,
5. The output format should "New Response:"
Instruction: How is the cheeseburger described in terms of size and ingredients?
Original Response: The cheeseburger is described as massive and containing double cheese layers.
Your Turn
New Response:
Chosen Response: The cheeseburger is described as massive and containing double cheese layers.
Hard-negative Rejected Response: The cheeseburger is described small and containing single cheese layer.
Figure 11: Example prompt and generated hard-negative for VaPR sample of size task category.
22

---

Published as a conference paper at COLM 2025
How would you describe the size of the group of people flying kites?
Instruction:
Instruction: room?
What is the general size of the
The group people flying kites is large, indicating that it a popular and well-attended event or gathering.
Chosen Response:
The room is described as small living area, which implies that is not very large or spacious.
Chosen Response:
Rejected Response: The group people fly- ing kites small, indicating that it is more intimate and possibly less publicized event or gathering.
The room is described as expansive living area, which implies that is quite large and spacious.
Rejected Response:
(a) Task requiring thematic size perception, here room.
(b) Task requiring thematic size perception, here gathering of people.
Instruction: What's the size of the refrigerator in the image?
Instruction:
What is the size of the cake in the
image?
Chosen Response: The refrigerator in the im- age is small or mini-sized refrigerator.
Chosen Response: The cake in the image is quite large and tall, comprising multiple layers.
Rejected Response: The refrigerator in the image is a large, full-sized refrigerator.
Rejected Response: The cake in the image is small and flat, consisting of a single layer.
(c) Task requiring size perception of an object.
(d) Task requiring size perception of an object.
Figure 12: Qualitative examples of the size task category, like size perception of an object or broadly a theme (eg. room, gathering of people)
23

---

Published as a conference paper at COLM 2025
B.2.4 Background
Prompt:
You ResponseEditorGPT who is given instruction and response generated by vision-language model. The instruction asks about the "time", "weather", or "environment" of events, surroundings, or themes. Your task as follows:
1. Modify the "time", "weather", or "environment" of events, surroundings, or themes to make the "Original Response" about the "time", "weather", or "environment" incorrect.
2. "New Response" must be linguistically very similar to "Original Response" and must be incorrect.
3. You must ensure changes must be realistic given world knowledge.
4. You can minimally change other spans of the sentence to grammatical correctness and fluency,
5. The output format should "New Response:"
Instruction:
What time of day is it in the image?
Original Response: is nighttime in the image, evidenced by the dark sky background.
Your Turn
New Response:
Chosen Response: is nighttime in the image, as evidenced by the dark sky background.
Hard-negative Rejected Response: is twilight in the image, evidenced by the darkening sky background.
Figure 13: Example prompt and generated hard-negative for VaPR sample of background task category. Notably, the slight linguistic change from "dark" to "darkening" has significant change in semantics.
24

---

Published as a conference paper at COLM 2025
Instruction: What kind of environment is the giraffe in?
The giraffe is in dry, arid environment with brown vegetation, scrub bushes, and sand in the field. It appears to be somewhat harsh, wild habitat that the giraffe navigating.
Chosen Response:
Instruction:
What's the weather like the image?
Chosen Response:
The weather in the image is creating a wet atmosphere on the street.
The giraffe is in lush, green environment with dense vegetation, leafy bushes, and grassy field. appears to a somewhat fertile, wild habitat that the giraffe navigating.
Rejected Response:
Rejected Response: The weather in the image is snowy, creating cold atmosphere on the street.
(a) Task requiring perception of weather.
(b) Task requiring perception of environment.
Instruction: What type of lighting is featured in the living room?
Instruction: What are weather conditions in the image?
The living room features spot lighting, which provides focused illumi- nation on specific areas or objects within the room.
Chosen Response:
The weather conditions in the image are rainy, evidenced by the group people holding umbrellas to protect themselves from the pouring rain.
Chosen Response:
Rejected Response: The living room features ambient lighting, which provides general, diffused illumination throughout the room.
The weather conditions in the image are rainy, as evidenced by the group of people holding unbrellas to protect themselves from the light drizzle.
Rejected Response:
(c) Task requiring perception of background light- ing.
(d) Task requiring perception of weather.
Figure 14: Qualitative examples of the background task category, like perception of weather, environment, lighting, and time of the day.
25

---

Published as a conference paper at COLM 2025
B.2.5 Counting
Prompt:
You ResponseEditorGPT who is given instruction and response generated by vision-language model. The instruction asks about the counts of objects. Your task is follows:
1. Modify the "counts" all objects in the response to make "Original Response" about "counts" incorrect.
2. You must only change the "counts", that "New Response" is linguistically very similar "Original Response" and is incorrect.
3. The "New counts" you use to replace original colors must be unique and not be too descriptive.
4. The "New counts" must be realistically possible, considering the object they describe.
5. You cannot use counts the penalty list, neither the word form in the penalty nor its numerical
form.
6. You can minimally change other spans of the sentence to grammatical correctness and fluency.
7. List the "New counts" you replace within the response.
Penalty list: [two, seven]
Instruction: How many people are visible near the truck in the image?
Original Response: There are four people visible near the truck in the image. Three people are standing in front of the truck, while another person is in the background.
Your Turn
New Response:
New Colors:
Chosen Response: There are four people visible near truck the image. Three people are standing in front the truck, while another person is in the background.
Hard-negative Rejected Response: There are people visible near the truck in the image. Five people are standing in front the truck, while another person background.
New Counts: [six, five]
Figure 15: Example prompt and generated hard-negative for VaPR sample of counting task category. Here, the penalty list is simulated for illustration. New counts represent additional information generated alongside the hard-negative rejected response to maintain the penalty list. Similar to color task category, We maintain list of most perturbed count values (size 2), updated after every 10 hard- nagd-negative rejected response generations. Note, if more than one count perturbed, "New Counts" will have more elements in the list.
26

---

Published as a conference paper at COLM 2025
Instruction:
How many bears are present in the image?
Instruction: How many motorcycles are in the image?
Chosen Response: There are three bears present in the image adult bear and two bear cubs.
Chosen Response: There are three motorcycles in the image.
Rejected Response: There are seven bears present the image- an adult bear and six bear cubs.
Rejected Response: There are six motorcycles in the image.
(a) Task requiring counting capability.
(b) Task requiring counting capability.
ONE WAY
JHH OND
ercaar Visa
Instruction: How many road signs there the image?
Instruction: How many women are in the image holding teddy bears?
There are several road signs in the image, including two One Way signs, one of which upside down.
Chosen Response:
Chosen Response: There are three women in the image holding teddy bears.
Rejected Response: There are road signs in the image, and the streets empty of any directives.
Rejected Response: There are two women in the image holding teddy bears.
(c) Task requiring counting ability.
(d) Task requiring counting capability.
Figure 16: Qualitative examples of the counting task category.
27

---

Published as a conference paper at COLM 2025
B.2.6 Spatial Reasoning
Prompt:
You are a ResponseEditorGPT who is given an instruction and a response generated by a vision-language model. The instruction asks about the "spatial relation", of objects. Your task is as follows:
1. Modify the "spatial relation" of objects to make the "Original Response" about the "spatial relation" incorrect.
2. "New Response" must be linguistically very similar to "Original Response" and must be incorrect.
3. You must changes must be realistic given world knowledge.
4. You can minimally change other spans the sentence to grammatical correctness and fluency.
5. The output format should be "New Response:"
Instruction: Where the toilet located in the image?
Original Response: The toilet is located outdoors, surrounded by field of grass and trees, in the middle of the woods.
Your Turn
New Response:
Chosen Response: The toilet located outdoors, surrounded by field of grass and trees, in the middle of the woods.
Hard-negative Rejected Response: The toilet is located indoors, surrounded by white-tiled walls and sink, the corner of bathroom.
Example prompt and generated hard-negative for VaPR sample of spatial reasoning task category.
Figure 17:
28

---

Published as a conference paper at COLM 2025
Where the person riding the bicycle in the image?
Instruction:
What position is the cat in while laying the rug?
Instruction:
Chosen Response: The person riding the bicycle on city street, specifically in bicycle lane near many street signs.
Chosen Response: The cat is laying on its back the rug.
Rejected Response: The person is riding the bicycle on a dirt path a park, far away from any street signs.
Rejected Response: The cat is laying on its side on the rug.
(a) Task requiring instance location.
(b) Task requiring instance location.
Instruction: How are the motorcycles arranged in the image?
Instruction: What is the position of the skier's 's ski poles?
Chosen Response: The motorcycles are ar- ranged in rows or parked together in a row, which creates an organized and neat appear- ance.
The skier has tucked his ski poles under his arms while racing through the snow.
Chosen Response:
Rejected Response: The motorcycles are ar- ranged in circle, with each facing outwards, which creates symmetrical and organized appearance.
Rejected Response: The skier is holding his ski poles parallel on either side, with each pole pointing outward from his body as he navigates through the snow.
(c) Task requiring relative spatial reasoning.
(d) Task requiring relative spatial reasoning.
Figure 18: Qualitative examples of the spatial reasoning task category, involve instance location or relative spatial reasoning.
29

---

Published as a conference paper at COLM 2025
B.2.7 Existence
Prompt:
You are ResponseEditorGPT who given an instruction a response generated by a vision-language model. The instruction asks about an "existence" of an object, object attribute, object count, object spatial relation, object comparison, background or theme. Your task is as follows:
1. Modify the original response to change the polarity of the response, that is, make "Yes" "No" and "No" "Yes".
2. Paraphrase both the "Original Response" and the "New Response", such that it says, "Yes" or "No" followed by the ask in the question.
3. "New Response" must be linguistically very similar to "Original Response" and must be incorrect.
4. You must ensure changes must be realistic given world knowledge.
5. You can minimally change other spans of the sentence to grammatical correctness and fluency.
6. The output format should "Original Response: " "New Response
Instruction:
Would this man score touchdown?
Original Response: No, the would not touchdown.
Your Turn
New Response:
Chosen Response: No, the man would not score touchdown..
Hard-negative Rejected Response: Yes, the man would score touchdown.
Figure 19: Example prompt and generated hard-negative for VAPR sample of object task category.
30

---

Published as a conference paper at COLM 2025
Instruction:
Are there any people in the pic-
Instruction: Are the batters ankles twisted?
ture?
Chosen Response: Yes, the batter's ankles are twisted.
Chosen Response: No, there are no people shown in the picture.
Rejected Response: No, the batter's ankles are not twisted.
Rejected Response: Yes, there are people visi- ble in the image.
(b) Task requiring perception of an object's state and action
(a) Task requiring object perception
Instruction: Does the elbow pad to the right of the other elbow pad have black color?
Instruction: Does the light say is ok to walk?
Chosen Response: No, the elbow pad to the right the other elbow pad does not have black color.
Chosen Response: safe walk.
Yes, the light indicates it's
Rejected Response: Yes, the elbow pad to the right of the other elbow pad has black color.
Rejected Response: No, the light says it's not safe walk.
(c) Task requiring color perception spatial reason- ing
(d) Task requiring object perception knowl- edge
Figure 20: Qualitative examples of the existence task category.
31

---

Published as a conference paper at COLM 2025
B.2.8 Referential VQA
Prompt:
You are ResponseFditorGPT who is given an instruction a response generated by a vision-language model. The instruction asks about counts, color, spatial location, comparison or existence of objects. More than one of the tasks can be asked in an instruction. Your task is as follows:
1. Identify the different tasks asked in the question. You do have to output this, only understand the intent.
2. Modify the spans in the response which answer the different tasks in the instruction to make "Original Response" incorrect.
3. You must only change the "spans", so that "New Response" linguisticallyy very similar "Original Response" and is incorrect, while maintaining rest the response.
4. You can minimally change other spans of the sentence to semantic correctness, grammatical correctness and fluency.
5. the task is about colors or counts, ensure that you change the span with wide range of colors and counts respectively.
6. The "New colors" "New Counts" must be realistically possible, considering the object they describe.
7. If the response is one word or small phrase, paraphrase both the "Original Response" and the "New Response", such that it says "New Response" is incorrect with respect to the "Original Response" while being semantically sensible. Both "Original Response" and "New Response" must now be full sentences.
Instruction: What is the size difference between these two boats?
Original Response:
There noticeable size difference between the two boats, with one being considerably larger than the other smaller boat.
Your Turn
New Response:
Chosen Response: There noticeable size difference between two boats, with one being considerably larger than the other smaller boat.
Hard-negative Rejected Response: There no noticeable size difference between the two boats, they appear be identical in size.
Example prompt and generated hard-negative for VAPR sample of referential VQA task category.
Figure 21:
32

---

Published as a conference paper at COLM 2025
What is the size of the dog com- pared to the child?
Instruction:
Instruction:
How many people are in the im- age and where are they located?
The dog is described as large in comparison to the child. This implies that the dog may be of a bigger breed or per- haps fully-grown adult dog.
Chosen Response:
Chosen Response: There are two people, a man and woman, in the image, and they are located in kitchen.
The dog is described as small in comparison to the child. This implies that the dog may be smaller breed or perhaps
Rejected Response:
Rejected Response: There are five people, consisting of three men and two women, in the image, and they located in living room.
still a puppy.
(b) Task requiring perception of an object's state and action
(a) Task requiring object perception
Instruction:
How many giraffes visible in the image, and what are their relative sizes?
Instruction: How many colors are the cows in the image?
Chosen Response: There are two giraffes vis- ible in the image: large adult giraffe and smaller kid giraffe, likely its offspring.
There are three main colors of cows the image: black, brown, and white.
Chosen Response:
Rejected Response: There are three giraffes visible in the image: medium-size adult giraffe along with two smaller giraffes, possibly its offspring.
Rejected Response: There four main colors of cows in the image: black, gray, tan, and white.
(c) Task requiring color perception spatial reason- ing
(d) Task requiring object perception and knowl- edge
Figure 22: Qualitative examples of the existence task category.
33

---

Published as a conference paper at COLM 2025
B.2.9 Image Captioning
Prompt:
You are a ResponseAnalyzerGG who is given an instruction and a response generated by a vision-language model. The response will consist of one or more visual elements objects, object relationships, object attributes, environment information actions. Each visual element modified by one
dimension, where dimension belongs to the set ("Color", "Number", "Size", "Shape", "Other object physical attribute", "Weather Time", "Background" "Spatial relationship" "Comparative relationship", "Other object relationship")
Your task is to list (visual element, dimension, phrase) triplet, where visual element is an element in the response, dimension modifies the visual element and phrase is span from the response that shows how dimension modified the visual element.
You must follow the guidelines given below:
1. Do not repeat the same triplet.
2. The dimension must always belong to the set ("Color", "Number", "Size", "Shape", "Other object physical attribute", "Weather Time", "Background", "Spatial relationship", "Comparative relationship", "Other object relationship")
3. Output format Must be "Triplet List: []" where "[]" is a list of triplets
Instruction: Write a detailed description of the given image.
Original Response:
The image features a unique scene of a green jetfighter airplane on display in open area of the city. The airplane has white and pink accents painted on its design, making it visually striking. It is situated in the middle of the road, with tall buildings surrounding the area.
There are several people admiring and standing next to the green plane. Some of them can be found on the right side of the image, while another person is seen closer to the plane on the left side. In addition to the airplane, there is truck parked nearby on the left side of the road.
Your Turn Triplet List:
Triplet List:
[("jeffighter airplane", "color", "green jetfighter airplane"), ("airplane", "color", "white and pink accent"), ('it", "spatial relationship,, , "in the middle of the road"), ('buildings", "size", "tall buildings") ),( people" "counting", "several people"), (""le")))"somee "spatial relationship", right side of the image"), ( ("people", "spatial relationship" "standing next to the green plan"), ('person", "spatial relationship", closer to the plane on the left side"), ("truck", "spatial relationship', truck parked nearby on the left side of the road")]
Figure 23: Example prompt for extracting the different possible dimensions of perturbing an image caption (perception & reasoning). Note, we simulate the dimension extraction in this example.
34

---

Published as a conference paper at COLM 2025
PrompT:
You are a ResponseEditorGPT who is given an instruction and a response generated by a vision-language model. The response will consist of one or more visual elements - objects, object relationships, object attributes, environment information or actions. Each visual element is modified by one more dimension, where dimension must belong the set ("Color", "Number', "Size", "Shape", "Other object physical attribute", "Weather Time", "Background", "Spatial relationship" "Comparative relationship", , "Other object relationship"). You will also be given a list of (visual element, dimension, phrase) triplets, where visual element is an element in the response, dimension modifies the visual element and phrase is span from the response that shows how dimension modified the visual element.
Your task is as follows:
1. For each triplet, modify the phrase in the "Original response" corresponding to each triplet along the dimension mentioned in the triplet to make the "Original response" incorrect.
2. "New Response" must be linguistically very similar "Original response" and must be incorrect.
3. You must ensure changes must be realistic given world knowledge.
4. You can minimally change other spans of the sentence to grammatical correctness and fluency.
5. The output format should be "Original Response: " "New Response
Instruction: Write a detailed description of the given image.
The image features unique scene of green jetfighter airplane on display an open area the city. The airplane has white and pink accents painted on its design, making it visually striking. It is situated in the middle of the road, with tall buildings surrounding the area.
Original Response:
There are several people admiring and standing next to the green plane. Some of them can be found on the right side of the image, while another person seen closer to the plane the left side. In addition the airplane, there a truck parked nearby on the left side of the road.
Triplet List: [('jttfihterr airplane", "color", "green jetfighter airplane"), " ("airplane", "color", white and pink accent"), ("it", "spatial relationship", "in the middle of the road"), ('buildings", "size", "tall buildings") "iaass""""""""""" ("people", "spatial relationship" "standing next to the green plan"), ("person", "spatial relationship", closer the plane on the left side"), ("truck", "spatial relationship", truck parked nearby on the left side the road")] Your Turn
New Response:
The image features a unique scene of green jetfighter airplane on display in an open area of the city. The airplane has white and pink accents painted on its design, making it visually striking. It is situated in the middle of the road, with tall buildings surrounding the area.
Chosen Response:
There are several people admiring and standing next to the green plane. Some of them can be found on the right side of the image, while another person is seen closer the plane on the left side. addition to the airplane, there is truck parked nearby on the left side the road.
Hard-negative Rejected Response: The image features a unique scene of yellow jetfighter airplane on display an open area the city. The airplane has purple and orange accents painted on its design, making it visually striking. It is situated on the side of the road, with short buildings surrounding the area.
There are several people admiring and standing in the distance from the yellow plane. Some of them can be found on the right side of the image, while another person seen farther from the plane on the left side. In addition to the airplane, there is a truck parked far from the right side the road.
Example prompt for generating hard-negative rejected response for image captioning category. We randomly select 50% to 75% triplets and feed them to the prompt as shown above. Similar to colors and counting task categories, maintain a list of most perturbed dimensions (size 2), updated after every 10 hard-negative rejected response generations. If a dimension is used more than once, its frequency is proportionately increased. Note image is the same as the previous prompt (see Fig. 23).
Figure 24:
35

---

Published as a conference paper at COLM 2025
Instruction: Analyze the image in a comprehensive and detailed manner.
The image presents orange and white motor scooter with two passengers riding down the middle of wet street. The person in the front driving the scooter, while the person sitting behind them is holding a purple umbrella over both of them, providing protection from rain.
Chosen Response:
There potted plants the sidewalk, adding some greenery to the scene. Cars be parked or driving in the background, sharing the road with the scooter. scene depicts typical rainy day on an urban street with vehicles and pedestrians going about their daily routine.
Rejected Response:
The image presents a black and white motor scooter with two passengers riding down the middle of dry street. The person in the front is driving the scooter, while the person sitting behind them is holding red umbrella over both of them, providing protection from the sun.
There are potted plants on the sidewalk, adding some greenery to the scene. Cars can seen parked or driving in the background, sharing the road with the scooter. The scene depicts typical sunny day suburban street with vehicles and pedestrians going about their daily routine.
(a) Task requiring color, object, and background perception.
Figure 25: Qualitative example of the image captioning task category.
36

---

Published as a conference paper at COLM 2025
B.2.10 General Reasoning
Prompt:
You are ResponseFditorGPP who is given an instruction a response generated by a vision-language model. The instruction asks about the "reasoning" of objects, events, environments, or themes. Your task is as follows:
1. Make the reasoning in the original response incorrect.
2. You can modify the objects, their attributes, related objected, or action and make the "original response" about "reasoning" is incorrect.
3. "New Response" must be linguistically very similar "Original response" and must be incorrect.
4. You must ensure changes must be realistic given world knowledge.
5. You can minimally change other spans of the sentence to grammatical correctness and fluency.
6. The output format should be "Original Response: " "New Response
Instruction: What might be a possible reason for the man taking a picture the dirt cake?
Original Response: A possible reason for the
taking picture of the dirt cake could be that the cake is unique and creative design, which features construction scene on top. He might want to capture the design and decoration before is served or share the photo with others to showcase the artistic and aesthetic aspects of the cake. Additionally, the image could be used as memory of a special occasion or event for which the cake has been prepared.
Your Turn
New Response:
Chosen Response: A possible reason for the man taking picture of the dirt cake could be that the cake unique and creative design, which features construction scene on top. He might want capture the design and decoration before it served or share the photo with others showcase the artistic and aesthetic aspects of the cake. Additionally, the image could used as a memory of a special occasion or event for which the cake has been prepared.
Hard-negative Rejected Response: A possible reason for the man taking picture of the dirt cake could be that the cake is a rare and delicate design featuring an underwater scene on top. He might want to capture the design and the aquatic elements before it is served or share the photo with others to highlight the intricate and sea-themed aspects of the cake. Additionally, the image could be used as memory of beach-themed event occasion for which the cake has been prepared.
Figure 26: Example prompt and generated hard-negative for VAPR sample of general reasoning task category.
37

---

Published as a conference paper at COLM 2025
B.2.11 Examples Showing Generation Steps for Task Categories with Penalty List
In this section, we use the color task as an example to illustrate generation steps involving the penalty list (counting task will have a similar way of prompting). We use penalty list of size K=10, updated every 10 samples, with one retry generation attempt possible. Note, until we have atleast K (here =10) distinct colors, the penalty list can be < K.
Since generation is stochastic, the exact penalty list used for a given sample may vary. Therefore, we simulate several cases:
Stochastic Generation:
Case-1: Empty list When the penalty list is empty-
Case-2 When the penalty list is non-empty, with sub-cases:
Case-2a: First Attempt The generation succeeds on the first attempt without conflicting with the penalty list.
Case-2b: Second Attempt The generation initially conflicts but succeeds on retry attempt.
Case-2c: Rerun Script If both attempts fail due to conflicts, the sample is put back into the pool. This situation happens neglibible number of times and the solution is to re-run the script. When re-running, it typically encounters a new penalty list because:
-
* Some samples may have already been processed and help seed the penalty list.
The script randomizes sample order, ensuring that failed samples do not repeatedly see the same penalty list.
For the purpose of illustration, assume no generation failures by the LLM and K=10, when penalty list is not empty. The /rejected response (new response) is actually sample from the dataset, and hence kept as the correct response across scenarios. Now, we provide two examples showcasing different cases:
38

---

Published as a conference paper at COLM 2025
SRT
1126
PhotomagBen
Prompt:
You Restonse who is given instruction and response generated by vision-language model. The instruction asks about the colors objects, environments, or themes. Your task is as follows:
1. Modify the "colors" of all objects, environments, themes in the response to make "Original Response" about "colors" incorrect.
2. You must only change the "colors", so that "New Response" linguistically very similar "Original Response" and is incorrect.
3. The "New colors" you use to replace original colors must be unique and not be too descriptive.
4. The "New colors" must be realistically possible, considering the object they describe.
5. You cannot use colors in the penalty list.
6. You can minimally change other spans of the sentence grammatical correctness and fluency.
7. List the "New colors" you replace within the response.
Penalty list:
Instruction: What colors are present on the subway train the image?
Original Response: The subway train in the image is orange, blue, and silver.
Your Turn
New Response:
New Colors:
39

---

Published as a conference paper at COLM 2025
Original Response: The subway train in the image is orange, blue, and silver.
Case-1: Empty list
Penalty List:
New Response: The subway train in the image is pink, turquoise, and white. / New Colors: l'pink', 'turquoise', 'white']
Case-2a: First Attempt
Penalty List: l'yellow', 'black,, 'beige', 'teal', 'green', 'burgundy', 'sepia', 'lavender', 'purple', 'orange')
New Response: The subway train in the image is pink, turquoise, and white. /
New Colors: l'pink', 'turquoise', 'white"]
New Response: The subway train in the image is pink, black, white. x New Colors: l'pink', 'black, 'white'] x
Reason: Color black conflicts with penalty list, retry.
Case-2b: Second Attempt
Penalty List: ['yellow', 'black, 'beige', 'teal', 'green', 'burgundy', 'sepia', 'lavender', 'purple', 'orange']
New Response: The subway train in the image is pink, turquoise, and white. /
New Colors: l'pink', 'turquoise', 'white']
New Response:
The subway train in the image is yellow, green, white. X New Colors: l'yellow', 'green', 'white'] x
Reason: Colors yellow green conflict with penalty list, put sample back un-annotated pool and rerun the script with un-annotated samples.
Case-2c: Rerun-script (penalty list updates
'olee''green'' Penalty List: ['yellow', bback',, 'beige', 'blue', 'ggeee'n, 'red', 'silver', 'lavender', 'purple', 'orange']
New Response: The subway train the image is pink, turquoise, and white.
New Colors: l'pink', 'turquoise', 'white']
New Response: The subway train the image red, turquoise, white. New Colors: l'red', 'turquoise', 'white'] X
Reason: Color red conflicts with penalty list, go to Case-2b.
40

---

Published as a conference paper at COLM 2025
COLLAGE
Prompt:
You ResponseEditorGPT who is given instruction and response generated by vision-language model. The instruction asks about the colors objects, environments, or themes. Your task is as follows:
1. Modify the "colors" of all objects, environments, or themes in the response to make "Original Response" about "colors" incorrect.
2. must only change the "colors", so that "New Response" linguistically very similar "Original Response" and is incorrect.
3. The "New colors" you use to replace original colors must be unique and not be too descriptive.
4. The "New colors" must be realistically possible, considering the object they describe.
5. You cannot use colors in the penalty list.
6. You can minimally change other spans of the sentence to grammatical correctness and fluency.
7. List the "New colors" you replace within the response.
Penalty list:
Instruction:
What color the bags packages containing the office supplies?
Original Response: The office supplies are contained small pink packages or a pink envelope.
Your Turn
New Response:
New Colors:
41

---

Published as a conference paper at COLM 2025
Original Response: The office supplies contained in small pink packages or pink envelope.
Case-1: Empty list
Penalty List:
New Response: The office supplies are contained in small teal packages teal envelope. / New Colors: ['teal'] V
Case-2a: First Attempt
Penalty List: l'purple', 'yellow', 'white', 'orange', 'green', 'brown', 'silver', 'red', 'pink', 'maroon']
New Response: The office supplies are contained in small teal packages or a teal envelope. / New Colors: ['teal'] /
New Response: The office supplies are contained small white packages or white envelope. New Colors: ['white'] x
Reason: Color white conflicts with penalty list, retry.
Case-2b: Second Attempt
Penalty List: l'purple', 'yellow', 'white', 'orange', 'green', 'brown', 'silver', 'red', 'pink', 'maroon']
New Response: The office supplies are contained in small teal packages or a teal envelope. / New Colors: ['teal'] /
New Response: The office suppiie contained in small orange packages transparent
transparenteevellpee. envelope.
New Colors: l'orange', 'transparent'] X
Reason: Color orange conflicts with penalty list, even when transparent is reasonable and has no conflict. Put sample back to un-annotated pool and rerun the script with un-annotated samples.
Case-2c: Rerun-script (penalty list updates
Penalty List: ['yellow', 'goold, 'blue', 'black, 'brown', 'silver', 'red', 'pink', 'maroon']
New Response: The office supplies are contained in small teal packages or a teal envelope. / New Colors: ['teal'] /
New Response: The office supplies are contained small silver packages or silver envelope. New Colors: ['silver']
Reason: Color silver conflicts with penalty list, to Case-2b.
42

---

Published as a conference paper at COLM 2025
Table 4: Comparison the VAPR Preference Dataset with Related Works: We compare VAPR to HA-DPO (Zhao et 2023), (Zhouu 2024a), RLAIF-V al, 2024b), and CSR (Zhou al., 2024b) based on stylistic and length analysis, using linguistic similarity via word-level Levenshtein distance (LD) and token-level sequence length differences, respectively. We report the percentage of samples where chosen responses are longer (chosen > rejected) or shorter (rejected >> chosen) for each overall, short, and long response category, with corresponding token differences following the same color notation. A lower LD stands for higher stylistic similarity, and lower token-level sequence length differences stand for higher length similarity, Note CSR only had long responses.
Ours
HA-DPO POVID RLAIF-V
CSR
Overall Samples (%)
(21, 79)
(41, 59)
(24,
(45, 55)
(38, 62)
Linguistic Similarity
6
49
30
62
97
Avg. Token Length Difference
3
18
16
15
27
Token Length Difference
(10, 1)
(15, 20)
(27, 13)
(17, 14)
(23, 29)
Short Response Samples (%)
(19, 81)
(49, 51)
(10, 90)
(43, 57)
Linguistic Similarity
3
24
16
17
- Avg. Token Length Difference
4
16
10
10
Token Length Difference
(7, 1)
(14, 18)
(11, 10)
(14, 7)
Long Response Samples (%)
(33, 67)
(35, 65)
(46, 54)
(45, 55)
(38, 62)
- -Linguistic Similarity
19
65
53
76
97
Avg. Token Length Difference
8
20
27
18
27
-Toeen Length Difference
(17, 4)
(19,21)
(32,23)
(19, 17)
(23,29)
B.3 Fine-grained length & Stylistic analysis
We analyze stylistic and length similarity between chosen and rejected responses, both overall and split by response length (see Table 4). We define responses with <=100 tokens as short and >100 tokens as long. Notably, even for long responses where other methods exhibit higher dissimilarity and greater length differences VaPR maintains higher linguistic similarity and lower token-level differences through its targeted response editing approach. This highlights the hard hard-negative nature of VAPR rejections and the nuanced distinctions its models are trained to optimize for.
43

---

Published as a conference paper at COLM 2025
B.4 Human Study
To evaluate the dataset, we conducted a human annotation study with three annotators per sample, including the authors of this paper. The setup and process detailed as follows:
Sample Selection:
A total of 500 samples were randomly stratified across ten task categories.
To prevent bias from task repetition, consecutive samples were ensured to come from different task categories.
Annotation Procedure:
The study spanned two days, with 250 annotations collected per day.
Each day consisted of four hours of annotation, divided into one-hour sessions, with 30-minute break after each session mitigate annotator fatigue.
Annotators were presented with the following elements for each sample:
* An image.
* An instruction.
chosen response.
A rejected response.
The task required a binary annotation determine whether the rejected response qualified as a hard-negatice was similar to the chosen response.
Annotation Criteria:
Annotators were instructed to evaluate the content of the responses exclusively. Factors such as response length or linguistic similarity were explicitly excluded, as these were analyzed in prior studies (see $3.4) to avoid the confounding factors of human subjectivity in length and linguistic preference.
Quality Assurance:
Results indicated that 97% of the samples aligned with the hard-negative response criteria.
Inter-annotator agreement (IAA), calculated using Fleiss' kappa, was 86%, signifying high level of consistency among annotators.
These findings demonstrate that the dataset achieves high quality and reliability, despite being synthetically generated.
44

---

Published as a conference paper at COLM 2025
C Experimental Setup
C.1 Training Setup
This section outlines the training setup and hyperparameters, summarized in Table Consistent hyperparameters were applied across all VAPR models, including LLaVA-VAPR (7B and 13B), Qwen2VL-VAPR (2B and 7B) and Qwen2.5VL-VAPR (3B and 7B). A shallow hyperparameter search was conducted for the learning rate, with 1e-6 yielding optimal results; le-5 caused model forgetting, while 1e-7 was insufficient for effective learning. An effective batch size of 32 was selected for training efficiency. The models were trained for epuchs using two A100 GPUs.
Hyperparameter
Value
Learning rate
le-6
Learning rate Scheduler
Cosine
Warmup Ratio
0.03 (IILVA) & 0.1 (Qwen2VI. 2.5VT.)
Batch size
32
Lora
128
Lora alpha
256
DPO Loss
sigmoid
DPO
0.1
Max Sequence Length
2048
Table Overview of training hyperparameters
C.2 Evaluation Benchmarks & Setup
C.2.1 Benchmarks
Open-ended & Descriptive Benchmarks This includes LLaVA W (LLaVA-in-the- wild) (Liu et al., 2023c), ConTextual (ConT) (Wadhawan et al., 2024), & MM-VET (MMV) (Yu et al., 2023). LLaVA W assesses open-world visual reasoning and description. ConTextual tests joint reasoning over embedded text and visual elements across diverse text-rich image scenarios. MM-VET evaluates how well LVLMs perform on tasks integrating core-VL capabilities like OCR, spatial reasoning, and math. We report the GPT-4 scores obtained using the respective I.L--as--jjudge prompts of LLaVAW ConTextual, and MM-VET.
Vision-Centric Benchmarks This includes SEED' (SEED Bench image split) (Li al., 2023a), CV (CV Bench) (Tong et al., 2024) and MMStar (MMS) (Chen et al., 2024a). SEED & MMS comprehensively evaluate perception and reasoning. CV assesses counting, spatial reasoning, and comparative reasoning (depth & distance). We report the overall accuracy for each benchmark.
Academic & Math Reasoning: This category includes Math (MV) (Lu et al., 2023), testmini subset, which evaluates mathematical reasoning across diverse problem types, and MMMU (Yue et al., 2024), which tests college-level academic reasoning across domains such as physical sciences, social sciences, finance, etc. For MMMU and Math Vista we report the overall accuracy.
Hallucination & Adversarial This category includes POPE (Li et al., 2023d) and Nat- uralBench (Li et al., 2024a). POPE assesses object hallucination by testing scenarios of object presence and absence. NaturalBench measures visio-linguistic compositionality and vision-blind behavior (bias to provide identical answers regardless of the image) of LVLMs by pairing two questions with two similar images yielding different answers. For POPE, we report the overall Fl score across three categories. For NaturalBench, we provide overall accuracy, per-image accuracy across two questions, per-question accuracy across paired images, and group accuracy, reflecting the model's ability to answer all four image-question combinations correctly.
45

---

Published as a conference paper at COLM 2025
C.2.2 Evaluation Setup
Model Generation Settings To ensure consistency and equitable comparison with prior works (Liu et al., 2023b; Zhou et al., 2024a;b; Zhao et al., 2023; Wang et al., 2024b; Sun et al., 2023; Yu al., 2024b), we set the generation temperature to 0 and the number of beams to for all base, prior work and VAPR models.
For benchmarks such as LLaVA-bench, ConTextual, and MM-Vet, which require OpenAI APIs for evaluation, utilizee GPT-4 to align with prior methodologies. is important to note that GPT-4 versions evolve due to periodic updates. For instance, prior works may have used GPT-4-0314, which was deprecated in June 2024 To ensure consistency, we fix the version GPT-4-0613, the current stable release, and calculate scores for these benchmarks across all baseline and VAPR models. Further, MathVista utilizes GPT-4-Turbo calls for answer extraction. To minimize our costs, we instead used GPT-4o, which is a more performant OpenAl model that is significantly cheaper. This evaluation incurred total cost of approximately $800 (in addition to $300 for data generation using GPT-4o).
LLM-as-a-Judge :
Significance analysis For statistical significance analysis, we employ bootstrap resampling (Koehn, 2004), which involves randomly sampling 50% of each benchmark's data and evaluating model performance over 1,000 iterations for comparing two models. We assess each preference-tuned baseline and VaPR model against the base SFT model, reporting statistical significance when the win rate is > 95% (p 0.05). Notably, VaPR models demonstrate improvements over base SFT models, with most results achieving statistical significance. In contrast, while showing improvements, prior works fail to meet the 95% confidence threshold under the bootstrap test (except CSR-13B on the Pope dataset), as shown in Table 2.
1GPT4 Depracation History
46

---

Published as a conference paper at COLM 2025
Table 6: Performance comparison of LLa SaLaVA-v15-Instruce Qwen2VL-Instruuc Qwen2.5VL- Instruct, and DPO models finetuned on VAPR at three data scales 3K, 10K 30K, on ten benchmarks. Higher scores indicate better performance across all benchmarks, with the highest score for each benchmark highlighted in bold. PFT size refers to the preference for fine-tuning dataset size, where "-" represents no additional dataset. Each model is trained under identical hyperparameter settings.
Row
Miroio
PFT
LLAVA
CovT
MMV
SEED!
CV
MV
MMMU
MMS
POPE
NB
LLaVA-155-B
64.8
16.8
30,9
66.2
62.1
30.1
35.4
32.6
85.9
12.7
+ VAPR DPO
3K
69.9
19.2
31.4
66.1
61.9
30.1
35.4
32,9
85.1
13.6
VAPR DPO
10K
74.4
20.2
32.3
66.4
62.3
30.4
35.6
34.0
85.2
14.0
+ VAPR DPO
30K
76.2
20.6
32,9
66.7
62.9
30.8
35.7
34.7
85.4
14.5
LLaVA-15-13B
30.7
72.3
18.6
36.7
68.2
62.5
30.7
36.1
33.8
86.0
14.9
+ VAPR DPO
3K
75.6
19.4
36.3
68.3
63.5
31.1
35.2
34.9
86.0
15.8
VAPR DPO
10K
78.9
20.3
37.0
68.4
64.2
31.8
35.6
35.3
86.2
17.4
8
+ VAPR DPO
30K
80.5
21.2
37.3
68.7
64.6
32.3
35.8
35.6
86.3
18.2
9
Qwen2VVL-28
83.2
27.7
53.3
73.6
66.5
51.0
38.7
43.4
86.5
24.3
+ VAPR DPO
3K
83.8
31.6
52.6
73.6
67.5
50.2
38.7
43.4
87.6
24.9
VAPR DPO
10K
84.3
33.2
53.4
73.8
68.3
50.5
43.5
88.2
25.2
12
+ VAPR DPO
30K
88.1
34.8
54.1
74.0
69.0
50.9
39.2
43.7
88.3
25.7
Qwen2VL-7B
92.5
39.7
62.1
76.4
75.7
57.5
50.7
56.7
87.3
30.8
+ VAPR DPO
3K
92.8
41.9
63.0
76.4
75.8
57.2
50.6
57.1
87.0
31.7
VAPR
10K
87.2
16
+ VAPR DPO
30K
96.2
43.9
65.4
76.8
76.3
58.2
50.0
57.8
87.3
32.5
Qwen225VVL33
98.1
37.2
67.3
75.0
71.5
52.5
54.7
86.3
25.4
18
+ VAPR DPO
3K
95.1
38.0
66.8
75.0
71.6
52.5
45.3
55.0
86.0
23.5
VAPR DPO
10K
52.7
25.7
20
+ VAPR DPO
30K
97.1
40.3
67.4
75.5
72.7
53.4
44.9
56.1
86.4
26.3
21
Qwen2.5VI.
101.4
53.3
71.0
77.7
80.1
58.6
50.9
61.9
86.3
32.0
22
+ VAPR DPO
3K
100.6
53.1
71.2
77.6
80.5
58.6
50.9
62.0
86.4
32.0
24
VAPR DPO
30K
101.5
53.4
72.4
77.8
81.1
59.8
50.6
62.5
86.9
32.8
D Extended Results
D.1 Data Scaling
47

---

Published as a conference paper at COLM 2025
D.2 Benchmark detailed results
We provide the breakdown of model performance across different task categories for two comprehensive dataset MMStar and CV Bench.
Table 7: MMStar:
Performance is compared across LLaVA-v1.--Instrucc Qwen2VI-IInsrruc" Awen2.5VVI-Instruue and DPO models preference finetuned on VAPR (30K subset created with GPT-4o). Higher scores indicate better performance, with the top result for each benchmark shown in bold. All models share the same hyperparameters. Abbreviations: CP Coarse Perception, FGP Fine- Vine-grained Perception, IR Instance Reasoning, LR Logical Reasoning, S&T = Science & Technology.
Model
Final Score
CP
FGP
IR
LR
S&T
Math
32.6
58.8
26.8
40.0
26.0
17.2
26.8
VaPR-LLaVA-7B
34.7
62.0
27.2
44.0
27.2
18.0
30.0
LLaVA-v1..5Instrrut-138
33.8
58.0
27.2
42.4
26.4
21.2
27.6
VaPR-LLaVA--3B
35.6
60.8
28.4
47.6
28.0
23.6
25.2
Qwen2VL-Instruct-2B
43,4
52,4
41.6
51.6
43.2
31.2
40.4
VaPR-Qwen2VL-22B
43.7
53.6
45.6
50.0
42.0
31.6
39.6
Qwen2VL-Instruct-7B
56.7
67.2
50.4
62.8
56.4
46.4
57.2
VaPR-Qwen2VI-77B
57.8
67.6
51.2
63.2
57.6
49.2
58.0
Qwen2.5VL--nnstuuc-BBB
54.7
66.4
46.4
60.8
54.8
39.6
60.4
VaPR-OWee2.5VL--BB
56.1
68.4
47.2
63.6
55.2
39.6
62.4
Qwen2.5VL-hnstruct-7B VaPR--Qwe2..VV--7B
61.9
72.0
54.0
70.8
63.2
44.8
66.4
62.5
71.6
55.2
70.0
64.4
45.6
68.4
Table CV Bench: Performance is compared across LLaVA-v1.5-Instrue Qwen2VL- Instruct, Owen2.5VL-Instruct, and DPO models prefferecce finetuned on VAPR (30K subset created with GPT-4o). Higher scores indicate better performance, with the top result for each benchmark shown in bold. All models share the same hyperparameters. Abbreviations: Overall Overall Accuracy, Count Count Accuracy, Spatial Spatial Relation Accuracy, Depth Depth (Order) Accuracy, Distance Relative Distance Accuracy, Depth: Determine which the two distinct objects is closer to the camera, and Relative Distance: Determine which of the two distinct objects is closer to the anchor object.
Model
Overall
Count
Spatial
Depth
Distance
LLaVAAvi..--nnsrrucee
62.2
54.3
71.2
70.0
56.3
VaPR-LLaVA-7B
62.9
57.2
70.8
71.7
54.3
LLaVA-v1.5Innttruct-1BB
62.5
58.8
68.2
69.7
55.8
VaPR-LLaVA-13B
64.6
58.9
69.9
72.7
59.7
Qwen2VL-Instruct228
66.5
66.6
67.5
65.8
67.5
VaPR-Qwen2VL--2B
69.0
68.2
68.3
68.5
72.2
Qwen2VL-Instruct-77
75.7
67.0
79.5
85.5
72.8
VaPR-Qwen2VI--7B
76.3
66.6
81.9
82.7
76.3
Qwenn..VVL-Instruut:BBB
71.5
68.5
74.8
78.2
66.3
VaPR-QWen2.5V1--3B
72.7
68.9
75.1
79.3
69.2
Qwenn.5VV--Instruc-77
80.1
68.5
90.0
86.7
78.5
VaPR-QWen2.5VL--78
81.1
69.2
90.6
86.8
81.2
From Tables 2,7 & 8, observe that VaPR models consistently improve on perception tasks (particularly fine-grained perception), and reasoning tasks, such as spatial relationships (even complex ones like distance and depth) and counting. These gains align with their improved
48

---

Published as a conference paper at COLM 2025
performance SeedBench and NaturalBench, both of which emphasize visio-linguistic compositionality, perception, and reasoning.
Notably, despite not being explicitly trained on OCR, textual reasoning, math tasks, VaPR models achieve strong performance in these areas. We attribute this to their enhanced fine-grained perception, spatial reasoning, and counting capabilities, which demonstrates that improvements in these areas support interpretation of embedded text (consistent with prior work (Fu et al., 2024)) and geometric figures. This trend is further corroborated by gains on ConTextual and MathVista Vista benchmarks. Interestingly, VaPR-Qwen2VL L--22 achieves the largest gains on Pope, which can be explained by pronounced improvements in fine-grained perception (as evident in MMStar). On the other hand, it shows slight degradation in math, logical reasoning tasks, which can explain why does not improve on Math Vista, where the other models do (see Table 2).
Lastly, prior work (Ivison et al., 2024a) indicates that preference optimization primarily enhances truthfulness and alignment, rather than factuality, which aligns with our observed improvements perception and reasoning but not in purely knowledge-basse tasks. We hypothesize that limited gains in MMMU (see Table 2) can be attributed to the alignment tax VaPR models possibly pay in knowledge based tasks.
49

---

Published as a conference paper at COLM 2025
D.3 Preference Dataset Comparison
D.3.1 Training Setup
We compare VaPR with two alternative preference data generation techniques: (1) POVID, which uses GPT-4V to generate rejected responses given the image, instruction, and ground truth response, and (2) SIMA, which follows self self-preference generation paradigm by synthesizing two responses one via greedy decoding and another via sampling (temperature critic to select the preferred response. For SIMA, apply the generation method to the 10K subset of VAPR dataset; for POVID, we use the publicly released data (17K). For VAPR we select the 10K subset for training ensure fair comparison. We preference finetune LLaVA-v1.5 and Qwen2VL models on the above datasets. To ensure consistency, all models are trained for 50K steps, under identical hyperparameter settings. This setup enables a balanced and computationally efficient comparison, allowing us evaluate all methods under fixed training budget without an excessive number of experiments.
= and using the same VLM
Table 9: Performance comparison of LLaV1-11.5, Qwen2VL-Instruct, DPO models finetuned on VaPR, SIMA & POVID across 2B, 7B, and 13B parameter sizes on ten benchmarks. Higher scores indicate better performance across all benchmarks, with the highest score for each benchmark highlighted in bold. PFT size refers to the preference for fine-tuning dataset size, where "L" represents no additional dataset.
Rew
METON
PFT
ILAVAW
CoNT
MMV
SEED!
CV
MV
MMMU
MMS
POPE
NB
LLVVA-15-7B
64.8
16.8
30,9
66.2
30.1
35.4
32.6
12.7
VAPR-SIMA DPO
10K
68.5
17.2
31.6
66.0
60.9
29.4
35.2
32.7
85.2
12.9
Puvid DPO
17K
67.2
18.0
30.9
66.1
61.6
30.1
35.4
33.3
85.9
13.2
VAPR DPO
10K
74.4
20.2
32.3
66.4
62.3
30.4
35.6
34.0
85.2
14.0
LLaVA-155-1BB
72.3
36.7
30.7
VAPR-SIMA DPO
10K
73.3
18.2
35,5
67.7
61.0
31.2
34.5
33.6
86.0
13.1
Povid DPO
17K
74.5
19.8
34.6
68.1
63.9
31.5
34.9
34.1
86.2
15.3
VAPR DPO
10K
78.9
20.3
37.0
68.4
31.8
35.6
35.3
86.2
17.4
Owen2VL-2B
83.2
27.7
53.3
73.6
66.5
51.0
38.7
43.4
86.5
24.3
10
VAPR-SIMA DPO
10K
81.6
29.2
49.1
73.4
66.7
50.0
38.8
43.1
86.8
23.4
11
Pavid DPO
17K
82.7
30.7
49.3
73.7
67.1
50.2
38.9
43.2
87.1
23.6
VAPR DPO
10K
84.3
33.2
53.4
73.8
50.5
39.0
43.5
88.2
25.2
13
Qwen2VL-7B
92.5
39.7
62.1
76.4
75.7
57.5
50.7
56.7
87.3
30.8
14
VAPR-SIMA DPO
10K
90.1
38.5
62.9
76.1
75.4
56.5
50.0
57.0
87.2
30.5
15
Povid DPO
17K
90.9
37.4
63.5
76.2
75.8
57.0
50.4
57.2
87.6
31.1
VAPR DPO
10K
57.4
50.5
87.2
D.3.2 Analysis
From Table 9, we observe that VAPR models consistently outperform both SIMA and POVID across model families. SIMA yields minimal to no improvements over base models and often degrades performance, especially for Qwen2VL. POVID achieves moderate gains for LLaVA but underperforms on Qwen2VL. On average, VaPR outperforms SIMA by 5-6% on LLaVA and 3-4% on Qwen2VL, and POVID by 4-5% and 2-3%, respectively. To understand these differences, we analyze DPO optimization using LLaV---1.--B as a representative model.
The DPO loss can be re-written as:
Cro log or (a [log uip (yw x) - log To(y1 xx) - (log (y xx) - log Tyer |t xx)])
where Ta is the learned policy, ahe the fixed reference model, and is the temperature scaling factor. Let:
Ag = log Top (Vw x) - log n (y) x),
Aref = log Trel (vw x) log T x).
50

---

Published as a conference paper at COLM 2025
Average Log Probabilities
0
Reward Accuracy (BMA Smoothed
0.8-
Log Probabiity
-20
-30
0.4
0.2
SIMA
-50
Chosen Oujected
VaPR
POVID
SIMA
POVID
VaPR
200
400
600
800
1000
1200
1400
Dataset
Training Steps
(a) Reference model prg--robabillties
(b) Reward accuracy trends
Figure 27: Comparison of preference datasets. (a) Average reference model log- probabilities for chosen vs. rejected responses across VaPR, SIMA, and POVID lower values indicate lower reference likelihood. (b) Reward accuracy trends over training steps show that SIMA improves gradually while POVID saturates quickly, Repeating the figure for ease of reference
the loss simplifies to: LDO - log (a(Ag Are)))
For POVID, we observe that Are already higher than VaPR while having a similar log
probability of chosen response under the reference model (see Fig. 27a), indicating that POVID rejected responses are substantially less likely under the reference model as compared VaPR's. This can be attributed to the higher linguistic and stylistic differences in POVID preference pairs as compared to VaPR (see Table 1). From Fig. 27b, we observe that the POVID model rapidly attains reward accuracy of 1, suggesting that the model may be overfitting based on preference signals derived from length and stylistic differences instead of content differences alone. In contrast, VAPR achieves its highest reward accuracy more gradually and does not converge to 1, indicating reduced overfitting due to exposure to more challenging preference pairs.
In contrast, for SIMA, we observe Arel R O on average (see Fig, 27a), indicating that chosen often nearly identical. Manual inspection confirms this, with ~ 20% of pairs being exact duplicates and many others highly similar. When Aref Rs 0, the DPO loss is driven entirely by Ag, removing the reference model's regularizing influence. This causes the optimizer to treat weak or noisy preference signals informative, leading to overconfident updates based on superficial differences. Consequently, the model struggles to distinguish between responses reflected in 50% reward accuracy (see Fig. 27b), potentially learning undesirable behaviors that harm downstream performance. Other self ooffprrfeerenc methods, such RLAIF-V and CSR, which employ multi-step generation and scoring procedures and generate preference pairs with greater stylistic and linguistic variation (see Table 1), can mitigate the above issue, but similar to POVID, they may also inadvertently exploit the stylistic and length biases.
and rejected responses
On the other hand, VAPR explicitly controls for stylistic and length similarity when construct- ing preference pairs, ensuring that DPO learns from content-level differences. However, in low-resource settings where high-quality SFT data may be limited (as required by VAPR), our method can complement preferreeee generation approaches. For instance, one could leverage high-confidence responses from methods like CSR or RLAIF-V chosen responses, and apply the VaPR pipeline to generate targeted hard negative creating high-quality preference pairs in an unsupervised manner. We view this as promising direction for future work and invite the community to explore such hybrid strategies.
51

---

Published as a conference paper at COLM 2025
Table 10: Performance compared across LLaVA-w1.5-Instruut Qwen2VL-Instruct Owen2.5VL-Instructt and DPO models, all preference finetuned on VAPR (10K subset created with GPT-4o) or VxPR-OS (8K samples generated by the Open-source model Qwen3- 32b). Aside from the model used for generation, the framework and prompts are identical. All models share the same hyperparameters. Higher scores indicate better performance, with the top result for each benchmark shown in bold. In case two models get the top scores, both are bolded, otherwise top score is bolded and the second highest score(s) is underlined.
Row
METHOD
LLAVA
CoNT
MMV
SEED!
CV
MV
MMMU
MMS
POPE
NB
1
LLaVA-15-7B-Insttee
64.8
16.8
30.9
66.2
62.1
30.1
35.4
32.6
85,9
12.7
2
+ VAPR-OS DPO
73.3
18.7
32.1
66.3
62.3
30.2
35.6
33.7
83.6
13.9
VAPR DPO
32.3
34.0
85.2
Owen2VL.22BInnsteee
83.2
27.7
53.3
73.6
66.5
51.0
38.7
43.4
86.5
24.3
5
VAPR-OS DPO
84.1
32.8
53.3
73.7
67.9
50.2
38.9
43.5
88.0
25.2
+ VAPR DPO
84.3
33.2
53.4
73.8
68.3
50.5
39.0
43.5
88.2
25.2
7
22een25VL-300-nneee
98.1
37.2
67.3
75.0
71.5
52.5
45.7
54.7
86.3
25,4
8
+ VAPR-OS DPO
95.7
39.0
63.4
73.3
72.0
52.4
45.5
55.4
86.2
25.7
VAPR DPO
96.5
66.9
52.7
55.6
86.1
25.7
D.4 Open-Source Editor data results
We preference finetuned LLaVA-v1.5-Instrucee I-Inntruct--,, @wen2VI--nnsrructe and Qwen2.5-VL. Instruct-3B on VaPR-OS (VaPR Open source), where the difference between the models are two aspects: (a) is missing reasoning and captioning samples (1K each), reason shared below (b) uses an open-weight model for generating responses. Key findings include:
Dataset Quality: VaPR-OS rejected responses exhibit similar hard-negative properties to those generated by GPT--4,, with an average token length difference of 6 (compared to 3 in VaPR) and a Levenshtein distance of 10 (vs. in VaPR).
Model Performance: From Table 10, observe that models trained the open-weight dataset achieve 99% of the performance of those generated with GPT-4o, with both consistently outperforming the baseline most benchmarks. This is expected, VaPR-OS samples overlap with VaPR and exhibit similar linguistic properties and average token length. These results highlight that the VaPR pipeline generalizes effectively and not limited to weight models.
Interestingly, we find that the contribution of captioning and abstract reasoning tasks to preference learning is limited, which is consistent with prior work (Lai et al., 2024) suggesting that complex tasks like reasoning may benefit stepwise decomposition of preference datasets. In our case, captioning and abstract reasoning tasks (e.g. Considering the presence of two clocks on the building, what purpose might this architectural design serve?), can be decomposed into simpler components like fine-grained perception (e.g., attribute recognition) and spatial reasoning (e.g,, object location). Training models using these atomic tasks as step-wise preference samples may collectively support learning for more complex tasks, direction we plan to investigate further.
Limitations and Future Directions for OS editors: We observed that data generated using Qwen3 sometimes fails to consistently perturb dependent spans (e.g,, object perturbation: " bathroom ... has a large bathtub
... kitchen... ... has large bathtub... ... whereas GPT-4o correctly changes it to ".. kitchen has large oven ...".. "), with this issue being prominent in captioning and abstract reasoning tasks. This could potentially add noise to the dataset and thus we omit these samples in our analysis. To mitigate this issue, we plan to experiment with more open-source models and generation of step-wise preference datasets.
52


Reviews:
[{"review_id": "Z6L2aqVea6", "reviewer": "Reviewer_2MKa", "summary": "The main contribution of the paper is a creation of balanced preferences dataset for vision-language (VL) tasks. The novelty is creating a preference data for a sample of LLaVA dataset (30K preference pairs). The experiments show performance improvements via DPO optimizaiton compared to multiple baselines. The paper is well written, the contribution is clearly defined, and the empirical investigation is sufficiently thorough. The main concern I have is whether a dataset contribution (moreover, a derived dataset) is a sufficient contribution for a COLM paper.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Well written paper, and a potentially valuable dataset contribution\n* Table 2 demonstrates consistent improvements across benchmarks via DPO\n* Nice analysis of things like scaling, and ablations of other approaches, like SFT", "reasons_to_reject": "* It is not clear whether the contribution of a dataset, while self-contained and significant, is novel enough to warrant an acceptance. Preference-based optimization is known to be effective from prior work, and the dataset construction (sampling + generating a preference pair) is not novel either. **\n\n** The author responses mostly addressed this concern.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pH29Mep7iC", "reviewer": "Reviewer_iz1D", "summary": "The paper introduces VaPR, a preference dataset of hard negative answers for vision-language tasks generated using a three-step pipeline. The idea is to collect meaningful negative responses as hard negatives while keeping the same length and style as the correct answer. The authors perform several experiments, using this dataset as a training mixture for vision-language models (LLaVA, Qwen VL 2 and 2.5), and test their performance on popular benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Models trained on VaPR perform remarkably on popular benchmarks. Additionally, the authors provide comparative results against models trained on alternative datasets.\n- A meaningful subset of dataset instances was manually validated by humans, ensuring the high quality of the dataset.\n- Extensive study of the generated outputs, with an interesting analysis regarding \"Yes/No\" questions.", "reasons_to_reject": "- The proposed pipeline doesn't introduce any meaningful novelty with respect to the state-of-the-art. Many works use Large Language Models for synthetic generation. Additionally, there are works that consider their usage for hard negatives (e.g., in mme5 [1], GPT4-4o was used for plausibility checks of hard negatives obtained from a CLIP model).\nThe dataset was generated using GPT-4o, limiting the study's reproducibility. Furthermore, the proposed pipeline's generalizability and soundness are limited due to its reliance on GPT-4o. The work would benefit from additional experimentation, potentially incorporating open-weight models.\n\n[1]: Chen, Haonan, et al. \"mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data.\" arXiv preprint arXiv:2502.08468 (2025).", "questions_to_authors": "While the paper focuses on stylistic and length differences, it is unclear how this is addressed in the generation step. After checking the Appendix of the work, it seems to me that this is provided as conditioning information, that is, the prompt expressively states to follow guidelines regarding the style and length of the responses. Is this correct? If so, can this be better described when the generation step of the pipeline is showcased?", "extra_scores": {}}, {"review_id": "m1olAfkn6n", "reviewer": "Reviewer_W6PJ", "summary": "This paper introduces VAPR, a new dataset and training framework designed to improve preference alignment in Large Vision-Language Models (LVLMs) via hard-negative generation. The authors construct a 30K-sample dataset where the rejected responses are synthetically generated to closely match the length and style of the accepted ones but contain task-specific semantic errors. The resulting models, fine-tuned via DPO across LLaVA and Qwen2VL families, demonstrate consistent improvements across 10 benchmarks, with particular gains in reasoning and adversarial settings.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper clearly articulates a real and pressing issue in preference data construction for LVLMs—namely, that stylistic/length biases contaminate learning. The dataset can have an impact on post-training of vision language models. \n\nVAPR-tuned models show consistent performance gains on vision-centric benchmarks such as SEED, CV-Bench, and NaturalBench with different backbones such as Llava and Qwen-2-VL, which outperform strong baselines including SIMA, CSR, and RLAIF-V.\n\nThe paper includes thoughtful metrics like Levenshtein distance, log-probability gaps under the reference model, and reward saturation curves, These results provide additional empirical evidence to verify that VAPR mitigates shortcut learning and improves true preference understanding.", "reasons_to_reject": "The method depends on GPT-4o’s ability to precisely introduce task-relevant semantic errors. There is no ablation on using weaker or open-source editors, leaving it unclear whether the pipeline is usable in constrained settings.\n\nIt would be even better if the authors could provide some additional analysis on why this proposed method will lead to performance improvements. For example, which types of tasks will benefit from the synthetic data?", "questions_to_authors": "The VAPR SFT results sometimes degrade performance, especially in strong pretrained models (Qwen2.5VL). While DPO seems to mitigate this, it raises questions about how sensitive the models are to overfitting on synthetic negatives.\n\nHow does VAPR perform when combined with other reward modeling approaches (e.g., PPO or GRPO)?", "extra_scores": {}}, {"review_id": "eVK6KmZQ2g", "reviewer": "Reviewer_gWgY", "summary": "This paper studies how preference dataset should be created for Large Vision-Language Models (LVLMs) fine-tuning.\nThe authors introduce a hard-negative response generation framework that produces rejected responses with targeted errors, maintaining stylistic and length similarity to the accepted ones. Using this framework, the authors develop the VaPR dataset, comprising 30K high-quality samples, to fine-tune three LVLM families.. The resulting models achieve performance improvements across ten benchmarks. The authors conduct scaling analysis, which shows that performance consistently improves with data size.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. I think it is a great contribution to create a such a preference dataset. I think the authors' idea to generate rejected responses while maintaining stylistic and length similarity to the accepted ones is reasonable and interesting.\n\n2. The performance improvement after applying the dataset is significant. The authors conducted extensive experiment in many benchmark problems using several models. I think the empirical evaluation is reliable.", "reasons_to_reject": "1. While I do admit that releasing new datasets is a huge contribution, the data generation process in this paper is reasonable yet not very novel. \n\n2. To be honest, I would like to see more examples of the dataset and check the source code. I am not fully sure whether this is possible according to the COLM rule.\n\n### Minor comments\n- Please properly use \\citep or \\citet for citations.", "questions_to_authors": "[Q1] While the authors discuss the scaling of the proposed method, Figure 5 does not seem to use a logarithmic scale. Also, x-axis is {3k, 10k, 30k}, but this these three values are placed at the same interval. This is strange to me and makes the readers understand how the proposed method scale for training set size. Could you explain why the authors decided to use this Figure 5 and how this relates to so-called scaling law discussed in the existing work?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a data generation framework via hard negative example synthesis for VLMs. The reviewers all agrees the novelty and the nice analysis in the experiments.
Decision: Accept