Probing Visual Language Priors in VLMs
Authors: Tiange Luo
Ang Cao
Gunhee Lee
Justin Johnson
Honglak Lee
Venue: ICML 2025 poster (2025)
Keywords: Visual Language Model
Visual Language Priors
DPO
Primary area: applications->computer_vision
TL;DR: Exploring the visual language priors in VLMs by constructing Question-Image-Answer triplets that deliberately deviate from the training data distribution. Proposed Image-DPO to encourage the model to use more visual inputs.

Abstract: Vision-Language Models (VLMs) may over-rely on visual language priors from their training data rather than true visual reasoning. To investigate this, we introduce ViLP, a benchmark featuring deliberately out-of-distribution images synthesized via image generation models and out-of-distribution Q\&A pairs. Each question in ViLP is coupled with three potential answers and three corresponding images: one that can be resolved by text priors alone and two that demand visual reasoning. Although humans achieve near-perfect accuracy, modern VLMs falter; for instance, GPT-4o achieves only 66.17\% on ViLP. To alleviate this, we propose a self-improving framework in which models generate new VQA data and then apply pixel-level and semantic corruptions to form ``good-bad" image pairs for self-training. Our proposed training objective, Image-DPO, compels VLMs to focus more on the actual visual inputs, and we demonstrate its effectiveness in LLaVA-v1.5 and Cambrian. Project Page: \href{https://vilp-team.github.io/}{ViLP}.

Probing Visual Language Priors in VLMs
Tiange Luo Ang Cao 1+
Gunhee Lee²
Justin Johnson' Honglak Lee 1
Abstract
with atypical spot patterns (Figure 1), which humans would easily discemn. This raises important question: do today's s VLMs still over-rely on learned visual language priors, es- pecially given that they rely on far fewer image-text pairs than the internet-scale text corpora used for pretraining?
Vision-Language Models (VLMs) may over-rely visual language priors from their training data rather than true visual reasoning. To investi- gate this, we introduce ViLP, a benchmark fea- turing deliberately out-of-distribution images syn- thesized via image generation models and out-of- distribution Q&A pairs. Each question in ViLP is coupled with three potential answers and three corresponding images: one that can be resolved by text priors alone and two that demand visual reasoning. Although humans achieve near-perfect accuracy, modern VLMs falter; for instance, GPT- 4o achieves only 66. 17% on ViLP. To alleviate this, we propose a self-improving framework in which models generate new VQA data and then apply pixel-level and semantic corruptions to form "good-bad" image pairs for self-training. Our pro- posed training objective, Image-DPO, compels VLMs to focus more on the actual visual inputs, and we demonstrate its effectiveness in LLaVA- v1.5 and Cambrian. Project Page: ViLP.
To investigate this, we probe the Visual Language Priors of VLMs by constructing Question- Answer (QIA) triplets that deliherately deviate from the training data dis- tribution. Unlike existing benchmarks that typically rely internet-sourced images (Goyal et al., 2017: Tong et al., 2024), which inadvertently favor the visual language priors embedded in VLM training data, we utilize modern image generation models, including DALL-E-3 (Ramesh et al., 2021) and Flux, to synthesize out-of-distribution image- answer pairs that also feature notable variation in texture, shape, conceptual combinations, hallucinated elements, and proverb-based contexts.
Our benchmark, ViLP, contains 300 carefully designed ques- tions, each paired with three distinct answers: a Prior An- swer and two Test Answers, resulting in total of 900 QIA triplets. To further challenge the priors of VLMs, we am- plify language priors in questions by introducing distractor facts: each question is structured to present distractor fact followed by question. The distractor fact directly leads to the Prior Answer. In contrast, the two Test Answers are crafted to challenge the priors by requiring both textual and visual cues for accurate reasoning. While human partici- pants achieved ~98% accuracy easily, current VLMs exhibit considerable difficulty, evidenced by significant per- formance drop on our benchmarks, with GPT-4o (OpenAI, 2024) scoring only 66. 67%.
1. Introduction
Lasign-Languae Models (VLMs) have advanced text-image interaction, bridging the gap between visual and textual data (Achiam et al., 2023; Team et al., 2023). However, persistent challenge for learning-based models, such ResNets and CLIPs, lies in their reliance on learned priors from the training data, sometimes overlooking visual cues when answering questions (Agrawal et al., 2016; Prabhu et al., 2023). For example, when shown a torus-shaped soc- cer ball (Figure I), model might incorrectly identify as a sphere due to strong language priors. Simultaneously, these models may adhere to learned visual priors (Thrush al., 2022; Sterz et al., 2024), which hampers their ability to comprehend dustof-distribution visual cues, such as zebra
Motivated by the results of ViLP, we propose Image-DPO, a self-improving method for enhancing VLM visual reasoning by increasing reliance on visual inputs. Our method employs self- generated VQAs using image generation and editing models (Podell al., 2023; Ren et al., 2024; Brooks et al., 2023a) and applies controlled corruptions to create "good- bad" image pairs for DPO-like training (Rafailov et al., 2024). Experiments with open-source VLMs. including LLaVA-v1.5 (Liu et al., 2024a) and Cambrian (Tong et al., 2024), demonstrate its effectiveness. Moreover, we theoreti- cally prove our objective optimizes an upper bound on the RL fine-tuning objective (Rafailov et al., 2024).
'Jint fist authorship, 'Equal advising 'University of Michi- gan *LGAI Research. Correspondence to: Tiange Luo <tian- gel@ umich.edus, Ang Cao <ancao @ umu.h.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Probing Visual Language Priors in VLMs
Sphera
Cube
Torus
Stripes
Spots
Chackerboard
A soccer ball typically sphere. What is the geometric shape of the large soccer ball the image?
Zebras usually have stripes. What pattern is the zebra's body in the image?
Flamingo
Tiger
Penguin
One
Five
Flamingos often stand on one leg. In image, what pink animal is standing on leg in the water?
A common phrase is "kill two birds with stone" In the image, how many stones are being used to kill two birds?
Einstein
Newton
Tesia
Pisa
Eiffel
Bigteen
E-aMC2
Einstein created the famous equation E=mc Which scientist is shown drawing this equation in the image?
The Leaning Tower of Pisa is famous for it's tilt. What tower is leaning the image?
Brick
Pitaye
Fork
Sheep
Pig
Chicken
Houses are usually built from bricks. What object is the house the image built from?
Sheep usually produce wool. What farm animal in the middle of the image is producing wool?
Umbrella
Plunger
Blender
Elephant
Camel
An umbrella keeps you dry in the rain. What does the center character in the picture use on his head to stay dry?
Giraffes are known for their long necks. What is the tallest long-necked animal shown in the image?
Figure 1: Sample data from ViLP. For the same question, ViLP provides three distinct images and corresponding answers (upper-left corner). All questions follow consistent structure, combining distractor fact with a question. The Prior Answer (first column) can be directly inferred from the question, while Test Answers (second & third column) rely on visual cues. Our answers designed to be single words, and both the model and human evaluators tasked with open-domain answering, rather than selecting from predefined options. To support this, we have developed robust synonym and plural detection pipeline, ensuring that open-ended responses do not hinder the evaluation process. This approach also enables evaluation without relying on LLMss. Please refer to Appendix B. for more data samples from ViL.. We investigate the impact of image styles in Appendix B.3, where we generate more realistic images using image generation. Furthermore, we include both qualitative and quantitative comparison results with Winoground (Thrush al., 2022). Whoops!(Bitton Guuttta et al., 2023a), and HallusionBench(Guaa al., 2023) Appendix B.4.
2

---

Probing Visual Language Priors in VLMs
2. Related Work
Dancette al., 2021; Wu al., 2022; Ramakrishnan et al., 2018; Gouthaman & Mittal, 2020) addressed these issues with curated simulators (Johnson et al., 2016) or internet imagery (Zhang et al., 2016). In this paper, we present a novel VQA benchmark featuring carefully designed ques- tions, fact-based distractors, rare-distribution answers, and image generation techniques to produce realistic visuals that challenge learned visual language priors (Figure 1).
VQA Dataset: Significant efforts have produced
VQA datasets from various angles, including general
VQA (Agrawal et al., 2015; Gurari et al., 2018: Fu al., 2023; Liu et al., 2023d; Li et al., 2023a; Yu et al., 2023b; Liu et al., 2024a), reading text or charts (Singh et al., 2019a; Mathew ct al., 2020; 2021; Masry et al., 2022), complex reasoning (Lu et al., 2022: 2023), composition probing (Hudson& & Manning. 2019; Ma et al., 2022; Thrush et al., 2022; Hsieh et al., 2023; Li et al., 2024a), hallu- cinations (Rohrbach et al., 2018; Li et al., 2023c; Guan et al., 2023), common-sense reasoning (Bitton-Guetta Guetta et al., 2023b;a), and more (Majumdar et al., 2024; Sterz et al., 2024). We propose a benchmark that tests VLMs' visual rea- soning when questions, answers, and images defy common patterns. Following the balanced dataset design of (Goyal et all., 2017), each question is accompanied by three answers: one aligns with language priors, and two deviate, prompting reliance on visual cues. By leveraging state-of-the-art image generation models, our benchmark challenges these priors more effectively than previous datasets built from internet images (Goyal et al., 2017: Tong et al., 2024). Furthermore, unlike the "trick" category in (Sterz et al., 2024), we first generate question-answer pairs before synthesizing images under specified constraints, creating more difficult visual reasoning tasks. Comprehensive comparisons with existing datasets appear in Appendix B.4.
Self- Rewarinn VLM: Self-rewarding LLM (Yuan et al., 2024) has shown that LLMs can generate and improving themselves in the process via Directed Peference Optimiza- tion (DPO) (Rafailov et al., 2024). This approach extends to VLMs by generating new answers for DPO training (Zhou et al., 2024a; Deng et al., 2024; Zhou et al., 2024b; Wang et al., 2024c;a; Yue et al., 2024; Liu et al., 2024b; Xiao et al., 2024). Our work aligns with these self- reffrrenrddigg VLMs but differs in two key ways: (1) our proposed Image-DPO generates multiple images for single question-ine image pair (rather than multiple answers); (2) rather than relying solely on existing images (Zhu et al., 2024a), Image-DPO creates diverse new images using pre-trained generative models (SDXL (Podell al., 2023), GroundedSAM (Ren al., 2024), InstructPix2Pix (Brooks al., 2023a)). Further- more, Image-DPO DPO deliberately corrupts images to produce multiple degraded versions that serve as rejected data in DPO training. Concurrent works (Wang et al., 2024a; Xie et al., 2024) explore similar methods but lack a benchmark to verify enhanced visual focus, fail to establish thcoreti- cal connections between their proposed objective and DPO, and utilize limited image transformations (e.g.. only ran- domly cropping). In contrast, we introduce ViLP (Section 3) to assess visual reasoning and provide theoretical founda- tions (Appendix A), alongside multi-category image corrup- tions (semantic editing, Gaussian blurring, pixelation).
Vision Language Models and Language Priors: Multi- modal reasoning is crucial for machine intelligence, with VLMs integrating visual perception, text reasoning, instruc- tion following, and generation for complex tasks (Tan & Bansal, 2019; Li et al., 2019; Kim et al., 2021; Wang et al., 2021b;a; Alayrac et al., 2022; Li et al., 2023b; Chen al., 2022; Jia et al., 2021; Shen et al., 2021; Singh et al., 2021;
Liu et al., 2023c;a; Zhao et al., 2023a; Chen et al., 2023;
3. ViLP Benchmark
Zhu et al., 2024b; Li et al., 2024c; Dai et al., 2023; Li et al.. 2024e; Yu et al., 2024; Dai et al., 2024; Deitke et al., 2024). Inspired by the success of large language models (Brown et al., 2020; OpenAI, 2023a; Touvron et al., 2023a;b; Chi- ang et al., 2023) and pre-trained visual encoders (Radford et al., 2021: Desai & Johnson, 2020; Caron et al., 2021: Chen et al., 2024), many recent methods leverage relatively small vision-language paired datasets (Liu et al., 2024a; Tong et al., 2024) to fine-tune connectors between LLMs and visual backbones (Liu et al., 2024a). However, these datasets are far smaller than the vast text corpora for LLM pre-training (OpenAI, 2023b; Soldaini et al., 2024), and freezing the visual encoder and LLM parameters often pre- serves language biases, causing visual inputs to be over- shadowed (Thrush et al., 2022; Sterz et al., 2024). This challenge is amplified by deliberately generated images that expose such biases, as shown in our study. Previous works and datasets (Goyal et al., 2016; Agrawal et al., 2017:
3.1. Design Principles
"What's the tall animal with the longest neck shown?" Hu- mans readily guess "giraffe" based on learned priors, yet as shown in the bottom-right of Figure 1, it could be an ele- phant or camel - where visual reasoning corrects the answer. This highlights a potential shortfall in Vision-Language Models (VLMs), which may over-rely on learned visual language priors instead of true visual reasoning. particularly since VLMs are typically fine-tuned on limited image-text data, which is several orders of magnitude smaller than the trained text corpus (Liu et al., 2024a; Tong et al., 2024). Specifically for the scope of visual language priors in this paper, we target (1) strong language priors that lead VLMs to derive answers solely from text, and (2) potential visual priors causing models to overlook critical uncommon visual cues (e.g., unusual zebra spots in Figure 1).
3

---

Probing Visual Language Priors in VLMs
To evaluate how VLMs handle learned visual language pri- ors, we introduce ViL.,, specialized benchmark of out-of- distribution Question-Image-AAswee (QIA) triplets guided by two core principles. First, text-only inference ensures that each question can be answered with high confidence using textual clues alone. Second, visual inference requires that the correct answer-sometines contradicting common sense-only only emerges once an ditt---diitributionn image is considered. By forcing models to integrate both textual and visual information, ViLP reveals whether they truly engage in visual reasoning or merely rely on memorized patterns.
3.2. Question-Imag-AAsserr Generation
Following Criterion Three, deett should be highly improb- able based on Q alone yet the correct choice when paired with I. Since such images do not exist naturally, we use generative models like DALL-E-3 (Ramesh et al., 2021) and Flux to blend unusual elements that override typical lan- guage priors. We incorporate substantial human imput and leverage advanced LLMs such as OpenAl-ol and Claude- 3.5-Sonnet to ensure alignment with all the criteria. More details, including text prompts and average cost, are pro- vided in Appendix B.2. Note that as more advanced image generative models such as the recently introduced 4o im- age generation become available, we anticipate generating increasingly abundant and high-quality data for our bench- mark, yet future updates will remain consistent with the dataset construction criteria outlined in Section 3.1.
Mathematically, let Q be question, I an image, and A = (aptior..... ...., ..est.... } the set of possible answers. We define P(a Q) as the probability of answer a given Q alone and P(a 1) as the probability given both Q and 1. We consider prior model p. which may represent either human cognition (P (mmmmn) or VLM//LL''s learned visual- language prior (Po). For constructing our benchmark, we used the following guidances:
For each question, we design three answers: one aprior in- ferred solely from Q, and two Geest that defy language priors, requiring visual cues for correctness. We rely on GPT-4 to generate text prompts, produce large-scale images, and then conduct human filtering and refinement. This process faces two main challenges: (1) producing diverse out-of- distribution QA pairs, and (2) synthesizing images that defy specific priors, sometimes necessitating hundreds of sam- ples to find one that accurately matches Q and Crest-
The question Q alone should strongly favor Oprions where 51 a high-confidence threshold. Aprior usu- ally satisfies common knowledge, such as "soccer ball is sphere" and "Einstein created E mc?" (Figure 1).
Criterion One:
P(aprior
()= 2 81
(1)
Criterion Two: With the image J, the correct answer shifts to Gest, where 82 is another high-confidence threshold. The image also significantly impacts the answer distribution, where we let p = P(- Q) and q P(- Q, denote the answer distributions before and after seeing the image, respectively. Here, D is a divergence measure, and 63 is a threshold indicating significant difference. For instance, the image in the 1st row and the 3rd column of Figure I turns the answer to torus.
Ultimately, we curated 300 questions, each paired with three distinct sets,
Table 1: Category Statics.
Type
Frequency
image-answer totaling 900 QIA triplets. These cover a broad range from low-level recognition (texture,
Texture
Shape
20
Conceptual combinations
276
Hallucinated Components
Proverbs
17
shape) to high-level reasoning (conceptual combinations. hallucinated com- ponents, proverbs). Table 1 summarizes their categorical distribution, with each question spanning an average of 1.6 categories. To reinforce text priors, we present distractor fact before the question. Rigorous human review ensures that all final QIA triplets are clear and interpretable. as reflected by our human evaluation results in Table 2. Besides, in Appendix B.3, we investigate the impact of image styles by generating more realistic images via 4o image generation and comparing them to those produced by DALL-E-3 (Ramesh et al., 2021) and Flux. We find that realistic images can increase the difficulty of the task, highlighting their importance for future studies.
P(auat Q.11) 82. Dis(p, q) 2 5g
Criterion Three: The answer Grest should be rare and un- likely from Q alone, while Aprior becomes clearly incorrect when considering T. This enforced by a low-confidence threshold 6 (e.g., Newton as Gret inferred from the image, contradicting Einstein, shown by the image from the 3rd row and 2nd column of Figure 1).
P(arett Q) < 84, P(aprior Q,1) < 64
(3)
In designing ViLP, we leverage the human cognition prior Phuman as our guiding principle, ensuring each QIA con- figuration aligns with typical human expectations while requiring visual evidence to override strong textual assump- tions. We then compare the learned priors of VLMs and LLMs, denoted Pg. Pruman to evaluate whether these models genuinely engage in visual reasoning rather than relying on memorized patterns.
3.3. Dataset Evaluation
All of our questions are designed to elicit single-word an- swers, an approach that is more efficient and more reliable than sentence- based evaluations that rely on LLM judgment.
4

---

Probing Visual Language Priors in VLMs
?
Not
?
? A3
Soitaos Comuptcd
Goussian Bunndd
Image by
Comupted Image |1
A1
A
ImageDPO
VLM
Chosen
Rejeeted
(a) Image-DPO vs. Text-DPO: In the green dashed box, we illustrate Image-DPO, DPO, which uses a single Q&A pair paired with multiple corrupted images. In contrast, the purple dashed box presents TeXt-DPO, involving single input image paired with multiple, distinct Q&A pairs.
(b) Illustationn of Image DPO, We construct chosen and rejected pairs by cor- rupting the image with set of perturbations while keeping the Q&A unchanged. Perturbations include semantic editing, Gaussian blurring, and pixelation. The mathematical formulations and implementation details provided in Ap- pendix A and Appendix C, respectively.
By avoiding using LLM, we reduce API fees, computational overhead, and the risk of occasional inaccuracies due to in- correct model reasoning. We explicitly instruct the model to provide single- word answer, and we evaluate the cor- rectness of each response using binary system. To ensure fair evaluation, devote significant efforts to building comprehensive
VLM, is the reference VLM (typically an earlier version of To). S is the dataset of good and bad triplets, o is the sigmoid function, and a is scaling factor. In Proposition 1, we show that objective 4 optimizes an upper bound on the RL fine-tuning objective (Eq. (3) in (Rafailov er al., 2024)).
of synonyms and plural for each swer detect other valid alternative answers. This ensures that the model is only penalized for actual errors, not for providing synonymous or alternative correct responses.
LiT,, Teat) = -.
log Doga(o
(a_ (o | Qw)w) Teo Q
To (A(A O Te (A Q,
4. Image DPO
Intuitively, since the textual inputs and outputs are identical in both good and bad cases, the gradients of this objective push the model to rely more on the vision branch, driving shift in gradient direction when processing normal im- ages 4 compared to corrupted images (Figure 6). This behavior encourages the model to focus more image inputs rather than relying solely on text-based reasoning, thereby enhancing its performance on visual-related tasks. Our experiments demonstrate Image-DPO objective (Eq. 4) outperforms varies self- self-impune improve VLM baselines on ViLP
Inspired by our benchmark, propose Image DPO, self- improvement method for enhancing VLMss visual reason- ing, featuring new objective and data generation pipeline using VLMs themselves and pre-trained image models.
4.1. Objective
Existing approaches for VLM self-improvement follow the way used in DPO paper (Rafailov et al., 2024), where the model is trained to distinguish between good and bad an- swers for fixed image and question (Figure 2a right). How- ever, this straightforward adaptation may not the best for vision models. as the model sometimes distinguish good and bad answers from the text alone without needing to analyze the image. In contrast, propose Image DPO, fosuse- objective that creates good and bad question- image-answer pairs by corrupting the image while keeping the question and answer unchanged (Figure 2a left). An example our synthetic data illustrated in Figure 2b.
Proposition 1. Let CRLFT(TO, Neet' S) be the KL- constrained reward maximization objective used in the RL fine-tnning phase (Appendix Eq. 7), where the dataset S = {(Q, A, Tw, Ti)} contains good images I and cor- rupted images Н, Let LimageDPO (To, S) be the ob- jective from Eq. 4, which compares (Q, (Q, A) against (Q, JI, A). Then for any policy To and reference model Traf, we have
CRLFT (TP, Tyef; <
LimageDDO Trefi
Formally, given an image I question Q, and its cor- responding answer A, we generate a comrupted image I via image-editing operations, including Gaussian blur, pix- elation, or semantic modifications. The triplet (Q, A) forms degraded question-image-an
Proof Sketch. Following (Rafailov et al., 2024), we express the optimal KL-constrained policy in terms of latent re- ward function. Applying a Bradley-Terry preference model to irestion-image-anee
pair compared to (Q, Iw, A). We train the model to distinguish between good and bad triplets using the objective 4, where my the target
triplets (Q, Ju/ Tt, A) and using Jensen's inequality yields an upper bound whose minimiza- tion is equivalent to LP (TO, Reef; S). A full deriva- tion appears in Appendix A.
5

---

Probing Visual Language Priors in VLMs
Flamiigg
Tigor o
Panguin
Moscow
London
Pariss
AVi
Flamingos often stand on one leg. the image, what pink animal is standing on one leg in water?
The Red Square is in Moscow. From the image, which city is the Red Square located" answer with a single word or abbreviation, NYC.
GPT4o:
Flamingo
GPT4o: Flamingo
GPT4o: Penguin
GPT4o: Moscow
GPT4o: Moscow
GPT4o: Moscow
Flamingo
Claude: Flamingo
Claude: Flamingo
Moscow
Claude: Moscow
Parts
Gemini: Flamingo
Gemini: Flamingo
Gemini: Penguin
Gemini: Moscow
Gemini: Moscow
Gemini: Paris
LlamaV: Flamingo
LiamaV: Flamingo
LiamaV: Penguin
LlamaV: Moscow
LlamaV: Moscow
LlamaV: Moscow
Round
Hexagon
Dodacagon
Camel
Horse
Tiger
The moon is typically depicted as round. What shape does the moon appear to take image of the night sky?
Camels have humps. Which the image humps?
GPT4o: Hexagon
GPT4o: Hexagon
GPT4o: Polygonal
GPT4o: Camel
GPT4a: Camel
GPT4o: Camel
Claude: Hexagon
Claude: Hexagon
Claude: Octagonal
Claude: Camel
Claude: Camel
Claude: Camel
Gemini: Round
Gemini: Hexagon
Gemini: Octagonal
Gemini: Camel
Gemini: Camel
Gemini: Camel
LlamaV: Hexagon
LlamaV: Hexagon
LlamaV: Octagonal
LlamaV: Camel
LiamaV:
Figure 3: Qualitative examples. We show the results from GPT-4o, Claude-3.5- Gemini- 1.5-Pro, and Llama-3.2- Vision-90B for some challenging cases, Please refer Appendix B.6 for categorized failure analysis.
4.2. Data Generation
and their corresponding answers one QlAprior and two QIA for total of 900 QIAs. The QlAprior examples (300 in total) align with common language priors (i.e., they can usually be answered correctly by relying on textual cues alone). In contrast, the QlA examples (600 in total) challenge these priors by requiring visual reasoning.
Training VLMs demands large-scale ientttioniimae--ansswe (QIA) triplets, which are often scarce. To address this, we introduce scalable data generation pipeline (Appendix Fig- ure 16) that repurposes existing image datasets via VI.M themselves and image generative models. Given a seed im- age from COCO (Lin et al., 2014), Texi2VQA (Singh et al., 2019b), or Visual Genome (Krishna et al., 2017). VLMs are tasked with simultaneously selecting appropriate functions (e.g., image generation or editing models) and generating corresponding instructions. These instructions then used to produce new images, in addition the seed image, illustrated in Figure 17. The same VLMs are then employed to generate QA pairs for these newly created images. Fol- lowing, we apply the mentioned three types of image corrup- tions to the generated images, constructing good bad pars (Q. Iw. A) and (Q, A). Specifically, we employ Stable Diffusion XI. (Podell et al., 2023; Rombach et al., 2022) for image generation, and use Instruct-Pix221 2023a), and Grounded-S. et al., 2024) for image editing. Example generated data, prompts, and more details are included in Appendix C.
ViLP features two evaluation settings:
ViLPF, where both distractor facts and the ques- tions provided;
ViLP?, where only the questions themselves are given (i.e., no distractor facts).
We report two metrics in Table 2: average accuracy on OlAye (noted as Score) and average accuracy on QlAprior (noted as Prior). Our benchmark emphasizes the perfor- mance in Score.
(Brooks et al., (Rombach et al., 2022: Ren
Is the QIA easy for humans? We begin by evaluating our benchmark through a human study. Participants achieved nearly perfect accuracy ViLPF-Prior and over 98% on Vil.pf Score and Vil.pP- Score,,, confirming that our question- image-answer combinations are unambiguous for human interpretation. Notably, despite QLA being designed as out-of diitof-distribution examples, humans were still able to cor- rectly distinguish them.
5. Experiments
We introduce ViLP, new benchmark comprising 300 ques- tions. Each question is paired with three unique images
6

---

Probing Visual Language Priors in VLMs
Love
Pautan Lovel
Loval
Figure 4: Comparison of benchmark scores under different image transformations. Solid line and dotted line refer to ViLPP-Score and ViLP -Prior, respectively.
Table 2:
Benchmarking on ViLP. Please refer to the left text for symbol definitions. & indicates the model often fails to follow the instructions.
Nanwnll
Dolprin
DetogCo
Round
STOP
STOP
ViLP
ViLP
Model
Score
Prior
Score
Prior
Patlch TIOI mamiaa With long buskiss degicted in the imag??
tig Winee the #hape of the stop dian the image?
Baseline
OP14:: Narwhol
GPT4o: Duighin
OPf4c: Octagon
GPTAo: Sphene
LLek/i: Narwhel
LLaVA: Nanshal
LL Octegon
LLeVA: Octegen
Human
98.33
99.67
98.67
96.67
GPrder Narwital
Karwhal
Octagon
Dotegio
GPT-4o (text only)
0.0
92.33
0.17
71.33
Dulghin
LLaVA: Octegon
API call only
Figure 5: Qualitative results before and after removing distactor facts. GPT-4o and LLaVA-1.5-13B models yield completely opposite behaviors.
GPT-4o
66.17
91.00
56.00
87.67
GPT-4V
57.67
88.33
38.33
85.33
GPT-4o-Mini
57.67
89.00
46.67
84.67
Claude-3.5-
70.00
84.33
59.33
86.67
Are our QIAs aligned with the learned priors of VLMs? We tested GPT-4o (text only) on our questions (removing all image references). Despite no visual content, it correctly answered 92.33% on ViL.F-Priin The accuracy drops to 71.33% once distractor facts are removed, showing that these facts significantly guide the answer. For QlAjest GPT- 4o (text only) accuracy nearly falls to 0% (ViLP" -Score & ViLP?-Score' -core), indicating the QlAjest cannot be answered using text alone.
Claude-3-Opus
59.17
74.00
43,00
82.67
Claude-3-Sonne
48.83
83.67
40.33
81.33
Claude-3-Haise
43.67
82.67
34,83
82.33
-.5-ro-
60.50
79.33
48.00
83.00
Gemini-1.5-Flash
54.50
83.33
69.17
79.67
Open weights
Llama-3.
1.mma-3.2--al 2- 2-Visionn
67.33
76.67
61.17
79.33
Llama-3. 2- -0V
64.00
91.67
63.17
83.33
MolmoE- IB
48.67
57.33
47.83
69.00
Molmo-7B-4
57.83
60.67
47.33
76.33
How do VLMS perform our benchmark? Although our benchmark questions are distinguishable for humans, they are challenging for VLMs. Even the advanced VLM models like GPT-4o, have a clear performance gap (66.174 17% v.s. 98.33%) compared to humans' performance on ViLP*. Score, indicating the difficulty of these questions for VLMs. CJaude-3.5-Somen achieved the best score 70%, while most of the commercial VLMs are below 60%. Figure 3 high- lights sample outputs from top commercial and open-source models, including GPT-4o, Claude-3.5 3.5nnnnnet, Gemini- 1.5- Pro, and Llama-3.2 Vision-9OB. They face significant chal- lenges when addressing these cases in our ViLP, whereas humans can arrive correct answers after consideration. Notably, it encouraging see that some open-source models achieved over 60% accuracy on ViLPF-Score with performance nearing that of their commercial counterparts, including Llama-3.2 2-Vison and Molmo-721 72B. Additionally, we provide more detailed failure-case analysis in Appendix B.6, where we group the errors into six recurring modes (i) -hape-reco slips, (ii) counting mistakes, (iii) prior-
Molmo-7B-D
54.5
69.00
46.17
72.33
Molmo-72B
60.33
85.00
47.17
82.33
VI.-7-
50.50
83.00
48.67
80.33
Qwen2-VLL-7BB
56.50
92.33
53.83
83.00
VL2-SB
47.00
66.67
43.00
75.00
IntemVLL2-76B
42.67
47.67
50.84
74.33
LLaVA-I.5-7B
29.67
71.33
37.67
65.67
LLaVA-1.5-13B
35.33
81.00
41,50
73.67
Cambrian-1-8B'
8.67
43.67
32.50
63.67
LLaVA-OneVision->
54.17
82.33
49.67
75.00
LLaVA-One 722
1.67
3.00
5.22
11.67
Humans performed slightly better on ViLP" ViLPF-Prior when dis- tractor facts were provided, as they could easily identify that these facts aligned with the correct answers. Moreover, ViL.p-- PP--Score was marginally lower when facts were intro- duced, as the distractor facts added some noise and caused minor confusion, although the impact of this noise is rela- tively small. These findings are consistent with the design principles of our benchmark.
7

---

Probing Visual Language Priors in VLMs
Table 3: Effectiveness of Image-DPO on General VQA benchmarks.
VLMs
ViLP* Score
VilPP
Score
NBg
NB,
NBg
NBB
MM-Vet
CHAIR"
CHAIR'
LLaVA-L.5-7B
29.67
37.67
37.7
43.8
12.7
67.3
31.1
49.1
14.8
LLVVA-15-7BB+
Image-DPO
34.177 4.5
39.33 1:66
39.79 2009
45.471 167
14.16"
1:46
68.45 1:15
32.3 12
45"-1 -4
12.3
LLaVA-1.5-13B
35.33
41.5
39.6
44.6
14.8
68.9
36.1
48.3
14.1
LLaVA-1.5-13B Image-DPO
38.17 2.84
42.5"
42,68
308
47.37 3.70
17.16 2:36
70.36" 1:46
37.5
42.6
11.6
255
Cambrian-8h
8.67
32,5
44.6
47.9
19.4
71.5
51.4
14.5
4.7
Cambrian-80 8B Image-DPO
20.83 12:06
39.3
1.883
46.5
50.2
1:23
20
1.06
72 0.5
51.7
11.4
4.4
Table 4: Benchmark Performance with More Fine-Grained Categories. We extended our evaluation system to include three categories-Corre Wrong, and Invalid. "Invalid" responses (not single-word) are further split into Invalid-Cone Correct (noted as Invalid-v/) ) and Invalid-Wrong (noted as Invalid-x) based on GPT assessment.
ViLP
ViLP
Model
Score
Prior
Score
Prior
Correct
Wioog
Invaidd
Invald-x
Conect
Weong
Iovalid>
Invalid-x
Conect
Wrong
Invalid-
Inviid-x
Canect
Wrong
Iovalid<<'
Invald-xx
Cambrion-88
R.67%
28.83%
29.17%
53.33%
43.67%
2.35%
54,00%
6.07%
32,50%
37.50%
11,67%
18.33%
63.67%
6.35%
22.00%
6,00%
54.17%
42.89%
1.38%
1.67%
82.39%
15.33%
2,396
0.008
49.67%
47.00%
1,388
3,00%%
75.00 000
20.38
3.67%
1.0%%
ILaMA-DeeViinnn-
1.00%
43,338
54,00%
90.339
6.33%
5.22%
50,00%
11.67%
1.338
78.675
8.338
GPT-4
33.77%
0.00%
0.03%
91.00%
0.33%
56.00%
43.17%
0.50%
87.07%
11.33%
1,00%
0.00%
biased rejection of counterfactual scenes, (iv) proverb- driven hallucinations, (v) misclassification of creative con- cepts, and (vi) text-over-vision errors on blended images.
Gaussian blur, and pixelation, affect ViLP performance. The results, shown in Figure 4, reveal that the ViLP* -Score rapidly decreases as the severity of the transformations (x- axis) increases, while the ViLP' -Piior score remains around 50%. Interestingly. GPT-4o, when using degraded images, performs worse in ViL ViLPF-Prior than when no images are used, i... GPT-40 (text only) in Table 2.
Do distractor facts really distract? In ViLPF setting, we add a distractor fact before the question. Since these facts implicitly suggest incorrect answers for QlAiest, we expected this change to make the questions more suggestive and lower the ViLPF-Scorm as the distractors would mislead the VLMs. Surprisingly, GPT-4o benefits from distractor facts, improving accuracy on Q[Ajesa We hypothesize these facts highlight question focus, narrowing the search space. However, weaker models like LLaVA-1.5-13B (Liu et al., 2023b) often get misled by the distractors, hurting their Score but boosting Prior. For instance, as shown in Figure 5, with including distractor facts, LLa LLLVA-1.5--3BB consistently predicts the distractor fact the answer. However, once the distractors are removed, it can then predict correctly.
Comparison of ViLP with other VQA datasets: To highlight the distinctions between ViLP and existing benchmarks, including Winoground (Thrush et al., 2022), Whoops! (Bitton-Guetta et al., 2023a), and Hallusion- Bench (Guan et al., 2023), we conduct a comparative analy- sis of both their high-level design principles and lowel data formats. This comparison incorporates qualitative and quantitative insights as detailed in Appendix B.4.
FinireGrained Evaluation Currently, we explicitly ask VLMs to produce a single-word response for evaluation, and penalize responses that include long sentences, as we believe instruction following is a foundation capability of thus, failure to comply warrants a penalty. Al- though this evaluation protocol is cost-efficient without call- ing I.M.M. fast, and reliable, it may conflate semantic errors with instruction-following failures.
For bad instructionn-ollloinng models like Cambrian- 8B (Tong et al., 2024), distractor facts significantly hin- der adherence to explicit instructions, such as providing single-word answers. With facts, Cambrian-8B fails to fol- low instructions in 62% of cases, compared 30% with- out (a nearly 2x increase). Manual review shows 59% of these failures are contextually correct, yielding an ad- justed accuracy of 47.92%. Similarly, LLLVV--One Vision- 72B (Li et al., 2024b) often generates detailed analyses despite explicit single-word prompts. This trend highlights a concerning trend: focusing on improving performance on well-established benchmarks may come at the cost of basic instruction-follovii abilities, ultimately limiting the practical utility of these models in real-world applications.
To address this, we introduce an optional finer-grained eval- uation protocol with three categories: Correct, Wrong, and Invalid. Correct and Wrong denote semantically accurate or inaccurate answers that comply with the single-word con- straint. Invalid captures responses that violate the format. For these, we further offer an optional GPT-based classi- fication into Anvalid-Corred or Invalid- greater granularity at the cost of increased evaluation over- head. We show these evaluation results in Table 4.
providing
How image transformations affect the results? We also investigate how image transformations, including resizing,
8

---

Probing Visual Language Priors in VLMs
Commercial models such as GPT-4o follow instructions reli- ably. In contrast, LLaVA-OmeeVision-222 matches other
Table 5: Comparisons of Image-DPO on ViLP.
ViLP
ViLp
models in reasoning ability (Correct yet ignores the single-word instruction in most instances. Cambrian-8B similarly yields a significant number of In- valid cases. Counterintuitively, performs markedly worse at instruction following than its smaller counterpart, LLaVA- MaeV-eeWeVVe Vision-7B, and shows a stronger bias toward learned priors reflected in its lower ViLP Score and higher ViLP Prior. We hypothesize that instruction following offers an additional lens for probing prior reliance: models dominated by priors tend to gener- ate verbose, prior-driven answers (i.e. long responses) that disregard explicit single-word directives.
Model
Score
Prior
Score
Prior
LLaVA-1.5-7B
29.67
71.33
37.67
65.67
+HADPO (Zhao et al., 2023h)
33.00
74.33
38.50
65.00
+RIHH-V (Yu et al., 2023a)
29.50
75.00
36,33
65.33
+FOS (Yue al., 2024)
31.33
67.00
38.67
65.67
+SIMA (Wang et al., 2024b)
27.83
68.67
36.17
66.00
+V-DPO (Xie et al., 2024)
29.50
72.67
37.83
67.67
+Text-DPO
31.34
71.67
37.83
65.67
+Image-DPO
34.17
75.00
39.33
68.00
5.1. Image DPO
TextDPO on Corrupted Images. Does Image-DPO's im- provement stem from the objective itself, or is merely due to training on more perturbed data? To investigate this, we conduct an ablation where we train TexDPPO with cor- rupted images. Specifically, for each pair of (Q, I, Aw) and (Q, Ar) used in Text-DPO, we apply the same corruptions as Image- -PO to / to form (Q, 1", Aw) and (Q, At). Re- sults show that LLLVAA-v.55775 trained with this TextDPO variant achieves a ViLPP- -Score of 31, a ViLP? -Scoe of 37.5, and 30.3 on MMVET, In contrast, ImageDPO out- performs across all metrics, achieving 34.17, ViLP*- of 39.33, and 32.3 on MMVET.
In this section, we evaluate Image-DPO (Section 4.1) on both ViLP and general VQA benchmarks. As an ablation baseline, we introduce Text-DPO, which uses the same Question-Image-Anse
(QIA) generation process as we used in Image-DPO but applies LLM self-rewarding objec- tive (Yuan et al., 2024) (the standard DPO objective). In Text-DPO, good and bad pairs stem from VLM-generated positive and negative answers, while the question and image remain fixed. As shown in the right of Figure 2a, the green box depicts Image-DPO, generating corrupted images via semantic edits, Gaussian blur, and pixelation while keeping the question and answer constant; the purple box illustrates Text-DPO, which fixes the image and varies the answers with associated ratings. This setup parallels other VLM self-rewarding work (Zhou et al., 2024a; Deng et al., 2024; Zhou et al., 2024b; Wang et al., 2024c;a).
ViLP"-Score of
Impact of Synthetic Image Augmentation. When we remove the image-generation stage and fine-tune ImageDPO only on QIA pairs drawn from the seed datasets (COCO, VG, Tex12VQA), performance slips across every metric: ViL.P'Score declines from 34.17 to 31.17 and ViL.P' from 39.33 to 38.00; the four novel-bias scores (NBq, NB, NBg, NBg) each lose roughly one point; MM-V Vet drops from 32.3 to 31.4; and hallucination rates rise (CHAIR
For baselines, we compare with VLM self self-improvemmnt methods, including SIMA (Wang al., 2024c), HADPO (Zhao et al. 2023a), and EOS (Yue et al., 2024), by using their publicly available checkpoints. Additionally, we train models using the dataset and code provided in RLHF-V (Yu et al., 2023a) and V-DPO (Xie et al., 2024). All models use LLaVA-7B for a compre- hensive comparisons as many paper only release 7B checkpoints. Table 5 shows that Image-DPO achieves the highest across all the metries.
45 + 47, CHAIR J: 12.3 - 13.6). These uniform declines confirm that synthetic images by image generative models broaden the visual distribution, inject the priors of powerful generative models, and are crucial for achieving the full gains of IPageDPO.
6. Conclusion
Besides, we evaluate the proposed Image-DPO algorithm across three VLM models Caaall--mmmiia---e 8BB ALaVA-1.5-7B. and -LL 1.5-i13BBusingg several popular VLM bench- marks that focus on different aspects, including composi- tionality & biases (NaturalBench (Li et al., 2024a)), general visual reasoning (MM-Vet (Yu et al., 2023c)), and hallu- cinations (CHAIR (Rohrbach et al., 2018)). The results, presented in Table 3, show consistent performance improve- ments across both datasets and models, further demonstrat- ing the effectiveness of our Image DPO method.
In conclusion, we present the ViLP benchmark to probe the challenge of visual language bias in Vision-Language Models (VLMs). By utilizing advanced image generation models and designing questions that demand visual cues for accurate responses, our benchmark includes images that defy language priors, revealing the limitations of current VL.Ss. Our method, Image-DPO, which incorporates self- generated VQA pairs and image corruption for training, has demonstrated promising improvements in enhancing visual reliance, as evidenced by performance gains on open-source VLMs, such as V..V and Cambrian.

---

Probing Visual Language Priors in VLMs
Acknowledgment
shot learning. ArXiv, abs/2204.14198, 2022. URL https:
/ seapi..smantccccholaar wwwpo/aaarrrrrrooooo www.ooog//ooor Cttpg//rrpsSD::::: 248476411.
This work was supported by the LG AI Research grant. We also extend our gratitude the OpenAI Researcher Ac- cess Program for providing credits used to access OpenAl's APIs. Additionally, we appreciate Chris Rockwell and Ji- aming Yang for their contributions to our human evaluation studies, and thank Yongyi Yang for discussions regarding the proposition proof.
Bitton-Guetta, N., Bitton, Y., Hessel, J., Schmidt, L., Elovici, Y. Stanovsky, G., and Schwartz, R. Break- ing common sense: Whoops! vision-and- lan-uuuge benchmark of synthetic and compositional images. 2023 IEEE/CVF International Conference on Computer Vi- sion (ICCV), pp. 2616-2627,, 2023a. URL https: semanticscholar. http://rwrrrgr/oossst:::22//// 2574967744
Impact Statement
Bitton-Guetta, N., Bitton, Y., Hessel,, Schmidt, L.,, Elovici, Y., Stanovsky, G., and Schwartz, R. Breaking common sense: Whoops! vision-and- language benchmark of synthetic and compositional images. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2616-2627, 2023b.
This work aims advance the field of Vision-Language Models (VLMs) by diagnosing and mitigating their reliance on learned visual language priors. By introducing the ViLP benchmark and Image-DPO training framework, our contri- butions promote more robust and interpretable visual rea- soning in AI systems. While ViLP dataset involves syn- thetic image generation, we have rigorous human review to avoid generating harmful deceptive content. Our findings may improve AI alignment with human perception, benefit- ing applications in accessibility, education, and Al-assisted becision-making
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023a.
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning follow image editing instructions. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023b.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint W:Xiv:2333.08774 2023.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Ka- plan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I. and Amodei, D. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020. URL https: /lapi.semanticcchoo//ooooo wwt.oogg///ooo 2000ppuuID:22897170000
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., and Batra, D. Vqa: Visual question answer- ing. International Journal of Computer Vision, 123:4- 31, 2015. URL https: /tpp. semanticscholar, org/ CorpusID: 31 3180429.
Caron, M., Touvron, H., Misra, I., J'egou, H., Mairal, J., Bo- janowski, P., and Joulin, A. Emerging properties in self- supervised vision transformers. 2021 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 9630- 9640, 2021. URL https: / //ppippppp.../ www.ntccscooll........ org/CorpusID:233 2334444273334
Agrawal, A., Batra, D., and Parikh, D. Analyzing the behav- ior of visual question answering models. arXiv preprint arXiv: 2016.
Internaa
Agrawal, A., Batra, D., Parikh, D., and Kembhavi, A. Don't just assume; look and answer: Overcoming priors for visual question answering. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4971- 4980, 2017. URL https: wer.iisemmatiiccholl..o org/CorpusID: 19298149 49.
Chen, L.,, Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12993, 2023.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, Has- son, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong. Z.,, Samangooci, S., Monteiro, M., Menick, J.. Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M.. Barreira, R., Vinyals, O., Zisserman, A., and Si- monyan, K. Flamingo: a visual language model for few-
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A. J., Padlewski, P., Salz, D. M., Goodman, S., Gryener, A., Mustafa, B., Beyer, I.., Kolesnikov, A., Puigcerver, J., Ding, N., Rong, K., Akbari, H., Mishra, G., Xue, L., Thapliyal, A. V., Bradbury, J., Kuo, W., Seyedhos- seini, M., Jia, C., Ayan, B. K., Riquelme, C., Steiner, A., Angelova, A., Zhai, X., Houlsby, N., and Soricut,
10

---

Probing Visual Language Priors in VLMs
R. Pali: A jointly-scaled multilingual language-image model. ArXiv, abs/2209.06694,, 2022. URL https: //api. sewanticschoolieeee wttp://co...rooooooooousoo222222
Sun, X., and Ji, R. Mme: A comprehensive evalu- ation benchmark for multimodal large language mod- els. ArXiv, abs/2306. 2023. URL / semanticscholar http:/aarr.rr/oopuostD/S 259243928.
Chen, Z., Wu, J., Wang, W. Su, W., Chen, G., Xing. S., Zhong, M., Zhang, Q.. Zhu, X., Lu, L., et al. Internvl: Scaling vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 24185-24198, 2024.
Gouthaman, K. and Mittal, A. Reducing language biases in visual question answering with visually-grounde ques- tion encoder. ArXiv, abs/2007.06 2020. URL https: //api. www.nticscoooaa..ooooom 2 220496567.
Goyal, Y., Khot, T., Summers-Stay D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. International Journal of Computer Vision, 127:398 414, 2016. URL /tt.tttppp/ppp...mmmiiiiicchhooo semanticscholar.... org/ CorpusID: 00818081220 124.
Chiang, W.-L.., Li, Z.., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang. S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL htttss://mmss..orgg 2023-03-23 11--300-vviim vicina/
Goyal, Y., Khot, T., Summers-Stay. D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 6913, 2017.
Dai, w., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B. A., Fung, P., and Hoi, S. C. H. Instruct- blip: Towards general-purpose vision-language models with instruction tuning. ArXiv, abs/2305.06500, 2023. URL http:/ppp..ssmmtiicccccccccccoee wwapcchollrroo/Corruuuse org//CrppssD: 25861 5266.
Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., and Zhou, T. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illu- sion in large vision-language models. 2024 IEEE/CVE Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 14375-14385, 2023. URL https: //api. swwanticscholar www.oorg//oooo.. wwwpg//CrpusID:::: orgg/CorpusID:255491111 265499116.
Dai, W., Lee. N.. Wang, B., Yang. Z., Liu, Z., Barker, J., Rintamaki, T., Shoeybi, M., Catanzaro, B., and Ping, W. Nvim: Open frontier-class multimodal Ilms. arXiv preprint arXiv:2409. 14402, 2024.
Dancette, C., Cadène, R., Teney, D., and Cord, M. Beyond question-based biases: Assessing multimodal shorteut learning in visual question answering. 2021 IEEF/CVF International Conference Computer Vi- sion (ICCV), pp. 1554-1563, 2021. URL https: /ttps://api... semanticscholar.
Gurari, D., Li, Q., Stangl, A., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P. Vizwiz grand challenge: Answering visual questions from blind people. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 3608-3608111 2018. URL https: //api. semanticscholar http:/cccooaarrrreerrrroosooooo 3831582.
233168994.
Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J. S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, L., et al. Molmo and pixmo: Open weights and open data for state-of-the art multimodal models. arXiv preprint arXiv:2409. 17 1146, 2024.
Hsich, C.-Y., Zhang, J., Ma. Z., Kembhavi, A., and Krishna, R. Sugarcrepe: Fixing hackable benchmarks for vision- language compositionalityy ArXiv, fiss/200..16111 2023. URL 259251493.
Deng, Y., Lu, P., Yin, F., Hu, Z., Shen, S., Zou, J., Chang, K.-W., and Wang, W. Enhaneing large vision language models with self-training image comprehension. arXiv preprint ar 4rXiv:2405.19716, 2024.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional ques- tion answering. 2019 IEEF/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 6693 6702, 2019. URL https: / sep..ssmanticccoolarr Org/CorpusID ID: :e:pDuuIDD:12222220000 152282269.
Desai, K. and Johnson, J. Virtex: Learning visual repre- sentations from textual annotations. 2021 IEEF/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 11157-11168, 2020. URL https: //api. semanticscholar http:iiccccooooooooorrorrrlsslla org//orrps1D::215736.8
Jaques, N., Gu, S., Bahdanau, D., Hemmández-Lobato, J. M., Turner, R. E., and Eck, D. Sequence tutor: Conserva- tive fine-tuning of sequence generation models with ki- control. In International Conference on Machine Learn ing, pp. 1645-1654 PMLR, 2017.
Fu. C. Chen, P. Shen, Y., Qin, Y., Zhang, M., Lin, X. Qiu, Z., Lin, W., Yang, J., Zheng, X., Li, K.,

---

Probing Visual Language Priors in VLMs
Jaques, N., Shen, J. H., Ghandeharioun, A., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, R. Human- centric dialog training via offline reinforcement learning. arXiv preprint arXiv:2010005888,, 2020.
for vision and language. ArXiv, abs/1908 2019. URL wwwhhhtp////aae 199528533.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and rong Wen, J. Evaluating object hallucination in large vision- language models. In Conference on Empirical Methods in Natural Language Processing, 2023c. URL https: //api. wtt.:seemaaiischhiee wwweaaar.rrr/oooo wwwp/oorpssi::55555.... 258740697.
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y.-H., Li, Z., and Duerig. T. Scaling up visual and vision-language representation learning with noisy text supervision. ArXiv, abs/2102.059) 2021. URL https://apt semanti oocttaao..o..ooo///rupsssS. 231879586.
Li, Y., Zhang, Y., Wang, C., Zhong, Z., Chen, Y., Chu, R., Liu, S., and Jia, J. Mini-gemini: Mining the vision language models.
Johnson, J., Hariharan, B., van der Maaten, L.,, Fei-Fei, Zitnick, C. L., and Girshick, R. B. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. 2017 IEEE Conference
potential of multi-modalee ArXiv, abs/2403. 18814, 2024c. URL https://api. semanticscholar. www.aaarrrrgcCooooo//.. 268724012.
Computer Vision and Pattern Recognition (CVPR), pp. 1988-1997, 2016. URL https: /api. semanti iisaannccccllarr hataaaiiicccca..ooo/////rrpllll org/CorppsID: 15458100
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCY 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Kim, W., Son, B., and Kim, I. Vilt: Vision-and-language transformer without convolution or region supervision.
In International Conference on Machine Learning, 2021. URL httpps:/aapp semanticscholatt http:p//.eeaaaiiccccooaarrrrrrrr org/CCorssID: 231839613.
Liu, H., Li, C., Li, Y., and Lee, Y. Improved baselines with visual instruction tuning. ArXiv, abs/2310.03744 2023a. URL https: http:/appppi..a www.stesccccccolaaa.... .rr.olllrrorrrrogoo wttp//opp//oroo/// 263672858.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J,, Shamma, D. A., et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In- ternational journal of computer vision, 123:32-73, 2017.
Liu, H., Li, C.,, Li, Y., and Lee, J. Improved base- lines with visual instruction tuning. arXiv preprint 20.244
Liu, H., Li, C., Wu, Q. and Lee, Y. J. Visual instruction tuning. ArXiv, abs//230400448, 2023c. URL https:
Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y. and Shan, Y. Seed-bench: Benchmarking multimodal iims with gen- erative comprehension. ArXiv, abs/2307.161 16125, 2023a. URL https: //api. www.eiseeaanchcllle.ooo www.s/ccchooaaaororroroossseoeee org/CorpsII:: 260334888.
//api. www.nticscoolarrrreeeeeeeee http:a/roooo/ooooooeeeeeee 258111::25879777 258179774,
Liu, H., Li, C., Wu, Q. and Lee, J. Visual instruction tun- ing. Advances neural information processing systems, 36, 2024a.
Li, B., Lin, Z., Peng, W. Nyandwi, J. D., Jiang, D., Ma, Z., Khanuja, S., Krishna, R., Neubig, G., and Ra- manan, D. Naturalbench: Evaluating vision-language models on natural adversarial samples. arXiv preprint arXiv:2410. 14669, 2024a.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z.., Chen, K., and Lin, D. Mmbench: Is your multi-modal model an all-around player? ArXiv, abs/2307.0628/ 2023d. URL https: //api sematticscholar /org///oosSID:::: 2 259837088.
Li, B., Zhang, Y., Guo, D., Zhang, R. Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z.,, and Li, C. Llava- onevision: Easy visual task transfer. arXiv preprint AXXV:24008.033266 2024b.
Liu, Z., Zang, Y., Dong, X., Zhang, P., Cao, Y., Duan, H., He, C., Xiong, Y., Lin, D., and Wang, Mia- dpo: Multi-image augmented direct preference optimiza- tion for large vision language models. arXiv preprint 2410 7637, 2024b.
Li, J., Li, D., Savarese, S., and Hoi, S. C. H. Blip-2: Boot- strapping language-image pre- pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 2023b. URL https: Seapi.semanticsse http.//mmattiicccllaa.a..rrrrrrs
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for sci- ence question answering. ArXiv, (bbs/2209.09511. 2022. URL httph::////t/ //api.pp...../ wwpii.semantccscho.... 252383606.
Li, L. H., Yatskar, M., Yin, D., Hsich, C.-J., and Chang, K.WW. Visualbert: A simple and performant baseline
12

---

Probing Visual Language Priors in VLMs
Lu, P., Bansal, H., Xia, T., Liu, J., yue Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating math reasoning in visual con- texts with gpt-4v, bard, and other large multimodal models. ArXiv, abs/2310.02255, 2023. URL https: //api. semanticscholaat .aweeorrrrooooo ott.://o......oo//oooooo2222235o 26769996..
Prabhu, V., Yenamandra, S., Chattopadhyay, P., and Hoff- man, J. Lance: Stress-testing visual models by generating language-guider counterfactual images. Advances in Neu- ral Information Processing Systems, 36:25165-255 251844,, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Ma, Z., Hong, J., Gul, M. O., Gandhi, M., Gao, I., and Krishna, R. @ crepe: Can vision-language founda- tion models reason compositionally? 2023 IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 10910-10921, 2022. URL https: //api. wwwpi/sssaanncscoola....o. http://ccccooooooooorrrrrooooaaa otrg///rrrusS:255555555111 254685851.
URL https://ppp..mmmattcccoooolla 231591445 445.
Majumdar, A., Ajay, A., Zhang, X., Putta, P., Yenamandra, S., Henaff, M., Silwal, S., Mevay. P., Maksymets, O., Arnaud, S., Yadav, K., Li, Q. Newman, B., Sharma, M., Berges, V.-P., Zhang, S., Agrawal, P., Bisk, Y., Batra, D., Kalakrishnan, M., Meier, F., Paxton, C., Sax, A., and Rajeswaran, A. Openeqa: Embodied question answer- ing in the era of foundation models. 2024 IEEE/CVF Conference Computer Vision and Pattern Recogni- tion (CVPR), pp. 16488-16498, 2024. URL https: //api. www.nticscnnchllrr..oooooo www.eorg///ooo.oo prgg/oorpsII::26350 268066655.
Rafailov, R.,, Sharma, A., Mitchell, E.,, Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Ramakrishnan, S., Agrawal, A., and Lee, S. Overcoming language priors in visual question answering with adver- sarial regularization. In Neural Information Processing Systems, 2018. URL https: / /appi. semanticscholar. org/CorpusID: 52946763.
Masry, A., Long, D. X., Tan, J. Q. Joty, S. R., and Hoque E. Chartqa: A benchmark for question an- swering about charts with visual and logical reason- ing. ArXiv, abs/2203.10244 2022. URL https://api. semanticscholar.c wttp://oopuII::44593777
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In International conference on ma- chine learning, pp. 8821-8831, Pmlr, 2021.
Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F, Yang, J., Li, H., Jiang, Q., and Zhang, L. Grounded sam Assembling open-world models for diverse visual tasks, 2024.
Mathew, M., Karatzas, D., Manmatha, R., and Jawahar, C. V. Docvqa: A dataset for vqa document images. 2021 IEEE Winter Conference on Applications of Computer Vision ( (WACV), pp. 2199-2208, 2020. URL https:// sei.st
org. :ttg//orrpsID::22000 220280200.
Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T. and Saenko, K. Object hallucination in image cap- tioning. In Conference on Empirical Methods in Nat- ural Language Processing, 2018. URL //aps.:/api.... semanticscholar org/CorpusID: httpa//sssscooooorrro//pssssssss 52176506.
Mathew, M., Bagal, V., Tito, R. P., Karatzas, D., Valveny. E., and Jawahar, C. Infographicvqa. 2022 IEEE/CVF
Winter Conference on Applications of Computer Vision (WACV), pp. 2582-2591, 2021. URL https://api. semanticscholar orgg/CorpuSD::23394122
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
OpenAL. Gpt-4 technical report. 2023a. URL https:
OpenAl. Gpt-4 technical report. arXiv, 2023b.
Shen, S. Li, L. H., Tan. H., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., and Keutzer, K. How much can clip benefit vision-and-language tasks? ArXiv, abs/2107.06383, 2021. URL https://api. semanticscholar wtwpehhoarr..rr///ousSDD:::::::. 235829401.
OpenAl. Gpt-4o: A multimodal, multilingual generative pre-trained transformer. https://openaa wwm.index// Ballo-gpt-4o// 2024. Accessed: 2024-06-03.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for resolutin image synthesis. arXiv preprint rrXiv2237.001922 2023.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Ba- tra, D., Parikh, D., and Rohrbach, M. Towards vqa mod- els that can read. 2019 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 8309-
13

---

Probing Visual Language Priors in VLMs
8318, 2019a. URL https: http:/pp..ssmaniicschhooollr agg/CorpuSID:8553666
Azhar, F,, Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. ArXiv, abs/2302. 13971, 2023a. URL https: wttp://rr///oosot/// 257219404. 2572199444
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317- 2019b.
Touvron, H.,, Martin, L.,, Stone, K. R., Albert, P., Alma- hairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D. M., Blecher. L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V.. Goyal, N., Hartshorn, A. S., Hosseini, S., Hou, R., Inan, H., Kar- das, M., Kerkez, V., Khabsa, M., Kloumann, M., Ko- renev, A. V., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J.. Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poul- ton, A., Reizenstein, J.. Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307 09288, 2023b. URL https: //api wwmanticscoollareeoeeeeoeec hrrprriiiicccooaaoorrrrrrrrrre wtt.://////ppus1022255999999 259950998.
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W.. Rohrbach, M., and Kiela, D. Flava: A foundational language and vision alignment model. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15617-15629, 2021. URL https://apt semanticscholar org//orppu1D::24955255
Soldaini, L.,, Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A. H., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Ravichander, A., Richardson, K., Shen, Z., Strubell, E., Subramani, N., Tafjord, O.. Walsh, P., Zettlemoyer, L., Smith, N. A., Hajishirzi, H., Beltagy, I., Groeneveld, D., Dodge, J., and Lo, K. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint, 2024. URL https:ttp////// http://axxvvvvo//o
Wang, F., Zhou, W., Huang, J. Y., Xu, N., Zhang, S., Poon, H., and Chen, M. mdpo: Conditional preference opti- mization for multimodal large language models. arXiv preprint arXiv:2406. 1839, 2024a.
00159.
Sterz. H., Pfeiffer, J., and Vuli'c, I. Dare: Diverse visual question answering with robustness evaluation. 2024. URL https: //ap::://ap.... www.seissatcccooooooeoooooeo hatpattttccccoaaooor/rrrrssssss org/CorppsID: 272910642.
Wang. W., Bao, H., Dong, L.,, and Wei, F. Vlmo: Unified vision-language pre-training with mixture-of-modality experts. ArXiv, abs/2111.02333 2021a. URL https: seapiiismantcsccoolar wtwp/oopprII/:: SSSoorpuSI:2410355500 241035439.
Tan, H. H. and Bansal, M. Lxmert: Learning cross moos-mod encoder representations from transformers. In Confer- ence on Empirical Methods in Natural Language Process- ing, 2019. URL https: semamniccmtci /orgg CorpusID: 2011 103729.
Wang. X., Chen, J., Wang. Z., Zhou, Y., Zhou, Y., Yao, H., Zhou, T., Goldstein, T., Bhatia, P., Huang, F,, and Xiao, C. Enhancing visual-language modality alignment in large vision language models via self-improvement. ArXiv, 159733 2024b. URL https://apt: wttp://cccoooo.ooor/o/oooro////0
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. /805, 2023.
Wang. X., Chen, J., Wang. Z., Zhou, Y., Zhou, Y., Yao, H., Zhou, T., Goldstein, T., Bhatia, P., Huang, F, et al. Enhancing visual- modality alignment in large vision language models via self-improvemmnn. arXiv preprint arXiv:2405. 15973, 2024c.
Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., and Ross, C. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5238-5244 2022.
Wang. Z., Yu, J.. Yu, A. W., Dai, Z., Tsvetkov, Y., and Cao, Y. Simvlm: Simple visual language model pretraining with weak supervision. ArXiv, abs/2108.11904 202 URL https: / //ppipappp... 237291550.
Tong, S.. Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., al.
http:/ppp.smaaiccssiiieee www.seeettaenooiieeeeee. org orgg//rpss//
Cambrian-1: 1: A fully open, vision-centric exploration of
multimodal lms. arXiv preprint arXiv:2406. 16860, 2024.
Wu, Y., Zhao, Y., Zhao, S., Zhang, Y., Yuan, X., Zhao G., and Jiang, N. Overcoming language priors in visual question answering via distinguishing superfi- cially similar instances. In International Conference on
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E.,
14

---

Probing Visual Language Priors in VLMs
Computational Linguistics, 2022. URL https://api. semanticscholar. wttp://oorrossSD222257777777
Zhao, Z.,, Wang, B., Ouyang, L., wen Dong, X., Wang, J., and He, C. Beyond hallucinations: Enhancing Ivims through balluciination-wware direct preference optimiza- tion. ArXiv, abx/2311.16839, 2023b. URL https: //api. seapiissmantcccholaar .owaaa...rrggreee wwtpecchoooaa.oooooouusuueeeeeee :otg://oopSI::253555 265466428.
Xiao, W., Huang, Z., Gan, L., He, W., Li, H., Yu, Z., Jiang, H., Wu, F., and Zhu, L. Detecting and mitigating halluci- nation in large vision language models via fine-grained ai feedback. arXiv preprint 240.::444.1,4,33, 2024.
Zhou, Y., Cui, C., Rafailov, R., Finn, C., and Yao, H. Align- ing modalities in vision large language models via pref- erence fine-tuning. arXiv preprint arXiv:2402.111 2024a.
Xie, Y., Li, G., Xu, X., and Kan, M.-Y. V-dpo: Mitigating hallucination in large vision language models via vision- guided direct preference optimization. arXiv preprint arXiv:2411.02711 2024.
Zhou, Y., Fan, Z., Cheng, D., Yang, S., Chen, Z., Cui, C., Wang, X., Li, Y., Zhang, L., and Yao, H. Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405. 2024b.
Yu, T., Yao, Y. Zhang, H., He, T., Han, Y. Cui, G., Hu, J., Liu, 7.,, Zheng, H.-T., Sun, M., and Chua, T.-S. Rihf-v: Towards trustworthy milms via behavior align- ment from fine-grained correctional human feedback.
Zhu, K., Zhao, L., Ge, Z., and Zhang, X. Self-supervised visual preference alignment. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 291- 300, 2024a.
2024 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 13807-13816, 2023a. URL https: / /pp.. wwwasti semanticscholar www.ccchhhhhooooooooooooo. org/CorpusID: 265608723.
Zhu, L., D., Chen, T, Xu, P., Ye, J., and Liu, J. lbd: Alleviating hallucinations in large vision-language models via image-biased decoding. arXiv preprint 240.22402.14766 2024b.
Yu, T., Zhang, H., Yao, Y., Dang, Y., Chen, D., Lu, X., Cui, G., He, T., Liu, Z., Chua, T.-S., et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.117 17220,, 2024.
Yu, W., Yang, Z., Li, L., Wang. J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multimodal mod- els for integrated capabilities. ArXiv, abs/2308.02490, 2023b. URL https: / /ttpp/appppaa//a semammicccchichhoea.o.. org/ 277p1115:20661157
Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multi- modal models for integrated capabilities. arXiv preprint arXiv:2308. 02490, 2023c.
Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, Self-rewarding language models. arXiv preprint arXiv:4401.10020 2024.
Yue, Z., Zhang, L., and Jin, Q Less more: Mitigating mul- timodal hallucination from an eos decision perspective. arXiv preprint ar 2xXv:2240.145555 2024.
Zhang, P., Goyal, Y., Summers-Stay, D., Batra, D., and Parikh, D. Yin and yang: Balancing and answering binary visual questions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5014- 5022, 2016.
Zhao, Z., Wang, B., Ouyang, L., Dong, X., Wang, J., and He, C. Beyond hallucinations: Enhancing lylms through hallucination aware direct preference optimization. arXiv preprint Xrvi::231 16839, 2023a.
15

---

Probing Visual Language Priors in VLMs
A
QeWhhat is the color scheme the train ?
Image-DPO
lw
The
train
has
blue
and
yellow
color
Qc/What the color scheme the
1
The
train
has
a
blue and yellow color
Aw
w
Q:Whaat is the color schome the train 2
Text- Text-DPO DPO
The
train
has
blue and
yellow
color
A
Q:WWait the
color scheme
The
train has
a pink
and
red
color
Figure 6: Gradients difference between Image-DPO and Text-DPO. For Text-DPO (Rafailov et al., 2024), the model receives positive gradients (green arrows) for the preferred answer Aw and negative gradients A₁ (red arrows) for the dispreferred answer. In contrast, our proposed Image-DPO approach applies positive gradients when the preferred image T input and negative gradients for the dispreferred image 4, both based on the same output answer.
A. Image-DPO Mathematical Details
In this section, we give the complete proof of Image DPO. Simlarly to DPO, we start from the RL fine-tuning objective and then derive its variant for image dpo.
A.1. RL Fine-tuning for VLM
SFT Given question Q, answer A and image I, we can train a SFT model π SPT with supervised learning on high-quality data. As the SFT model is a language generation model, it still model modeling the text outputs with question and image.
SPT 1)
Reward Modeling Phase In this stage, we construct static dataset of comparisons S {A', Q', Li 1i), and we present the QIA pairs (Q, Tw A), (Q, It, A) to human for preference.
Following the idea of RL fine-tuning, the preference assumed to be obtained from latent reward function r (0, which are not tractable, and we we use BT model to represent the preference distribution p* as:
exp (r* (Q, A)) exp(r* Iw A) + exp(r* (Q, It, A))
p" p'((Q, A) (Q, 11,
Now given the human labeled preference, can try optimize a reward model Tá to estimate by using maximum likelihood. Framing this binary classification, have this negative log-likelihood loss:
CR(T,, S) = -ELA.Q. (log (roo A) T (Q,
(6)
Here is logistic function. Basically, this reward function gives score jointly considering image, question and image quality.
RL Fine-Tuning Phrase
During the RL phase, the learned reward function is used to provide feeback the VLM model. Following DPO
16

---

Probing Visual Language Priors in VLMs
paper (Rafailov al., 2024), the optimization is formulated
max (. [ra(Q. 1, 3Ki.. (To (ee (A|Q, 1),
where 3 is parameter controlling the deviation from the base reference policy Tiefs namely the initial SFT model T Due to the discrete nature of language generation, this object is also not differentiable and is typically optimized with reinforcement learning (Jaques et al., 2017; 2020).
A.2. Image DPO and RL Fine-Tuning
According to the DPO paper, straightforward optimal solution to the KL-constrainted reward function maximization object in Eq. 6 is:
(A(Q,11 Z(Q,1) T I(Q,1) (Al. (00
(8)
where Z(Q,7) 7) = EA (1Q, 1) exp( (Q, T, is a partition function. Here r should be any reward function, which makes hard to tract. We provide the proof this step in A.3.
Taking the logarithm both side, and with some algebra, we get
r(, (Q A) = Tr (A + Bog Z(Q,1)
(9)
(A
This parametrization could be applied to ground-truth reward r" and the corresponding optimal model n²
The BT model with the optimal policy is
exp (r" Lwr A)) exp (x A)) + exp (r,(
p' (QQ) 1, A) (2,
(10)
We plug Eq. 9 into the BT model, have:
exp log (A(Q.I))
(A(A))) + -Blog Z(Q, Iw)
,) A2)
/m 3 log nad (A((7 + Bog Z(Q,
Blog (1((Q..)) and A((.)) 3.l Z(Q. Ii)
1
1 + (xp(3 3 log r' ('((LQQQ) - 3 log r' T'Allwu:Q) An((Al
+ 8 log Z(I, Q) - 8 log Z(I Q))
क I1, Q) Boog x (A[ Q)
= exp((Blo
Teer [ ww Q) + 3 log 2(4 Q) - 3 log Z (( Q))
गei (A]4, Q)
Now we have the probability human preference data in terms of the optimal policy rather than the reward model, we can formulate maximum likelihood objective for policy my. Our policy objective is
[(To; = B(q.AA.a.. | logo (Blg
To Q) Bloog Tre( Q) Ta (A Q)
To (A Q)
+ Bog Z(I.. Q) - log Z(1, Q)
(11)
As f(x) = -log (2) is a convex function (a is the sigmoid function), we can apply Jensen's inequality f(tr + 44) s ff(z) fy))
no (A) Q) Te (Al Q)
(A Q)
[(To: < E
log a 28g
2,3 log
log (2 log Z(Iu, Q) 2,8 log Z(4, Q1)
Tiet (A Q)
(12)
17

---

Probing Visual Language Priors in VLMs
As log (2 Q)) is not function of Th. the above objective is equivalent the below 1q,13, where 23. the same objective listed in Eq.4 of the main paper.
no (w, Teat ( Q, (w) Ter Q,
To(A Q. Ii)
[ Tiet) S -.i
logo
(13)
In this sense, our optimization objective Eq.4 in main paper are optimizing the upper bound on the RL fine-tuning objective, i.e., Eq.7.
A.3. Deriving the Optimum of the Kl.-Constrained Reward Maximization Ohjective
In this appendix, we will derive Eq.8. Similarly to Eq.7, we optimize the following objective:
max B( (Q,, -000)-33000 3DKL.
(14)
under any reward function rQQ,1, reference model Tree. and general non-parametric policy class, We have:
max (rt.:arrr((0111. [r(Q.1,A) 3D (=|QQ, (e
I)
-max -..ns
rQQ, (,)) log
Tieel (A(Q,1 1)
min -
log
T(AlQ,1)
1
(Q)
(15)
Trae (A(Q,)
T(A|Q.1)
min BiQ.I)SSAAAAAAQ
log
log Z(Q,T)
Z(Q.I) #eed (A|Q, exp (2)(
where we have the partition function:
Z(Q,1) = Σ Tret (A(Q.))) (2)(0)1.4))
(16)
Observe that the partition function depends solely on (Q,I) and the reference policy Teef, and is independent of the policy π. We can now define the Equation
1 Z(Q,T) Toat (AIQ, I) exp (1).
T" (A(Q,1)
(17)
18

---

Probing Visual Language Priors in VLMs
B. More details and comparisons of our benchmarks
B.I. More data samples of ViLP
OUT
Light
Zebra
Whole
Seol
Modern drones lypically have four propellers, How many propeliers does the drone the picture have?
Zebra has black stripes. Which animal with black stripes shown the image?
1000
Round
Diamond
1KM
KM 200M
1 km Squats 100M
1000M
iolometer equals 1000 How kilometter, as indicated the image?
What of apple shown in
Image?
Round
Hesagon
Octagon
Two
Onn
Fiski
typically What image?
Bike usually have two wheels, How many wheels does the bike the conter the picture have?
Koela
Wombat
Soll
Cloud
Desert
jumpinng Which animal shown Image damonstratingg jump??
Based this orange
trea growling?
Smontle
Rome
Parts
Horan
2
Rabbit
Seal
Ihe Space Noedle situslted in Scatti, babed on the Image, What City io Horse oftom usodd for rainng Which anumai the image niddon by the Space Needle in? Respond with word abbrevistion, like NYC. humens?
Paper
Limbrollas usually mode the umbrella in the image?
could produce milic image produces milk for dairy products?
Person
Car
Sunglassee
Blander
Wallet
CAI
32
People usunlly the floor. Who map clean the floor in the picture?
Sunglasses protect eyes from What accessory image shields eyea from rays?
Figure 7: Randomly sampled data from ViLP.
19

---

What the of Probing Visual Language Priors in VLMs
Lion
Elephant
PIB
Map
Pizza
Pincopple
The lion is known as the king of the sevarnaah Can you identify the largest aniimal the image wearing crown?
mapp shows aceans without water and lands without buildings. Which object heve these cheracteristics!
Oowan
Tailet
Sink
Worm
Cint
Book
Snips typically oceen, Besed on the imags, what the ship selling
common piirese "the bird catches hhe the worm", What does the early
image?
Selomogiaph
Blander
Umbrella
Gavel
DAil
Wirench
seismograph is used to measure earthquake intensity. Which instrument in the picture records ground vibrations?
gavell used judges image, which ooject symbolizes judicial authority?
Peraciute
Guatar
Umbrelia
Anchor
Spoon
Toathbrush
Aepere momredhlltg slows ddsce Frrhughhon Liie Wirich m the used safe skydiving?
emichor steady From the which
drifting?
Drayon
Carrot
Une
Three
Plaoss usually have black and white keys. What object picturess covering the keys the pleno?
There only one moon orbiting the Earth, How mary moons 3r8 orbiting the planet this picture?
Irangle
paimting
pyramid consists the pyramid shown in the image?
Lisa in the Imaape? Please answer his/her first name.
Butterfy
Moth
Boe
Planyous
Kangaroo
Boar
caterpillar becomes butterfly. What does become
The platypus memmal shown layer?
mal that Which mammal the
imape?
Figure 8: Randomly sampled data from ViLP
20

---

Probing Visual Language Priors in VLMs
B.2. More details in ViL.Phencmarr data generation
In the below, try to propose questions along with three answers where the first answer is corresponding to the ques- tion text directly, while the other two are usual and counter- intuitive, which could lead to wrongs of VLMs. Please help generate more Question-3 answer pairs, which different from what have provided.
Our proposed dataset introduces Question-Image-Answe (QIA) triplets designed to challenge state-of-ther of-thh-art Vision- Language Models (VLMs) against visual language priors. The construction process combines human-guides and auto- mated efforts to ensure quality and alignment.
All the potential answers should a single world. Help me generate format where I can direct copy paste into Goole Sheet. Also, please a ; between question and each answers.
Question-Answer (QA) Generation: Most QA pairs are authored by humans following the design principles in Sec- tion 3.1. Additionally, candidate QA pairs are generated using models like OpenAT-O1 and Claude-3.5-Somee with carefully crafted prompts. One example prompts shown in Figure 10. These candidates undergo human review, where they are refined or removed to meet our quality standards.
Please be very creative and different from my provided examples the answer 2 & 3 should be very diverse and different compared to answer 1.
Every question contains statement the beginning which consists of the answerl as part of it. Please understand the principles and generate the QA very different from my provided examples
For each QA pair, we use GPT-4 to generate multiple descriptive image prompts (see Figure 9). These prompts are provided to image generation models, such FLUX and DALL-E 3, to produce candidate images. Human reviewers then select the most suitable image or request re-generation as needed to maintain consistency with the QA context.
Image Generation:
Some Examples:
A screwdriver is used for tightening screws. From the im- age, which tool is used to turn screws? Screwdriver Hammer Scissors
A pen is a tool used for writing. Which object in the image is used to write on paper? Pen Hammer Shoe
Clocks are used to measure time. Can you identify the item in the image that is used to measure time? Clock Spoon Candle
At every stage, human re- viewers rigorously evaluate the generated outputs to ensure quality, clarity, and challenge level. In addition to filtering out low-quality or insufficiently challenging triplets, we dy- namically test the QIAs to confirm that they remain intuitive for humans while being difficult for VLMs.
Human Review and Testing:
A violin has four strings and is played using a how. Accord- ing the image, which musical instrument is being played with bow? Violin Guitar Saxophone
Camels have humps. Which animal in the image stores fat in its humps? Camel Horse Tiger
Honey is made by becs. Which insect in the image produces honey? Bee Ant Dragonfly
An anvil is a tool used by blacksmiths. What object in the image is used by blacksmiths to forge metal? Anvil Fork Wrench
The complexity of our data creation process leads to a significant average cost of approximately $2.50 per QIA triplet in ViLP, excluding human labor costs.
Cost:
A gavel is used by judges in court. From the image, which object symbolizes judicial authority? Gavel Ham- mer Wrench
A syringe is used to inject medicine. From the image. which tool is used for administering injections? Syringe Scissor Drill
Task: Using the provided question and possible image-based answers, generate detailed text prompts for image generation. Each image prompt should reflect the question's 's context and incorporate one of the image-based based answers. Question: Question
An anchor keeps a ship steady in the water. From the image, which item prevents boats from drifting? Anchor Spoon Toothbrush
Image-based A Answer[[Answer For each possible image-based answer, create an image prompt that describes what the image might look like based on the question.
Answer2, Answer3]
A chainsaw is a power tool for cutting wood. What de- vice shown is typically used by lumberjacks to fell trees? Chainsaw Blender Stapler
Please be creativity. For example. if the question asks who using this mop to clean the floor in the picture? and the answer is craser. The image prompt should really describe the image of an craser uses a mop to clean the floor.
Figure 10: One prompt we used for potential QAs designs of ViLP
Format the output strictly as JSON list, like this example:
B.3. Ablation studies: realistic images
"prompt1": "Image Generation Prompt text here".
"promp22;: "Image Generation Prompt text here".
Our benchmark data are currently generated by DALL-E- 3 (Ramesh et al., 2021) and Flux, both of which produce cartoon-like, synthetic images rather than photorealistic ones. To assess the impact of image style, we regener- ated a subset of 45 QIA pairs using GPT-4o's 's latest image generation model to enhance realism, as illustrated in Fig- ure 11. We then measured changes in model correctness
"prompt3:: "Image Generation Prompt text here".
Figure 9: The prompt we used for generating text-prompt for image generation.
21

---

Probing Visual Language Priors in VLMs
Figure 11: Realistic image comparison. Each image pair shows our original benchmark data on the left and a corresponding realistic example generated by GPT-4o on the right.
when these more realistic images were used, with negative values indicating performance degradation. For definitions of these metries, please refer the beginning of Section The results in Table 6 show that increased realism slightly reduces performance for the "Score" metric in most cases, while its effects on "Prior" are generally negligible. These ablation findings suggest that introducing more realistic images may increase the task's difficulty, highlighting an important direction for future research.
Bench(Guan et al., 2023). While these datasets are impact- ful, their evaluation perspectives differ from ours, covering range from high-level design principles to low-level formats.
B.4.1. COMPARE TO WINOGROUND (THRUSH ET AL.. 2022)
Winoground centers on vision-linguistic compositional rea- soning by presenting models with two images and two cap- tions that contain the same words arranged differently. The goal match each image to correct caption based on the text's compositional structures and the visual content (as detailed the Introduction and Sec. 3.1 of (Thrush et al., 2022)). However, Winoground's captions do not chal- lenge language priors or introduce out-of nut-of-distribution visual information. Both captions adhere to common linguistic expectations, and there no explicit misleading information provided to test resistance to language biases. Additionally, most of their images are typical internet images, featuring common visual patterns.
ViLPE
ViLPE
ViL.É
ViLPE
GPT-4o
-2.2%
0
GPT-4o-mini
-2.2%
Clmud-Soonee-335
-2.2%
-4.4%
-3.3%
Blande-Opus-3
2.2%
-1.1%
-2.2%
-1.1%
Table 6: Impacts of Realistic Styles. Each value repre- sents the change in correctness when replacing the original images with realistic ones (Realistic- Original). Negative values indicate a drop in performance, suggesting increased task difficulty. The metric definitions are provided in the beginning of Section 5.
Qualitative Comparison: As shown in Figure 12 consist- ing of Winoground examples, both captions and images are normal and satisfy common linguistic expectations and com- mon sense. The evaluation focuses on whether the model can discern the compositional differences between the two images and two captions, then match them correctly. Com- paring Figure 12 and Figure 1, 7, 8, You discern the
B.4. Comparisons to other datasets
In this section, we compare our benchmark to other benchmarks, including Winoground (Thrush et al., 2022), Whoops!(Bitton- Guettta al., 2023a), and Hallusion-
22

---

Probing Visual Language Priors in VLMs
young an old person
dog white couch
dog brown couch
young person
than spoons
than spoons
the person with the lighter shoes is holding emoji the person with the derker shoes ian't
the person with derker shoes is holdling emoji and the person with the lighter shoes isn't
book la above the yellow book and below book
The yellow book is above the blue book and below the radl book
and two apples
and three apples
is red and right blue
left is blue and right
white land and black water
black land and white
a horse getting wet
getting a horse wet
red
Figure 12: Winoground Data Example. Our benchmark is different from Winoground, as Winoground focuses on vision-linguistic compositional reasoning. Both captions and images are normal and satisfy common linguistic expectations and common sense.
significant differences among the tested images.
Winoground does not challenge language priors eval- uate out-of-distribution scenarios; (2) the minimal score difference between the two captions confirms there is no significant variance in language priors, as examining how VLM models react to different language priors is beyond the scope Winoground. contrast, that aspect is precisely our focus.
Quantitative Comparison: Both ViLP; and Winoground benchmarks include paired textual information in their se- tups. In our benchmark, ViLP; Prior QAs and ViLP: P. Score QAs share the same question but differ their answers. Winoground, each example has two captions, and the task match each caption its correct image.
GPT eval- uate the commonness of these paired textual components. Specifically, GPT-4o rates the oddity of scenarios described in texts on scale from (very rare) to 10 (very common). The resulting scores are then compared.
Setting. To demonstrate the differences,
B.4.2. COMPARE TO WHOOPS! (BITTON-GUETTA
ET AL., 2023A)
Whoops! is designed to evaluate model's ability to de- tect weirdness in images, emphasizing tasks where images depict unusual or nonsensical scenarios. It heavily relies on common sense reasoning, requiring models to recog- nize visual elements and then identify subtle inconsistencies among them. For example, for the lit candle inside tightly sealed glass jar on the homepage, models must realize that "a candle needs a constant oxygen supply to burn, which would not exist in sealed jar". making a burning candle inside a sealed jar unlikely. This benchmark thus focuses on common-seen mensen-sense reasoning rather than challenging visual language priors.
Results. In our benchmark, Prior QAs scored 9.37, indicat- ing that these answers designed align with language priors and are highly common. Score QAs scored 1.65, showing that these QA pairs are rare, making them diffi- cult to infer without the corresponding visual information. Notably, Prior and Score QAs share the same question but differ their answers, and this significant contrast in scores showcases how we inject strong language priors to test model's vulnerability to linguistic distractions.
By comparison, Winoground's two captions scored 8.05 and 8.08, indicating two primary observations: (1) both captions align well with language priors, which means
Qualitative Comparison
Although Whoops! also includes creative, out-of-
distribution images, it does not focus on using language
23

---

Probing Visual Language Priors in VLMs
Q: where the mountains located in this scene?
Q: phone has what?
Q: what is inside blender?
Q: has what?
Whoops!: the distance
Whoops!: the physical buttons
Whoops!: A whole unpeeled
whoops!:
additional pincers
GPT-40: The mountains are in background beneath aurora borealis.
GPT-4o: smartphone has touchscreen and physical buttons.
GPT-40:
Bananas are inside the blender.
GPT-40: crab has pincers, hard shell (exoskeleton) and nultiple legs (typically 10, including claws).
Q: What's growing in the
0: Who is using an Apple laptop?
Where is the snowttan in relation the fire?
Q: What kind of fruit includes blue apple?
Whoops!: budding sunflowe
Whoops!:
Marilyn Monroe
Whoops!: Front
whoops!: group
GPT-4o: flower-like structure appears to be growing through the ice.
GPT-4o: woman with blonde hair using an Apple laptop. Her identity is not recognizable.
GPT-4o: The snowman sitting close to and facing the fire.
GPT-4o: fictional or artistic fruit.
Figure 13: Example of Whoops! Dataset. Dataset.Whoops. Whoops! also has creative images. While unlike ours, its questions common questions without strong language priors.
priors to test a model's susceptibility to linguistic distrac- tion, as our benchmark does. its QA mode (comparable to our task), the questions are straightforward and lack strong language priors. Some examples can be found Figure 13. Additionally, Whoops! uses open-ended questions, offer- ing greater freedom in answers while introducing potential ambiguity or divergence in responses.
and confident responses, whereas less suggestive question produces more varied answers. We calculate how many unique answers GPT-4o provides over five attempts at tem- perature 1.0 promote randomness. Semantic differences are normalized exclude synonyms.
Results. We find that Whoops! questions produce average of 2.58 unique answers (out of five attempts) with standard deviation of 1.48. For our benchmark, without facts, GPT- 4o provides an average of 1.53 unique answers (std 0.94), and with facts, 1.10 unique answers (std 0.42).
Quantitative Comparison
Setting. Unlike Winoground and our benchmark, Whoops! does not provide control groups or textual components for comparison. To measure language priors, analyze the suggestiveness of questions by evaluating GPT-4o's cer- tainty when answering them without any visual context. A more suggestive question typically yields more determined
Although both benchmarks use creative images, these re- sults indicate that Whoops! questions remain more general and do not push GPT-4o toward stereotypical responses. In contrast, our benchmark deliberately uses suggestive ques-
24

---

Probing Visual Language Priors in VLMs
Chart
Мар
Q: In 2828, did China have a larger population aged between 98 and 94 years than those aged between and 4 years?
q: According to the inage, is Miami the southernnost point of Florida?
Original
FLORIDA
FEORIDA
Modif foiffie
OCR
Table
According to the image, does (p1) range fron 3. 341599 to 3. 3.1455927
Does China have the most gold metals, follower by USA and Russia in 2008 beiing olynpis?
3=:3.4415 92653589793
T= 165
Gold
Siee
Storee
Gold
Siver
Bronce
shis
21
28
Chna
28
Stres
28
Unied Stte
16
38
36
92653589793
Resiin Fest
Rusan Fed
23
Great Briie
19
238443633383
238462643383
mon
Gemery
Astrola
2 279002884197169
279502884197169
Kase
399375 09937500588097480 494 459300781640225228888
39937510582097
Jpar
Mi
4592307816406286222
Fance
Farce
Figure 14: Example of HallusionBench. HallusionBench also has questions which can be answered without images. While is based on facts instead of stereotypes like ours. Moreover, its images limited Chart, Map, OCR and Table.
tions to elicit stereotype-consiste answers, reflecting our emphasis on testing language priors.
through subtle modifications (like changing digit in chart), which is fundamentally different from our approach of creating diverse visual contexts. Figure 14 showcases typical HallusionBench examples, illustrating the limited scope of both images and QAs, whereas our benchmark includes variety of out-of-distribution images and more suggestive QAs (see Figure 1, 7, 8).
B.4.3. COMPARE TO HALLUSIONBENCH (GUAN ET AL., 2023)
HellusionBench has two components: Visual Dependent, which focuses on testing models' general visual reasoning skills, and Visual Supplement, which examines model's visual reasoning in tandem with its parametric memory.
B.5. Human Study
For human evaluation, we hired Ph.D.-leve candidates to participate in testing. They were asked answer questions with single image provided each time, and the QIAs were randomly shuffled to avoid any sequential context. To ensure efficiency, we conducted an oral test instead of a written one, recording their responses. After the test, we updated the synonym sets for the QIAs based on their answers.
The Visual Supplement part is related to our benchmark, as its questions, like ours, can be answered without visual information. However, the key difference lies in their design. HallusionBenek questions rely on parametric memory and strict factual knowledge (e.g., "Which country has the most gold medals in the 2024 Olympics?") whereas our bench- mark questions are based on common stereotypes (e.g., "A soccer ball is round."). This distinction significantly con- strains the diversity of HallusionBench's questions--it of- while our
B.6. More failure cases analysis
ficial release contains only 50 question approach allows for broader range of scenarios, includ- ing texture, shape, conceptual combinations, hallucinated elements, and proverb-based contexts.
This subsection provides several typical failure cases ob- served the inferred results of VLMs on ViLP. We iden- tified several consistent failure patterns, illustrated Fig- ure 15. For deeper analysis, we also prompted the VLMs output their reasoning before finalizing answers. allowing to better understand how these errors arisc. Below,
Moreover, HallusionBench focuses on narrow set of image types (charts, tables, maps, and OCR). It tests reasoning
25

---

Probing Visual Language Priors in VLMs
usually stringe Question: Tower stuttec Farls From the Image you chantyy the lower bostach hane Cls".
typkallly shapa night sky image??
sttings doas have, showwn image?
Anasoning: with sumounding
Reasoningg:
exmmin
Reasoning: frst,
distrct tendmerics globsly
sugmented wrteete the brightness or orcato a spodilic visual affect the man body of the moon rebirol its tycioal rourateess but rw overal image more dhamalic and righigrod tie standard photograph the moon
possity to amphosson
queetion volii and newd its atrings vain has 4 ctring, upon looking disddy th mago
vileel tower depitres, resogrized symbal of paro. franc there aio also architcctura clemonts that risematoo those found boargss lor on nan aq moweer since it quasiiion speca loely ankk abou. the aili lowar and it sodi sodiusiilly losated in parts this can bad to the condlusion Bingle word Answer: Plris
prodieed, the volin depiced the pittow akid hee strings
Single word Answor: Round
Single word Answer: Four
Single word Antwwor Octagor
Comrect Single Anawor:
Single word Answor: Buijing
Question: common phrnse "he early oird catches the wom'.. What does the earty bird catch in the imaga?
Question: Microsoopes organisms. In the image, what piece of equipment viewiing organisme??
tiny
Question: umbrolis its handie. Which object n this image rapresents the handla of the umbrolla?
sexephone
Passsnning: the warly tind paiches oxam in the image
flaannng: luaed on its doscptiinn and Hiad zao in hu mago, EN uned yiw thy tepprtee *ie portruda mecroooinn undeczorenddaa "leea" maadd couttopy des gnad look dranader
Reasoning: obconving the Imago, thoro an object tha
aiiaaay resert kei a musizil ratrum uni and is comnaced the mann structure of the unbooll. fullling the tookal locanion function of an umbree hancle
jorns and ragee but sicats to rnpppod ohosse
Single word Arsover
Single word Anamer Microssop
Single Answer Secophore
Single
Single
Stopper
Single
Figure 15: Failure cases analysis. We list six typical failure cases generated by VLMs on ViLP.
summarize the failure patterns in the same order they appear in the figure, from left to right and top to bottom.
Sometimes the model overly relies memorized proverbs, resulting in predictions that align with these proverbs rather than the actual content of the input image.
Shape recognition can fail in VLMs, causing them to revert to priors rather than accurately interpreting visual input.
For images with creative concepts, the model may overly rely on its learned priors. As illustrated, a common prior that microscopes are used to view organisms, leading the model to answer "microscope" rather than identifying the creatively depieted stapler.
Models sometimes struggle to count accurately. Instead of performing an actual count, they default to relying on learned priors estimate quantities.
Models may refuse to accept visual information that contra- dicts their learned priors, whereas humans can comprehend hypothetical scenarios. For instance, the model recognizes the city as Beijing but rejects the correct answer because it expects the Fiffel Tower to be in Paris.
For images with blended features, the model may rely mostly on text input while overlooking the visual cues. As illustrated, the VLM heavily depends on textual input lead- ing to saxophone as the answer.
26

---

Probing Visual Language Priors in VLMs
C. Details in Image-DPO data generation and training
Our image-DPO data generation pipeline consists of two stages, In the first stage, we leverage the VLM we aim to enhance to perform self-guided data generation with the aid of pre- pre-trained image generative models. This stage produces large number of new pnestion-imaae-ama
(QIA) triplets. In the second stage, apply three types of image corruptions-Gauseee blurring, pixelation, and semantic editing to generate good-bad QIA pairs, denoted as I (good) and It (bad).
Details of the hyperparameters used in the experiments are provided at the end of this section.
C.1. VLM self-guided data generation
Generated Image
Generated QA
Seed Image
What the purpose of the green truck in the image? green
Pre-trained Model
VLM
VLM
GrOundingOINO Instruct Phx2Pxx
monster truck, the truck's green color and itss presence city be participating in promotional event show for the public.
suggest
Instructions
Tool Stable Diffusian,
Prompt: Generate
VLM
image inspired
truck driving the dirt road. The image
should feature futuristic cityscape with driving through streets. The truck should be the contral focus.
Rating of QA
The provides relevant information addressing of the truck and its purpose.... ... demonstrates an accurate understanding of the image's details Score:4 4
Justify:
Figure 16: Overview of our data generation pipeline. We begin with an image only, from which instructions are derived using a VLM. These instructions guide the creation of a new image or the modification of the existing one. The generated image is then processed by the VLM to generate QA pairs. Both the QA pair and the image are subsequently input back into the VLM assess the quality of the answers. No -writen in-context examples are used throughout this process.
As illustrated in Figure 16, data generation process begins by utilizing VLMS suggest modifications draw inspiration for input images without relying any in-context examples. The used text prompt is shown in Figure 18. Subsequently, pre-trained models such as Stable Diffusion XL (Podell et al., 2023), Instruct-Pix2Pix (Brooks et al., 2023b), and Grounded-SAA -AM (Ren et al.. 2024) are employed to either modify existing images or generate entirely new ones.
The altered or newly created images, along with the instructions that guided their generation, are then used by the same VLMs to produce corresponding question-answe pairs (QAs) based on the text prompt shown Figure 19. An example of this process is provided Figure 17. Importantly, all instructions, tool selections, and QA generation are autonomously handled by the same VLM aimed to improve.
In particular, Grounded-SAM requires the VLM to specify the object to be modified before generating images. To facilitate this, we use an additional text prompt (Figure 20) after the VLM generates the initial instructions (the pink region of Figure 17).
To provide a better understanding our generated QIAs, we randomly sampled and listed some examples of the generated QIA data, as shown in Figures 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, and 35.
27

---

Probing Visual Language Priors in VLMs
Instructions
Prompt Generate image a snowy night in city,
How does the image corvey the atmosphere o winter right the city? winter night through
main subject. Tew people wilking around, some benches, and few Make
The combination yellow
ground,
visuall ongazing by adding hine surrealism.
Image
Instructions
modification Generated QA
Tool Instruct-PhePPic
Prompt: Transform the show landccape people walking benches nearby, Adjust the colors make the
image?
A: The catiin enhanced the coler palette visual appeal the image
by adjusting the colors to create more vivid and ongaging scend..
Generated
Instructions
a: What elements were added or removed the transttion from the original image to the new image, and why might these changes have been mode? An the tranaition people were removed, and humanaid figure rehot of people and the additionn humanotd figures facus image sspect...
Tool Grounded SAM
Prompt: Replace the people the train with
added. The
figure.
suggest
element technologccal
Figure 17: Randomly sampled generation examples. Our proposed data generation pipeline utilizes pretrained model to generate diverse new images from existing seed image datasets.
Given this image, please suggest range creative edits, tasks, or transformations that could applied using advanced image processing tools. These tasks may include artistic transformations, vivid color adjustments, object detection and modification, or completely creating new image inspired by the original. Specify which tool would be hest suited for cach task, choosing from Stable Diffusion for image generation, InstructPix2Pix for image modification, GroundingDINO for object modification. Your recommendations should help in understanding the potential of the image and exploring creative possibilities.
Expected Response Format: Item Number:
Tool Used: (Specifyy the tool Stable Diffusion InstructPix2Pix GroundingDINO)
Text Prompt for Processing: [Detailed description the task transformation be performed. For image generation, please provide complete description based on the understanding the provided images, since we only feed text prompt for this task.]
Item Number: 2
Tool Used: [Specify the tool Stable Diffusion Pistr GroundingDINO)
Text Prompt for Processing: [Detailed description of the task transformation he performed. For image generation, please provide complete description based on the understanding the provided images, since we only text prompt this task.]
Item Number:
Tool Used: [Specifyy the tool Stable Diffusion or InstructPix2Pix GroundingDINO)
Text Prompt for Processing: [Detailed description of the task transformation be performed. For image generation, please provide complete description based the understanding the provided images, since we only feed text prompt for this task.]
Figure 18: The prompt for instruction generation. We ask the VLM to generate instructions for using pre-trained image models.
28

---

Probing Visual Language Priors in VLMs
Given this image, could you please generate series of insightful and diverse question-answar pairs hased the image and its descriptions? are interested exploring various facets of the image, including:
Holistic styles and layouts: Questions that analyze the overall design, style, and layout of the image.
Object-specific details: Questions that delve into particular elements objects within the image. discussing their characteristics functions.
Background context:
Questions that speculate about background story or the setting of the image.
Overall themes: Questions that interpret the thematic elements and messages portrayed in the image.
We encourage creative and thought- provoking questions that extend beyond the basics. Please generate questions that cover a broader range of observations and insights drawn from the image. Each question should be followed by comprehensive answer, providing depth and context.
Expected Multiple Response Format: Item Number:
Question: [Propose unique and insightful question based the descriptions and the images.]
Answer: [Provide comprehensive answer to the proposed question.]
Item Number:
Question: [Propose unique and insightful question based on the deseriptions and the images.
Answer: [Provide comprehensive answer to the proposed question.]
Please each question-answer pair is well-defined and informative.
Please provide at least 5 question-answe pairs based on the input provided.
The prompt for single-image QAs. We ask the VLM itself to generate single-image QAs based on the generated images by pre-trained models.
Figure 19:
Analyze the provided image and its accompanying modification instruction to identify the removed object description, the new object description, and the new image description.
Modification Instructions: <Text Prompt for Processing>
Expected Multiple Response Format: Item Number:
Removed Object Description: [Bricf description the object to be detected and removed]
New Object Description: [Description of a new, different object to replace the removed one]
New Image Description: [Description of the image after each object's removal, focusing on changes and remaining elements]
Item Number:
Removed Object Description: [Brief description of the object to be detected and removed]
New Object Description: [Description of a new, different object to replace the removed one]
New Image Description: [Description of the image after each object's removal, focusing on changes and remaining elements]
Figure 20: The prompt for instruction generation of Grounded-SAM. We ask the VLM generate designated instructions to use Grounded-SAM,
C.2. Image DPO data preparation and training details
This section details the construction of good-bad question- image- image-answer answer (QIA) pairs (L.,, Ii) based on the QIAs generated by the pipeline described in Appendix C.I. In brief, the data generation pipeline outlined in Appendix C. utilizes VLMs in conjunction with pre-trained image models to generate or modify images and create corresponding question-ansen answer pairs. This process results collection of QIA triplets, as illustrated in the Figures 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, and 35.
After generating the QIA triplets, apply three image corruption methods Gaussian blurring, pixelation, and semantic editing-ty to create god QIA pairs for ImageDPO training (Section 4. 1), while keeping the QA components unchanged.
For Gaussian blur, we use a kernel size of 40 for Cambrian and 80 for LLaVA, as the larger kernel size showed better performance for LLaVA. For pixelation, we apply block sizes of 32 for Cambrian and 64 for LLaVA. For these two types of corruption, we utilize data generated by all three pre-trained models: Stable Diffusson, Instruct-PixX2 Pix2rxx, and Grounded-SAr
29

---

Probing Visual Language Priors in VLMs
However, for semantic editing, rely solely on data from Grounded-SAM, which modifies single object in the image while leaving the rest unchanged. The object be altered and the type of modification determined by the VLMs based the text prompt (Figure 20). For instance, the VLM decides add headset the chef, as shown in Figure 21. Grounded-SS SAM then modifies the image accordingly by adding the headset, after which the VLM generates single QA pair based on the text prompt (Figure 19). Note that, we retain only images where the altered region covers more than 10% of the image size, as this ensures a significant and distinct change.
For model training, we constructed datasets comprising 800k and 400k DPO pairs to fine-tune LLaVA (7B and 13B) and Cambrian-8B, respectively. Batch sizes are set to 112 for LLaVA-7B, 80 for LLaVA-13B, and (with 4 gradient accumulation steps) for Cambrian-8B We employ LoRA with rank of 128, alpha of 256, and a learning rate of 5e-7, training each model for 2 epochs. The GPUs we used are 8-L40S.
Q: What is the significance of the headset in the man's work environment?
A: The headset is likely used for communication purposes, such as coordinating with other kitchen staff, receiving instructions from a manager or client, or listening to music create more enjoyable working atmosphere. In busy kitchen environment, clear communication and efficient coordination are essential to ensure smooth operations and maintain high standards of food quality and safety.
Q: What are the potential benefits of the man wearing a headset while working in the kitchen?
A: Wearing headset while working in the kitchen can provide several benefits. First, it can help the man communicate more effectively with his colleagues or clients, ensuring that everyone is on the same page and any issues are addressed promptly. Second, can help him focus on his tasks by minimizing distractions and external noise, allowing him to concentrate on his work. Finally, if the headset plays music, can create a more pleasant and up
Figure 21: Randomly sampled single-image QA generated by our data generation pipeline.
30

---

Probing Visual Language Priors in VLMs
Q: What is the purpose of the futuristic train depicted in the image?
A: The futuristic train in the image serves as mode of transportation, likely designed for high-speed travel and efficiency. The sleek, modern appearance of the train suggests that it is built for the future, with advanced technology and features to enhance the travel experience for passengers.
Q: What message does the image of the futuristic train convey about the city's progress and development?
The image of the futuristic train conveys a message of progress and development the city. The sleek and modern design of the train suggests that the city is constantly evolving and improving its infrastructure to meet the needs and expectations of its residents and visitors. The train's presence also indicates that the city is investing in advanced transportation systems to enhance the quality
A:
Figure 22: Randomly sampled single-image QA generated by our data generation pipeline.
Q: What is the purpose the fence in the image?
A: The fence in the image serves barrier between the sheep and the viewer, creating a sense of separation and emphasizing the boundary between the animals and the human observer. The fence may also serve a practical purpose, such keeping the sheep contained within a specific area or protecting them from potential threats. Furthermore, the fence could be seen as a symbol of the relationship between humans and animals, with the fence acting as a physical representation of the distance that can exist between the two.
Q: What is the significance of the sheep standing next to the fence in the image?
A: The sheep standing next to the fence in the image could be interpreted as a representation of the boundary between nature and human-made structures. The fence may symbolize the barrier that separates the animals from their natural habitat, while the sheep represent the resilience nature and the animals' ability to adapt to their environment. Additionally, the sheep's presence near the fence may also signify the potential for interaction between humans and nature, as well as the importance of maintaining a balance between the two.
Figure 23: Randomly sampled single-image QA generated by our data generation pipeline.
31

---

Probing Visual Language Priors in VLMs
Q: What is the tablecloth's design and how does it complement the colorful bowls and plates?
A: The tablecloth has a multicolored, patterned, and floral design. This design complements the colorful bowls and plates set atop the table, as they are visually appealing and create harmonious aesthetic. The vibrant colors within the colorful bowls and plates, as well as the lively tablecloth, offer a lively and inviting ambiance, making the dining table appear more vibrant and engaging. This can contribute to creating a lighthearted and cheerful atmosphere for diners enjoying their meal at the table.
Q: What is the purpose of the potted plants in the image?
A: The potted plants in the image serve decorative elements, adding touch of greenery and natural beauty to the setting. They can also help create more relaxed and welcoming atmosphere, making the dining experience more enjoyable for guests.
Figure 24: Randomly sampled single-image QA generated by our data generation pipeline.
What design elements can be observed in the kitchen that contribute to its modern and futuristic appearance?
Q:
A: The kitchen features sleek and minimalist design, with white appliances and surfaces that create clean and uncluttered look. The presence of a large island in the middle of the kitchen adds functionality and creates a focal point for the space. The lighting, particularly the presence of a light on the ceiling, enhances the futuristic atmosphere of the kitchen. The overall layout and design choices reflect a contemporary aesthetic, making the kitchen feel both stylish and efficient.
Q: What is the significance the presence of fruits in the kitchen?
A: The presence of fruits in the kitchen suggests that the homeowner values healthy eating and incorporates fresh ingredients into their meals. Fruits are an excellent source of vitamins, minerals, and fiber, and their inclusion in the kitchen indicates commitment to maintaining balanced and nutritious diet. Additionally, the fruits add a touch of color and natural beauty to the kitchen, enhancing its overall aesthetic appeal.
Figure 25: Randomly sampled single-image QA generated by our data generation pipeline.
32

---

Probing Visual Language Priors in VLMs
Q: How does the arrangement of the clocks and figures contribute to the overall design of the space?
A: The arrangement the clocks and figures in the image creates a visually striking and cohesive design. The clocks, with their different shapes and styles, are placed in way that draws attention to each one, while the figures surrounding them add an element of balance and harmony to the composition. The overall effect is a space that is both functional and aesthetically pleasing, inviting viewers to appreciate the artistry and craftsmanship of the clocks and figures.
1I M
Q: What the context or setting of the image?
A: Based the image, it is difficult to determine the exact context or setting in which the clocks and figures are displayed. However, it can be inferred that the space is likely a public or semi-public area, such as a lobby, gallery, or museum, where art and design are showcased. The presence of multiple clocks and figures suggests that the space may be used to convey message theme related to time human experience, or simply to provide an aesthetically pleasing and functional display.
Figure 26: Randomly sampled single-image QA generated by our data generation pipeline.
Q: What is the significance of the green-themed bathroom and the choice of plants used for decoration?
A: The green-themed bathroom with green plants on the walls and a green bathtub suggests a connection to nature and a focus on creating a relaxing and calming atmosphere. The choice of greenery decoration adds a touch of freshness and tranquility to the space, making it an ideal environment for unwinding and rejuvenation. Green plants also known for their air-purifying properties, which can contribute to a healthier and more comfortable indoor environment.
Q: How does the green color scheme and the use of plants affect the overall ambiance of the bathroom?
A: The green color scheme and the use of plants in the bathroom contribute to a serene, soothing, and natural atmosphere. The green hue is often associated with tranquility, relaxation, and freshness, which are all essential elements in a bathroom setting, The incorporation of green plants adds touch nature and life the space, making it feel more inviting and comfortable. Additionally, the plants can help create a more balanced and harmonious environment, promoting sense of well-being and relaxation.
Figure 27: Randomly sampled single-image QA generated by our data generation pipeline.
33

---

Probing Visual Language Priors in VLMs
Q: What is the purpose of the giraffes being in a line?
A: The giraffes are lined up in the image, possibly for variety of reasons. One possibility is that they are waiting for treat or meal, as giraffes often gather around food sources in the wild. Another possibility is that they are standing in line as part of a social behavior or a natural formation that occurs within their herd. This line formation could also be a result of the giraffes being trained or directed by zookeeper wildlife caretaker.
Q: What is the color palette used this painting, and how does it contribute to the overall mood or atmosphere?
A: The painting features a green and orange color palette, which is commonly associated with natural environments such as forests or grasslands. These colors evoke sense tranquility, warmth, and vitality, creating vibrant and lively atmosphere that complements the giraffes' natural habitat. The use of these colors in the painting also helps create a sense of depth and dimension, making the image more engaging and visually striking.
Figure 28; Randomly sampled single-image QA generated by our data generation pipeline.
Q: What is the significance of the teddy bear holding
cheerleader's baton?
A: The image of the teddy bear holding a cheerleader's baton is likely symbolic or playful in nature. could represent a celebration of team spirit, friendship, or support for sports team school event. The combination of the stuffed animal and the baton might evoke feelings of nostalgia, innocence, and youthful enthusiasm.
AUSTRALIA
UL Jæ
Q: How does the image of the teddy bear holding a cheerleader's baton relate to the theme of cheerleading?
A: The image of the teddy bear holding cheerleader's baton relates to the theme of cheerleading by incorporating symbol of team spirit, support, and enthusiasm. The presence of the baton suggests that the teddy bear is cheerleader, while the stuffed animal represents a more innocent, youthful, and friendly aspect of the cheerleading experience. The combination of these elements creates a visually appealing and emotionally evocative image that highlights the positive and uplifting aspects of cheerleading.
Figure 29: Randomly sampled single-image QA generated by our data generation pipeline.
34

---

Probing Visual Language Priors in VLMs
Q: How is the man's eating experience different from regular meal?
A: The man's eating experience different from regular meal because the sandwich he is eating is significantly larger than what one would typically consume. This unusual size could make the eating experience more challenging and time-consumingg, as he would have to consume the sandwich in smaller bites even take it apart to eat it. It could also be a novelty or a fun experience for the man, as he attempts to eat the giant sandwich.
Q: What is the significance of the man eating a giant sandwich in the image?
A: The image of the man eating a giant sandwich could signify various things, such the enjoyment of food, indulgence, or even a metaphor for taking challenges or overcoming obstacles. The sandwich might also represent a cultural or personal preference for the individual. The image could be used for advertising or promoting a restaurant or food establishment, or it could be piece of artwork meant to evoke a particular emotion or message.
Figure 30: Randomly sampled single-image QA generated by our data generation pipeline.
How does the color of the grass in the image contribute to the overall mood and atmosphere?
Q:
The lush green grass in the image creates serene and peaceful atmosphere. The vibrant green color symbolizes growth, renewal, and harmony with nature. The presence of elephants walking across the grassy field adds the tranquil and natural setting, evoking sense of wonder and appreciation for the beauty of the natural world.
A:
Q: How does the image of elephants in field relate to the broader topic of conservation and wildlife preservation?
A: The image of elephants a field serves as a reminder of the importance conservation and wildlife preservation efforts.
Elephants are considered vulnerable species, with their populations facing threats such as habitat loss, poaching, and human-wildlife conflict. The image highlights the need protect and conserve these majestic creatures and their habitats, ensuring their survival and well-being for future generations.
Figure 31: Randomly sampled single-image QA generated by our data generation pipeline.
35

---

Probing Visual Language Priors in VLMs
Q: What is the purpose of motorcycle's kickstand?
A: The kickstand used to keep the motorcycle stable and upright when it is parked.
Q: What is the overall design and style of the motorcycle?
A: The motorcycle has a sleek, modern design, and it appears to be well-maintained and ready for use.
Q: What is the significance of the motorcycle's location on the street?
A: The motorcycle's location on the street suggests that is either parked or waiting to be ridden, and it is likely that the owner nearby. The motorcycle's position on the street could also indicate that the owner is taking break from riding or has stopped for a specific purpose, such as visiting nearby business or meeting someone.
Figure 32: Randomly sampled single-image QA generated by data generation pipeline.
What architectural style is the building with the clock tower?
A: The building with the clock tower appears be church, featuring clock tower and a cross. This suggests that the architectural style is likely influenced by Christian Gothic styles, which are commonly found in churches. These styles often include pointed arches, ribbed vaults, flying buttresses, and intricate stained glass windows.
Q: What is the color scheme of the church's clock tower and cross?
A: The church's clock tower and cross are predominantly white, which is common color choice in religious architecture symbolize purity, innocence, and divinity. The white color may also serve to emphasize the contrast between the clock tower and the cross, highlighting the them.
Figure 33: Randomly sampled single-image QA generated by data generation pipeline.
36

---

Probing Visual Language Priors in VLMs
Q: What the primary ingredient in the dish being cooked in the pot?
A: The primary ingredient in the dish being cooked in the pot is broccoli.
Q: What is the role of the wooden spoon in the cooking process?
A: The wooden spoon is being used to stir the broccoli and other ingredients in the pot, ensuring even cooking and a well-mixed dish.
Q: What is the dish being prepared the pot?
A: The dish being prepared in the pot stir-fry, which is a popular and quick cooking method that involves frying ingredients in small amount of oil over high heat.
Figure 34: Randomly sampled single-image QA generated by our data generation pipeline.
How does the image capture the essence of a typical day at a ski resort? The image captures the essence of a typical day at a ski resort by showcasing the various activities and elements that one would expect to see during a winter sports trip. There are people skiing and snowboarding down the slopes, which indicates that the resort offers different types of winter sports for visitors to enjoy. The presence snow-covered pine trees the background adds to the picturesque winter landscape, creating serene and inviting atmosphere for guests. Additionally, the fact that the resort bustling with activity suggests that it is a popular destination for winter sports enthusiasts, further emphasizing the essence of typical day at ski resort.
Q:
A:
Q: What is the significance of the snow-covered pine trees in the image?
A: The snow-covered pine trees in the image serve as a beautiful and natural backdrop for the ski resort. They add to the overall wintery atmosphere and enhance the picturesque quality of the scene. Additionally, the presence of pine trees is indicative of the type of environment that ski resorts are typically located in, which is a mountainous region with a significant amount of snowfall during the winter months. The snow-covered pine trees also provide a sense of tranquility and harmony with nature, which can be appealing to visitors seeking peaceful and serene winter experience.
Figure 35; Randomly sampled single-image QA generated by our data generation pipeline.
37


Reviews:
[{"review_id": "l2lLri0B9p", "reviewer": "Reviewer_4vBy", "summary": "This paper investigated the problem of over-reliance on visual language priors of Vision-Language Models (VLMs) instead of visual reasoning. To study this problem, this work further proposed a benchmark ViLP containing 900 question-image-answer triplets, covering both prior answers and test answers, revealing that GPT-4o only scored at ~66%. This work further proposed Image-DPO, a\nself-improving approach to enhance VLM visual reasoning performance by increasing reliance on visual inputs, applied on LLaVA-v1.5 and Cambrian.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the constructive feedback from Reviewer 4vBy. We seriously address your comments below.\n\n---\n\n> Regarding the style of images. \n\nWe appreciate this valuable point and acknowledge that some generated images appear cartoon-like or synthetic, due to the image generation tools selected—among the best available at the time. However, since our objective is to benchmark VLMs using out-of-distribution QIAs that are not readily available online, we adopt image generation models to achieve this goal – a motivation appreciated by Reviewer j6Xe *“Prior works focus on benchmarking real-world data…The proposed benchmark revisits and shifts this particular limitation”*. \n\nRegarding the cartoon-style concern, we would like to respectfully point out that current VLMs are commonly trained on datasets that include synthetic and cartoon-style images. For example, [LAION](https://laion.ai/blog/laion-aesthetics/), widely used in training VLMs, contains subsets focused on *anime*, *cartoon*, and *aesthetics* (filtered by community), reflecting the prevalence of synthetic imagery online. VLMs have shown robust performance on such data, and understanding synthetic visual representations remains an important capability, especially given humans’ ease in interpreting them. \n\nTo assess the impact of image style, we regenerated a subset of 45 QIA pairs using GPT-4o’s latest image generation model to enhance realism (**[Figure](https://postimg.cc/dZTtNwtW)**). We then measured changes in model correctness when using these realistic images, with negative values indicating performance degradation. The results show that increased realism slightly reduces performance in most cases for the “Score” metric, while most effects on “Prior” are zero. These ablation results suggest that incorporating more realistic images may increase task difficulty, highlighting an important direction for future research.\n\n| | $ViLP^F_{Prior}$ | $ViLP^F_{Score}$ | $ViLP^P_{Prior}$ | $ViLP^P_{Score}$ |\n| -| - | - | -| -|\n| GPT-4o                                  | 0 | -2.2% | 0 | 0 |\n| GPT-4o-mini                                    | 0                | -1.1%               | 0                | -2.2%               |\n| Claude-Sonnet-3.5                              | -2.2%               | -4.4%               | 0                | -3.3%               |\n| Claude-Opus-3                                  | 2.2%                | -1.1%               | -2.2%               | -1.1%               |\n\n\n---\n\n\n> Regarding Proposition 1. \n\nWe would like to improve this point in the revision. For clarification, proposition 1 is the theoretical foundation of our Image DPO algorithm, showing that minimizing our proposed Image DPO loss corresponds to minimizing an upper bound on the RLHF loss. Unlike typical DPO modifications that adjust the output, our method fixes the output and alters the conditioning. Proposition 1 thus rigorously establishes the connection between our modified objective and the original DPO formulation.\n\n\n---\n\n> Regarding the ambiguity and single-word evaluation. \n\nWe thank the reviewer for highlighting the inherent ambiguity in counterfactual visual reasoning. Although ambiguous scenarios occur, humans typically resolve them effectively after initial confusion, as evidenced by the strong human performance reported in our benchmark (Table 2), where evaluators had access to identical textual and visual inputs as the VLMs. Thus, the noted \"ambiguity\" intentionally examines whether VLMs can match human performance under controlled conditions.\n\nRegarding single-word outputs, we highlight their evaluation advantages—including efficiency, speed, and reliability—when combined with our synonym and plural detection pipeline. In contrast, sentence-based evaluations relying on LLM inference incur API fees, computational costs, sensitivity to hyperparameters, and occasional unreliability due to incorrect reasoning.\n\nWe appreciate the reviewer's concerns about potential limitations of single-word answers. To explore this, we conducted ablation studies prompting GPT-4o in a \"chain-of-thought\" manner, instructing it to reason before giving a sentence-level final answer. We evaluated these answers against ground truths using GPT, comparing both reasoning steps and final sentences with original single-word responses.\n\nInterestingly, allowing longer answers with more reasoning tokens did not yield performance benefits. As shown in (**[Figure](https://postimg.cc/XrpqpR57)**),  correct reasoning (second pic.) can still produce incorrect conclusions, and sometimes the reasoning process itself may be flawed (third pic.). Quantitatively, GPT-4o’s performance on \"Prior\" slightly improved from 91% to 92.67%, whereas on \"Score\" it decreased from 66.17% to 55.5%. Given that \"Prior\" aligns closely with language priors and \"Score\" represents out-of-distribution scenarios, we hypothesize that longer outputs may unintentionally increase reliance on priors during reasoning.", "claims_and_evidence": "The claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and benchmark dataset make sense.", "theoretical_claims": "The paper does not include theoretical claims.", "experimental_designs_or_analyses": "The overall soundness/validity of the experiments look good.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n* The paper is well-written and organized. The three criterions are clear and mathematical to understand how the vision-language priors are scoped.\n\n* The Image-DPO solution makes sense and is aligned with the motivation to highlight visual clues. Table 3 and 4 show that Image-DPO outperforms baselines and other preference optimization methods.\n\nWeaknesses & Questions:\n\n* The images are generated by DALL·E-3 and Flux, which look quite different from real-world images. The cartoon style may have impacts on VLMs as they are not directly trained on synthetic images. If real-world but rare images or realistic-style (or more styles) can be included to ablate the impacts of image styles. Therefore, the conclusions of this paper should be bounded by cartoon style.\n\n* It is not clear about the role of Proposition 1 as it is not mentioned in the main text after being introduced.\n\n* Some answers to the counterfactual questions are ambiguous. Take Figure 3 as examples, for the upper right example, it is hard to tell the cities are London and Paris, as Moscow can also build rivers or Tour Eiffel model. Also for the lower right example, the camel can also have a horse face by wearing a mask. There lacks strict definitions of these nouns and it is hard to tell the MLLMs' answers are not reasonable. The reasons might be that the models are forced to output a single word as answers. Actually, the explanations of the answer can provide more information on how the model observe the image.", "other_comments_or_suggestions": "Please see Weaknesses", "questions_for_authors": "Please see Weaknesses. My main concerns are the style of images and the ambiguous answers due to the single-word evaluation setting. My current score is actually borderline.\n\n\n--------------\n\nAfter reading the rebuttal, I decided to increase the score to accept. The analysis on the impact of image style is insightful and answered my questions. Since my major concern is fully addressed, I am happy to accept the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "47Qr4GHaCE", "reviewer": "Reviewer_1kEP", "summary": "The paper presents an investigation into the over-reliance of vision-language priors in existing Vision-Language Models (VLMs). To enable it, the paper presents ViLP, a carefully designed benchmark which consists of Prior Answer that can be directly inferred from the question, and Test Answers which rely on visual cues. Their evaluation reveals that all methods significantly fall short of human annotations. To improve upon this shortcoming, ImageDPO is developed, the core of which is to develop pair-wise images where a corrupted image is generated from a real image using a VLM-based pipeline which uses techniques and models such as image editing and Grounded SAM. The authors theoretically prove ImageDPO and empirically show that when applied to OSS models such as LLaVA, ImageDPO improves upon other methods in solving ViLP.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the positive review provided by  Reviewer 1kEP, and are grateful that Reviewer 1kEP “likes the paper structure”, and “convinced with the methodology adopted in the paper”,  believes “the  ViLP dataset is well motivated”, and “the results are promising as well”.\nBelow, we provide our detailed responses to your comments.\n\n---\n\n> I have the following question: a) Since data generation for DPO training uses image-editing, I wonder if there is some quality control established to threshold what is an accepted corrupted image?\n\nThat's a good point! Our guiding principle is to eliminate human involvement in order to make the pipeline fully automatic and scalable. While the generated data may contain some noise, we believe the pipeline is robust enough to help improve VLM performance, which is supported by the improvements in the experiments. For corrupted images, we control the hyperparameters of image editing, such as the strength of Gaussian blur and pixelation, to ensure that the corrupted versions are noticeably worse than the original ones. In the case of semantic corruption, we calculate the number of pixels that are modified and discard cases where the changes affect less than 5% of the total pixels, as such minor alterations may be too subtle to detect.\n\n---\n\n> The paper could do with a bit of re-writing and typos. For example, Line 1152.\n\nThank you for your thorough review! We have corrected the typo on line 1152 in the latest revision and will carefully proofread and refine the manuscript to enhance readability further.", "claims_and_evidence": "1. Overall, I like the paper structure. The paper introduces a new benchmark, has interesting experiments and ablations, and a DPO-based methods that help improve the benchmark.\n2. The ViLP dataset is well motivated -- humans do well despite having distractors whereas VLM's clearly fail as shown by the paper's evaluation protocol.\n3.  ImageDPO is intuitive as well, using divergent image data to force the model to learn through image differences. The results are promising as well.", "methods_and_evaluation_criteria": "1. Overall, I am convinced with the methodology adopted in the paper.\n2. I have the following question : \na) Since data generation for DPO training uses image-editing, I wonder if there is some quality control established to threshold what is an accepted corrupted image?", "theoretical_claims": "1. The paper derives the ImageDPO in Appendix B. I do not find issues with it.", "experimental_designs_or_analyses": "I checked the soundness of the experiments. I am convinced, overall.", "relation_to_broader_scientific_literature": "The paper covers previous related work and evaluates a large range of models, both open and closed-sourced.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. The paper could do with a bit of re-writing and typos. For example, Line 1152.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please check above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VZRtz1ykXi", "reviewer": "Reviewer_j6Xe", "summary": "This paper introduces ViLP, a benchmark to investigate how VLMs relies heavily on textual priors, and ignore visual inputs. ViLP consists of 300 questions (each including a distractor fact), and three image-answer pairs (a prior answer and two test answers). With extensive experiments, the authors show that existing VLMs struggle with this benchmark. \nTo improve the performance of VLMs, the authors also propose Image-DPO, a fine-tuning strategy (based on DPO approaches) that focus on the visual space/pixel space. The authors propose a method for generating training examples (seeded from COCO, Text2VQA, Visual Genome). Improvements of +2 to +3% are shown in their proposed benchmark.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer j6Xe QTDS’s thoughtful and encouraging feedback. Due to constraints in time and computational resources, we have prioritized addressing your question and will incorporate the suggested additional improvements in future revisions.\n\n---\n> Why is it necessary to fine-tune using additional synthetic images?\n\nThis is a very good point! We employ a comprehensive pipeline that uses multiple image-generation models to synthesize additional images from seed datasets. Our primary motivation is diversity: by incorporating a broader range of synthetic images, the VLM is exposed to more varied content and can, in turn, produce more diverse Question–Image–Answer (QIA) pairs. This process also implicitly leverages the knowledge locked in the existing image-generation models. Compared to relying solely on the original dataset, these additional QIA pairs enrich the training signal, helping the model learn more robust and generalizable representations.\n\nTo verify this point, we conducted an ablation study using the same dataset as in our ImageDPO experiments. Specifically, we directly generated QA pairs from images in the COCO, VG, and Text2VQA datasets without additional modifications or synthetic generation, i.e., no synthetic images. We then applied image corruptions to produce altered versions of these images and utilized these corrupted-image QA pairs for ImageDPO finetuning. The results, shown below, indicate notable performance drops. This outcome underscores that leveraging synthetic data, rather than relying solely on the original datasets, benefits the DPO training process.\n\n|                          | $ViLP^F_{Score}$ | $ViLP^F_{Prior}$ | $NB_Q$ | $NB_I$ | $NB_G$ | $NB_B$ | MM-Vet | $CHAIR^S \\downarrow$ | $CHAIR^I \\downarrow$ |\n| ------------------------ | ---------------- | ---------------- | ------ | ------ | ------ | ------ | ------ | -------------------------------------- | -------------------------------------- |\n| Ours                     | 34.17 | 39.33 | 39.79 | 45.47 | 14.16 | 68.45 | 32.3 | 45 | 12.3 |\n| Without synthetic images | 31.17 | 38 | 38.32 | 44.17 | 13.35 | 67.69 | 31.4 | 47 | 13.6 |\n\n\n---\n> The only weakness is the lack of comparison of their proposed Image-DPO to finetune more recent open VLMs (including LLaVA-NeXT, InstructBLIP, QwenVL and DeepSeekVL)\n\nThank you for this valuable suggestion! We agree that comparing Image-DPO with more recent VLMs (e.g., LLaVA-NeXT, InstructBLIP, QwenVL, DeepSeekVL) would provide additional insights. However, due to limited time and computational resources during the rebuttal period, we are unable to conduct these comparisons comprehensively. We appreciate this feedback and plan to include parts of the models in our future revisions. Nonetheless, we believe our current results on LLaVA-v1.5 and Cambrian sufficiently demonstrate Image-DPO’s effectiveness and hope they convey the potential of our approach.", "claims_and_evidence": "- Claim 1: VLMs ignore visual inputs and are biased towards textual priors -- this is backed and supported by empirical results. Extensive experiments on their proposed benchmark (ViLP) support this claim.\n\n- Claim 2: this can be fixes via Image-DPO: experimental results show some gains with their proposed method.\n\n- Challenging dataset: the methodology for dataset creation is sound and backed by human/manual evaluation (Appendix A.2)", "methods_and_evaluation_criteria": "The method, while simple, shows improvement over the baseline. The only problem is the lack of evaluation of more recent models that might perform better in the proposed benchmark (e.g., QwenVL, DeepSeekVL, etc).\n\nImage-DPO: leveraging existing datasets and modifying them via editing models/image generation approaches is time consuming, and the improvement seem marginal (~2 to 3%). However this is an interesting approach that future work could expand and build upon.", "theoretical_claims": "The proposed method seems correct, and aligns with the authors claims.", "experimental_designs_or_analyses": "Experimental analysis and design seems valid and sound.", "relation_to_broader_scientific_literature": "Prior works focus on benchmarking real-world data. VLMs tend to benefit from real-world common sense to answer questions in VQA datasets. The proposed benchmark revisits and shifts this particular limitation.", "essential_references_not_discussed": "No essential references missing (to the best of my knowledge)", "other_strengths_and_weaknesses": "The only weakness is the lack of comparison of their proposed Image-DPO to finetune more recent open VLMs (including LLaVA-NeXT, InstructBLIP, QwenVL and DeepSeekVL)", "other_comments_or_suggestions": "No additional comments", "questions_for_authors": "Why is it necessary to finetune using additional synthetic images?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b3pIjQGYwd", "reviewer": "Reviewer_hkyN", "summary": "This paper investigates visual language priors in vision-language models , analyzing how these models rely on textual biases rather than true visual reasoning. The authors introduce ViLP, a benchmark designed to expose such priors by presenting models with out-of-distribution  images and distractor text-based facts. The benchmark consists of question-image-answer triplets, where the correct answer contradicts common textual priors, requiring models to focus on visual reasoning.\n\nTo address the issue of over-reliance on text priors, the paper proposes Image-DPO, a training framework that introduces controlled image corruptions to force VLMs to develop stronger visual grounding. Experiments with LLaVA-v1.5 and Cambrian demonstrate that Image-DPO improves performance on ViLP and other multimodal benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the detailed and insightful feedback. Below, we have carefully addressed the opportunities for improvement you highlighted.\n\n\n---\n> Failure case analysis\n\nWe sincerely appreciate your suggestions and will include an in-depth failure case analysis section in revision.\nWe observe several consistent failure patterns, as shown in the  **[Figure](https://postimg.cc/18j0mVt4)**. For deeper analysis, we additionally prompted the VLMs to output their reasoning before answering. Below, we summarize the failure patterns in the order corresponding to the images.\n1. Shape recognition can fail in VLMs, causing them to revert to priors rather than accurately interpreting visual input.\n2. Models sometimes struggle to count accurately. Instead of performing an actual count, they default to relying on learned priors to estimate quantities.\n3. Models may refuse to accept visual information that contradicts their learned priors, whereas humans can comprehend hypothetical scenarios. For instance, the model recognizes the city as Beijing but rejects the correct answer because it expects the Eiffel Tower to be in Paris.\n4. For images with creative concepts, the model may overly rely on its learned priors. As illustrated, a common prior is that microscopes are used to view organisms, leading the model to answer “microscope” rather than identifying the creatively depicted stapler.\n5. For images with blended features, the model may rely mostly on text input while overlooking the visual cues. As illustrated, the VLM heavily depends on textual input leading to saxophone as the answer. \n\n---\n> Fine-grained evaluation and LLaVA-OneVision performance \n\nThanks for the suggestions! Currently, we explicitly ask VLMs to produce a single-word response for evaluation, as it is  (1) cost-efficient without calling LLM, (2) fast, and (3) reliable. For robustness, we implemented a comprehensive synonym-matching pipeline to validate synonymous responses accurately. We indeed penalize responses that include long sentences, as we believe instruction-following is a foundation capability of VLMs—thus, failure to comply warrants a penalty.\n\nWe acknowledge the value of your suggestion and have extended our evaluation system to include three categories: **Correct**, **Wrong**, and **Invalid**. “Invalid” refers to responses that do not meet the single-word requirement. Additionally, for “Invalid” outputs, we now provide an optional GPT-based evaluation that classifies them as either *Invalid-Correct* or *Invalid-Wrong*. The results is shown in **[Table](https://postimg.cc/SXgTQ1B4)**. The new evaluation enables us to assess the model’s performance which generates longer sentences.\n\n\n\n---\n> Regarding Winoground and HallusionBench benchmarks and unrealistic examples\n\nFor clarification, Appendix A.3 provides both quantitative and qualitative comparisons between our dataset and existing benchmarks such as WinoGround and HallusionBench, with key differences highlighted, ranging from high-level design principles to low-level formatting details.\n\nMoreover, our human study shows that participants can identify unnatural elements like tigers or moons. This underscores the human ability to reason about unrealistic or novel scenarios—an essential goal for advancing VLMs—and positions our dataset as complementary to existing benchmarks.\n\n\n---\n> Reference of [1]\n\nThanks for the suggestion, and we will discuss it [1] in the final revision. The key difference between ImageDPO and [1] is the training objective: while [1] follows the standard DPO setup by using preferred/dispreferred outputs for the same input, our approach fixes the output and varies the input conditions. This results in notable differences, like the distinct gradient behavior in Figure 13. Additionally, we introduce Proposition 1 and its theoretical proof in Appendix B, connecting our modified objective to the original DPO formulation.\n\n[1] Self-Supervised Visual Preference Alignment \n\n\n---\n> Connection between Image-DPO and the benchmark\n\nDirectly adjusting VLMs to improve specific abilities becomes challenging as the development of VLMs..  Our Image-DPO method addresses learned priors in a nuanced manner, where the proposed objective encourages the model to rely more heavily on the vision branch. Our objective drives a distinct shift in gradient directions when processing normal images $I_w$ compared to corrupted images $I_l$ (as illustrated in Figure 13). We appreciate your suggestion and will revise our wording to improve the connection.\n\n---\n> Potential better approach\n\nWe explored a similar idea but faced a key limitation: existing image-editing models often introduce visible artifacts or unnatural transitions between edited regions and the original background, making it easier for VLMs to detect anomalies. Additionally, they frequently struggle to follow instructions accurately. As editing technology improves, we plan to revisit this approach for future dataset extensions.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "See Weakness.", "theoretical_claims": "Yes. The paper provides a theoretical connection between Image-DPO and RLHF objectives.", "experimental_designs_or_analyses": "See weakness.", "relation_to_broader_scientific_literature": "Related to prior work on textual priors in VQA datasets (e.g., Winoground, HallusionBench).", "essential_references_not_discussed": "See weakness.", "other_strengths_and_weaknesses": "### **Strengths**\n- An interesting benchmark filling a gap in VLM research.\n- The proposed Image-DPO method improves model performance.\n- Comprehensive evaluation across multiple models.\n\n---\n\n### **Weaknesses**\n- **Lack of in-depth failure analysis.**  \n  - A more detailed breakdown of failure cases would be beneficial. For instance, what are the top three most common failure types in this benchmark?\n\n- **Methodological concerns regarding Image-DPO.**  \n  - Weak connection between Image-DPO and the benchmark itself. The proposed method does not seem inherently tied to addressing language priors, making Section 4 feel somewhat disconnected from the core benchmark motivation.  \n  - Lack of distinction from prior DPO methods. The paper does not clearly differentiate Image-DPO from existing DPO approaches, such as those in [1], which also augment images.  \n\n- **Benchmark realism and case selection.**  \n  - Some examples in the benchmark (e.g., Figure 3 with unnatural tiger and moon images) appear unrealistic.  \n  - Comparatively, benchmarks like Winoground and HallusionBench provide more realistic settings with real-world images and practical vision-language challenges.  \n  - A better approach might be to incorporate real images using image retrieval techniques, such as:  \n    - Identifying common real-world priors (e.g., cows grazing on grass)  \n    - Introducing subtle variations (e.g., a rabbit on the grass instead of a cow)  \n    - This approach aligns well with the paper’s motivation of probing visual priors.  \n\n- **Insufficient explanation of LLaVA-OneVision performance in Table 2.**  \n  - The paper does not provide a clear justification for LLaVA-OneVision’s performance in Table 2.  \n  - If the evaluation method penalizes long responses where the correct answer is embedded within a longer explanation, this could raise concerns about the robustness of the evaluation criteria.  \n  - Could you provide failure case of this model?  \n\n---\n\n### **Reference**\n[1] *Self-Supervised Visual Preference Alignment.*", "other_comments_or_suggestions": "See weakness.", "questions_for_authors": "See weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a benchmark, called ViLP, which constructs OOD image-text pairs to evaluate the reliance of VLMs on the language prior acquired during training to answer visual questions. The dataset is relatively small (contains only 300 examples) but the findings remain quite interesting and valuable to the community. For instance, the authors find that while humans can answer the questions easily, VLMs don’t (e.g. GPT4 achieves 66% accuracy). In addition, the authors recommend a DPO-like optimization algorithm for VLMs that is simple to implement and seems to outperform self-improvement baselines (which I find to be surprising). Proposition 1 helps give some insight for why the method works. Overall, this is a nice, insightful paper that is well-written. Overall, all reviewers and I have found the paper to be a valuable contribution.
Decision: Accept (poster)