vVLM: Exploring Visual Reasoning in VLMs against Language Priors

Authors: Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, Honglak Lee
Primary area: datasets and benchmarks
Keywords: Vision Language Model

Abstract:
The intersection of vision and language presents challenges, as vision language models (VLMs) may exploit language biases, reducing their reliance on visual input. To examine this, we introduce a benchmark that prioritizes visual reasoning in visual question answering (VQA). Our dataset generated using image generative models consists of visually intricate images that vary in texture, shape, conceptual combinations, hallucinated components, and proverb. Each question is paired with three answers and three corresponding images: one that can be easily inferred from the text, and two that must rely on visual cues. While humans can effortlessly discern all three answers, existing VLMs struggle as GPT-4o achieving only 66.17\%. Furthermore, we propose enhancing VLMs by self-generating VQA pairs and images via pre-trained image generation and editing models. These images are then subjected to pixel-level and semantic corruptions, creating good-bad image pairs for DPO training. This approach encourages models to rely more on visual input, and has shown to improve performance on LLaVA-v1.5 and Cambrian.

Decision: reject
Reviewer scores: 8, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1730655450104

thread_type:
official_review

<thread_object>
creator: reviewer_qtds

<content>
summary:
This paper investigates the bias of Vision-Language Models (VLMs) towards language by creating a synthetic Visual Question Answering (VQA) dataset of 900 images using state-of-the-art (SOTA) image generation models DALL.E-3 and Flux. The authors demonstrate that VLMs tend to rely heavily on language cues when interpreting visual questions, which can limit their performance in truly visual reasoning tasks. To address this issue, they apply Direct Preference Optimization (DPO) training on top of VLMs, using self-generated VQA pairs and corresponding images from pre-trained image generation and editing models. This approach aims to mitigate the language bias and improve the alignment of VLMs with visual content.

strengths:
1. This paper introduces an automatic way to check language bias of VLMs by synthetically generating images using SOTA image generation models.
2. This paper proposes Vision-Centered DPO by running DPO on automatically generated good/bad data where a QIA pairs, the QA remains the same and the Image is perturbed.

weaknesses:
1. The vVLM benchmark is interesting but there is no comparison with other datasets (e.g., HallusionBench[a]). The paper mentions that the generated​​ images vary in texture, shape, conceptual combinations, hallucinated components, and proverb. However, the distribution of these features in the data is not specified. It would be insightful to the community to learn in which dimension the existing VLMs are failing mostly. 
2. The DPO experiment and result section seems incomplete and is hard to follow through as there are little details of the training data, benchmarks and the purpose of choosing these benchmarks. It is unclear what amount of data has been synthetically generated for DPO training and what is the quality of the data. Image DPO works in some cases and not the best choice in others and there is no insight or empirical analysis to justify when Image DPO is useful. 


(a) https://www.researchgate.net/profile/Fuxiao-Liu-2/publication/375595895_HALLUSIONBENCH_You_See_What_You_Think_Or_You_Think_What_You_See_An_Image-Context_Reasoning_Benchmark_Challenging_for_GPT-4Vision_LLaVA-15_and_Other_Multi-modality_Models/links/65518863b86a1d521bd839cf/HALLUSIONBENCH-You-See-What-You-Think-Or-You-Think-What-You-See-An-Image-Context-Reasoning-Benchmark-Challenging-for-GPT-4Vision-LLaVA-15-and-Other-Multi-modality-Models.pdf

questions:
Q1. "Interestingly, we observe a strong correlation between the difficulty VLMs face in interpreting images and the challenges encountered by image generation models in producing those images. For example, when an image is easily generated, even weaker VLMs can correctly answer the associated QIA with relative ease." - What is the definition of difficulty here? Could you share some qualitative and quantitative analysis here?

Q2. "On vVLM$^P$-Score, the Image-DPO method achieved the second-best performance which indicates that with only question image combination, Image DPO is less effective. Is there any insight why this is the case?

Q3. In addition, in presence of prior, Image DPO is worse than RLHF-V. Is this because of the quality of the data? It would have been better to provide insights and empirical analysis on why Image DPO works on certain cases and why not in other.

Q4. It is not clear from the paper which VLMs have been used to create the DPO training data. Additionally, there has been no qualitative analysis on the quality of the generated  image and question-answer pairs.

Q5. It would be interesting to see a combination of both Image and Text-DPO and their impact on downstream tasks.

Q6. In table 3, we can see that with increasing model size the performance gap between Image DPO training and non-Image DPO training is reducing. It would be great to see any experiment that addresses the effectiveness of various DPO training with scaling the model size.

Q7. What is the performance of Image DPO on complex compositionality tasks (e.g., Winoground [1]) that checks image understanding ability of VLMs?

1. https://arxiv.org/abs/2204.03162
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720150953

thread_type:
official_review

<thread_object>
creator: reviewer_m3ut

<content>
summary:
This paper discusses the issue of language biases in Vision Language Models (VLMs) and introduces a benchmark that prioritizes visual reasoning in visual question answering (VQA). Additionally, it uses good-bad image pairs for DPO training, leading to improved performance on models like LLaVA-v1.5 and Cambrian.

strengths:
This paper addresses a significant and currently trending issue in VLMs: language bias. It proposes a scalable data generation pipeline, and the introduced DPO training method also appears to be intriguing. The writing of the paper is fluent and the sections are clearly structured.

weaknesses:
The examples presented in the paper appear somewhat confusing, such as the image corresponding to the question "In what city does the image depict the Statue of Liberty?" and the image for "What is the largest long-necked animal shown in the image?" These examples seem counterintuitive and are even not easy for humans to answer correctly. Additionally, the paper lacks a discussion on the quality control of the dataset.

questions:
1. How is the quality of the dataset ensured during the generation process to avoid discriminate, incorrect images and the awkward and meaningless questions? Are the dataset and code open source?

2. The abstract claims that “our dataset consists of visually intricate images that vary in texture, shape, conceptual combinations, hallucinated components, and proverbs,” but the paper lacks discussion on different categories.

3. In the experiments, it might be better to include experiments on benchmarks that eliminate bias, such as [NaturalBench](https://arxiv.org/pdf/2410.14669?) .
</content>
</thread_object>

</thread>
<thread>
creation date:
1730788072399

thread_type:
official_review

<thread_object>
creator: reviewer_nay6

<content>
summary:
This paper introduces a VQA benchmark that aims to ask visual questions that challenge the linguistic biases of vision–language models. This benchmark contains 300 questions with 3 image–answer pairs each. The questions are first generated by humans using OpenAI o1 and Claude-3.5-Sonnet. The three (single-word) answers are then written, such that the first $a_\text{prior}$ is likely and the latter two are unlikely given the question alone. An image is then synthetically generated for each question–answer pair using DALL-E 3 and Flux. Finally, a distractor fact is also included that reinforces the language prior in $a_\text{prior}$.

Many open- and closed- vision–language models are tested on this benchmark and show poor performance (< 70%) on the "unlikely" test examples. Human (98%) and text-only (GPT-4o; 0%) baselines are also provided.

This paper proposes a method for generating more VQA training examples (seeded from COCO, Text2VQA, Visual Genome) that are used for fine-tuning the models using DPO-based methods (the images are perturbed with pixel-level transformations to create negative examples). This method improves performance on the proposed benchmark by 1-3%.

strengths:
I believe the examples in this benchmark are very creative! The synthetic nature of this data permits very interesting and challenging combinations of concepts: e.g. houses built from pitayas or zebras with checkerboard patterns (Figure 1).

I appreciate that the benchmark is evaluated on a large set of vision–language models (Table 1). The benchmark is also sufficiently challenging.

The writing in this paper is reasonably clear.

weaknesses:
I have a few criticisms of this work. Unfortunately, I believe these are fundamental issues, but I would sincerely appreciate it if the authors could comment further.

1. While this work suggests that existing VQA benchmarks do not sufficiently "defy language priors", Winoground (Thrush 2022), (Hessel 2023), and Whoops! (Bitton-Guetta 2023) are vision–language benchmarks that do have this focus. Moreover, I believe these benchmarks remain sufficiently challenging, e.g. GPT-4v is SOTA and scores 60% on Winoground (Wu 2023).

2. I am quite concerned by the quality and content of the generated images. For example, in Figure 3: there is a flamingo present (alongside the tiger) in the image where "tiger" is the correct answer (and "flamingo" is supposed to be $a_\text{prior}$). Likewise, the "round" moon is actually boxed in a hexagon. Is the "horse with humps" sufficiently different from a camel? Horses don't have humps: is it fair to say that image is indeed a "horse"? Is the image of Paris with the Red Square really "Paris", or could it be "Moscow" with the Eiffel Tower? And so forth. I have looked at many examples and feel this issue is pervasive in this dataset.

3. The "Image DPO" method is not clearly motivated by the topic of "linguistic bias" that is explored in this paper. Why are the perturbations pixel-level and not linguistic? And, I am not sure that this benchmark demonstrates a particular need for more VQA training data. Altogether, the gains (1-3%) from the introduced methods are relatively small compared to the complex system involved (e.g. including image-editing and fine-tuning).

References:
- Winoground: Probing vision and language models for visio-linguistic compositionality. (Thrush, CVPR 2022)
- Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest (Hessel, ACL 2023)
- Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images (Guetta-Bitton, ICCV 2023)
- The Role of Chain-of-Thought in Complex Vision-Language Reasoning Task (Wu, 2023)

questions:
Could the authors please elaborate on how they constructed their dataset? In particular, it is not obvious how questions, answers, and images are specifically generated. E.g. what prompts are used for generating questions and images? I believe answers are entirely written by humans: is that correct? These are really important details that are unclear from my reading.

I am not sure that Sec. 3.1 needs to rely on mathematical formulations and the three objectives. I think this actually makes reading the following sections a bit confusing. Could the authors please clarify how these objectives specifically relate to the construction of the questions/answers/images (if at all)? E.g. Sec. 3.2 just says "we designed answers [for/against] language priors" but does not relate this to a particular objective.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730862441445

thread_type:
official_review

<thread_object>
creator: reviewer_dj1v

<content>
summary:
This paper introduces vVLM, a benchmark to address the issues of language biases in existing VQA benchmarks. vVLM consists of 300 questions, where each question is accompanied by a distractor fact and three image-answer pairs, where only one pair agrees with the distractor fact and the other two contradict the distractor. The intuition is that a VLM that is not guided by language priors should be able to ignore the distractor fact and answer. The authors show that existing SOTA VLMs such as GPT-4o struggle with this benchmark. To improve the performance of VLMs, the authors also propose a DPO-based method that self-generates VQA pairs for training using image generation and editing models, and then creates dispreferred pairs for DPO training by introducing pixel-level and semantic corruptions in the images.

strengths:
- Impressive data collection and curation effort for the benchmark, with multiple iterations of human filtering and human-AI collaboration. Manual creation of synonym set for the answers is a good
- Evaluated VLMs on both OOD and prior subsets separately, both with and without the distractor facts
- Good baselines for ImageDPO method, and also evaluated on multiple datasets

weaknesses:
- **The Image-DPO method is weakly motivated:** DPO is used to identify which is the preferred response out of two different responses for the same input, but the proposed method does the opposite — says which is the preferred input for the same output? Model is being trained to learn what is the preferred input, what is the intuition connecting this to the proposed goal (improving visual reasoning)?
    - repeat Text-DPO baseline, but with the generated images used for DPO as images for additional image-question-answer triplets (i.e. if you had a pair of triplets (Q, I, A_w) and (Q, I, A_l) for TextDPO, then for your perturbed images I’ with the blurring and pixelation perturbations, create a new triplet pair (Q, I’, A_w) and (Q, I’, A_l) ). My suspicion is that training with perturbed images at all is what is providing the additional performance benefit for your method, rather than the specific DPO objective you’re using.

- **Missing multiple seeds and significance testing:** The numbers in Tables 2 and 3 are the result of a single training run. Reporting the mean and standard deviation of 3 training runs would be much more informative about how reliable these results are.

- Language biases from internet-scale data (used to motivate the problem) is different from how language bias is operationalized here (distractor text included into the model context)

questions:
- Eq2 should have I as a conditioning variable, π(a_test | Q, I) ≥ δ
- Unclear how the alternate answers are selected; are they manually selected by the people designing the dataset?
- the acronym for VLM is expanded twice in L170
- What does vVLM stand for? It's never mentioned
- For the conclusion that images that were more challenging to generate were more difficult to answer for the VLMs (L464-469) — are there any numbers/tables pointing to the same? How is “easily generated” evaluated?
- Wrong citation for MMEC in L527: I think you meant this paper https://arxiv.org/pdf/2306.13394
</content>
</thread_object>

</thread>
<thread>
creation date:
1732339042105

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author response to all reviewers and AC

comment:
We would like to sincerely thank all the reviewers for their high-quality reviews. These reviews stand out as some of the most thoughtful and detailed the first author has received in a long time. As a professional courtesy, we have approached the rebuttal with the utmost seriousness and respect. While our responses are detailed, they include well-designed experiments and thoughtful rebuttals that address each concern comprehensively.

**To summarize the updates made during the rebuttal phase**:

- We compared our proposed benchmark with several other datasets, including Winoground (Thrush 2022 et al.), Whoop! (Bitton-Guetta 2023 et al.), and HallusionBench (Guan 2024 et al.). 
    - These comparisons are used to address potential reviewer misunderstandings and will be included in the appendix. We had related discussion in the related works. 
- We provided a theoretical proof demonstrating that our ImageDPO objective optimizes the upper bound of the RLHF objective. This encourages the model to effectively utilize its visual input branch by achieving gradient reversal when presented with different input images, while keeping the same input text and output answer [illustration figure](https://imgdrop.io/image/grad-diff.gZPlT). 
    - This proof is also included in the appendix, as it does not affect the objectives or experimental results presented in the paper.
- We improved ImageDPO performance by scaling the data.
    - The updated results are reflected in the tables in the main paper, but the overall pipeline remains unchanged. The only modification was the addition of more data generated by the same proposed pipeline.

Most updates during the rebuttal focused on clarifying misunderstandings and demonstrating improved ImageDPO performance. These revisions enhance the submission without altering its core contributions or the proposed benchmark dataset. All contributions remain consistent with the original submission, and for transparency, revised sections in the main paper are highlighted in red (we did not highlight some minor changes). 

We kindly seek the AC’s guidance in assessing whether our revisions constitute significant changes to the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733211510369

thread_type:
official_comment

<thread_object>
creator: reviewer_nay6

<content>
title:
Summary of reviewing thread [Nay6]

comment:
The thread under my review became quite long, so I will do my best to summarize the resolutions, my current viewpoints, and advice for the authors. I have read the full review thread and checked the main paper and appendices, once again. I will only address the *productive* discourse, i.e. that which helps further the evaluation of this work. I am not re-iterating "strengths" which can be seen in my original posting.

**Writing:** I believe the paper is in need of significant textual revisions. I appreciate the extensive additions to the appendices by the authors in revision. However, I believe the main paper needs re-writing; it should clearly cover all high-level details (whereas the appendices are for implementation-level details). One example is what a "semantic corruption" is — nearly all details are in the appendix, yet this is an important part of the proposed method and is even mentioned in the abstract. There was lots of back-and-forth in the reviewing thread about this. As another example, I sketched an outline (at the bottom of [this comment](https://openreview.net/forum?id=lCqNxBGPp5&noteId=xDoouAAKS6)) for what a clearer "dataset construction" section could look like. More examples of what is unclear may be found in the thread.

**Weaknesses:** The authors addressed Weakness (1) that I raised. In my view, the remaining weaknesses [(2) data quality and (3) relevance of the proposed method] are still not clearly resolved.

**(2) Data quality:** I mentioned the pervasive presence of visual distractors in my original review. One example is the "round" moon that is boxed in a hexagon (Figure 1). [In retrospect, I am realizing the flamingo example I originally highlighted may actually be fine. But I gave three more examples and continued to see this pattern through the data.] The authors pointed out that humans score well regardless of the distractors, which is good. However, the authors insist that *visual* distractors are part of the design of this dataset and are controlled in their studies. I really disagree with this claim. In the paper, I see many instances of *textual* distractors (i.e. "distractor facts") and evaluations of these settings (vVLM$^F$ and vVLM$^P$). As far as I can see, there is no commentary whatsoever about visual distractors (the writing should be updated if so). I do not see these being controlled for and I think these are significant confounders to the benchmark. This is actually a really fundamental issue and is the primary reason I consider this work less "sound".

**(3) ImageDPO:** I was concerned with the proposed method because of (a) complexity and (b) relevance. (a) In my original opinion, the method was overly complex relative to rather small gains. The authors scaled their experiment and the gains became larger, so this became a stylistic criticism from my point of view. (b) The research questions of this paper are about the linguistic biases of vision–language models. I was concerned that the proposed method is not motivated by this topic (and is more like a method that does well on the proposed benchmark). The authors insist that the "semantic corruptions" capture this. However, as above, I believe these are very underspecified in the main paper. And, even after revisiting appendices, I continue to find this unintuitive (now I have more (but not all) implementation-level details, like the prompts and the model used and no high-level details, about what semantic corruption is really supposed to do). I think the writing needs to be resolved, before this point can be revisited.

In all, I continue to vote to reject this paper for the above reasons, even after substantive discourse with the authors. I hope the authors will find my advice helpful in longer-term revisions to this work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces vVLM, a benchmark designed to evaluate vision-language models' (VLMs) reliance on language priors versus visual reasoning, along with a proposed ImageDPO training method to improve visual reasoning capabilities. The benchmark consists of 300 questions paired with three image-answer combinations each - one that aligns with language priors and two that require careful visual analysis. 

Initial reviewer opinions were mixed, with concerns raised about the relationship to existing benchmarks like Winoground, the quality and potential visual distractors in the dataset images, and the motivation and relatively small gains from the proposed ImageDPO method. Through the rebuttal period, the authors effectively addressed several concerns by demonstrating through quantitative analysis how their benchmark differs from existing ones in testing language priors and clarifying that visual complexity is an intentional design choice validated by high human performance. 

However, some reviewers remained concerned about the writing clarity, particularly around dataset construction details and objectives. There were also ongoing discussions about whether visual distractors should be controlled for in the evaluation. While the authors provided thorough responses and additional experiments, reviewers suggested the paper would benefit from significant revisions to improve clarity and better integrate the proposed method. The final consensus appears to lean negative, with reviewers acknowledging the interesting benchmark contribution but expressing reservations about presentation and methodological aspects.