MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models

Authors: Ziyu Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Conghui He, Yuanjun Xiong, Dahua Lin, Jiaqi Wang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Large Vision Language Models

Abstract:
Visual preference alignment involves training Large Vision-Language Models (LVLMs) to predict human preferences between visual inputs. This is typically achieved by using labeled datasets of chosen/rejected pairs and employing optimization algorithms like direct preference optimization (DPO).
Existing visual alignment methods, primarily designed for single-image scenarios, struggle to effectively handle the complexity of multi-image tasks due to the scarcity of diverse training data and the high cost of annotating chosen/rejected pairs.
We present Multi-Image Augmented Direct Preference Optimization (MIA-DPO), a visual preference alignment approach that effectively handles multi-image inputs.
MIA-DPO mitigates the scarcity of diverse multi-image training data by extending single-image data with unrelated images arranged in grid collages or pic-in-pic formats, significantly reducing the costs associated with multi-image data annotations.
Our observation reveals that attention values of LVLMs vary considerably across different images. We use attention values to identify and filter out rejected responses the model may have mistakenly focused on.
Our attention-aware selection for constructing the chosen/rejected pairs without relying on (i) human annotation, (ii) extra data, and (iii) external models or APIs.
MIA-DPO is compatible with various architectures and outperforms existing methods on five multi-image benchmarks, achieving an average performance boost of 3.0% on LLaVA-v1.5 and 4.3% on the recent InternLM-XC2.5.
Moreover, MIA-DPO has a minimal effect on the model's ability to understand single images.

Decision: accept (poster)
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730614382158

thread_type:
official_review

<thread_object>
creator: reviewer_dzhj

<content>
summary:
This paper proposes MIA-DPO, a method for improving the Large Vision Language Modle (LVLM) with multiple images. The main idea of MIA-DPO is finding the rejected prompts by using attention from pretrained LVLM and using the DPO training to fine-tune the LVLM for multiple image settings. Through the proposed MIA-DPO, the paper achieves better performance on multiple image settings than the baseline LVLM, without needing 1) The data for the multiple image setting and 2) external sources such as human/ChatGPT for creating feedback in the RLHF/RLAIF methods.

strengths:
- The motivation of this paper is clearly stated, which helps the reader a lot in understanding the idea.

- The proposed **Attention Aware Selection** can alleviate the problem of multiple image prompt datasets requirement and the need for external human/module in RLHF/RLAIF approaches.

- The experiments show the effectiveness of the method against the baselines.

weaknesses:
**Clarification**:
- Can the authors provide a more specific way to compute $A_{target}$ and $A_{sum}$ in L306? In L306-L307, the paper states that "$A_{target}$ be the amount of attention directed toward the target defined in $x$", however, as defined in L150, $x$ consists of $v$ and $t$, and as mentioned in Figure 2, only the attention on the given images $v$ is considered. 
- Which are the layers used to extract the attention $A_{target}$ and $A_{sum}$?
- In section 3.3.3, what are the parameters we need to optimize in the proposed method?

**Experiments**:
- In Table 1 and Table 2, why do the authors not include "RLHF", "HA-DPO" and "POVID" in InternLM-XC2.5?
- In Table 1, why LLaVA-RLHF, HA-DPO, and POVID are worse than LLaVA?

questions:
All the concerns and questions are listed in the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650908491

thread_type:
official_review

<thread_object>
creator: reviewer_zlyy

<content>
summary:
In this paper, the authors address the performance limitations of existing Large Vision Language Models(LVLMs) in multi-image tasks, identifying the difficulty of constructing multi-image preference data at low cost as a critical constraint. To tackle this issue, the authors propose Multi-Image Augmented Direct Preference Optimization (MIA-DPO), which can efficiently scale single-image training data to multi-image data without relying on human annotations or external APIs. In terms of data construction, MIA-DPO designs three expansion methods—Sequence, Grid Collage, and Pic-in-Pic to extend single-image data into multi-image version. Additionally, to address the common visual hallucinations of sequence confusion and element interference in  multi-image tasks, a hallucination detection mechanism based on attention values is designed for the constructed multi-image data. The authors conducted ablation experiments on LLaVA-v1.5 and InternLM-XC2.5, and the results demonstrate that the proposed MIA-DPO can enhance performance on multi-image tasks such as MVBENCH without compromising single-image metrics like MMMU.

strengths:
1. Research on visual hallucinations in large vision-language models is significant as it can improve the general capabilities of these models.
2. This paper presents an efficient pipeline for constructing and filtering multi-image preference data, which can generate large amounts of multi-image preference data at a low cost, making it highly applicable.
3. The paper analyzes the prevalent sequence confusion and element interference phenomena in large vision-language models for multi-image tasks from the perspective of attention, identifying that the key influential factor of these multi-image hallucinations is incorrect attention distribution.

weaknesses:
1. The authors mention at the beginning of the paper that existing multi-image SFT methods can degrade performance on single-image tasks while improving multi-image performance. However, they do not compare their approach with these methods in the experimental section.
2. Although the MIA-DPO method can easily convert data into SFT format, the authors do not compare the performance changes of the model after SFT using the same source data.
3. When evaluating the model's single-image performance, only MMMU is used, without assessing performance on other widely recognized benchmarks such as MMStar, MMBench, and OCRBench.
4. For evaluating the model's multi-image performance, the authors could use the Sequence, Grid Collage, and Pic-in-Pic methods to construct multi-image VQA test sets from existing single-image VQA data, directly assessing improvements in sequence confusion and element interference.
5. While handling more images is crucial for large vision-language models, the paper does not analyze whether MIA-DPO can still provide stable benefits when the number of simultaneously appearing images increases significantly (e.g., 10+ images).

questions:
1. How does the model's performance change on other single-image or multi-image metrics before and after applying MIA-DPO? (Refer to Weaknesses)
2. Can MIA-DPO still provide stable benefits when the number of images is expanded? (Refer to Weaknesses)
3. Can experiments be conducted on more advanced open-source large vision-language models or those explicitly supporting multi-images, such as the Qwen-VL series and Tarsier?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700206685

thread_type:
official_review

<thread_object>
creator: reviewer_mzdh

<content>
summary:
This paper introduces MIA-DPO, a visual preference alignment approach for LVLMs that handles multi-image tasks. The authors propose extending single-image data with unrelated images and using attention patterns to detect hallucinations, allowing construction of chosen/rejected pairs without human annotation or external APIs. The method shows promising results on both multi-image and single-image benchmarks.
Overall, I believe this is a solid paper that addresses an important and under-explored problem. It makes meaningful contributions to visual preference alignment for multi-image scenarios, though there are some aspects that need clarification.

strengths:
* While there's been significant work on visual preference alignment for single images, multi-image alignment remains largely unexplored despite its real-world importance. The authors identify key challenges like data scarcity and high annotation costs.
* Rather than collecting new multi-image data (which would be expensive), they augment existing single-image data through sequence, grid collage, and pic-in-pic formats. Using attention patterns to select rejected samples is an elegant way to avoid costly human annotation or API calls.
* The authors test on 5 multi-image benchmarks and 7 single-image ones, showing clear improvements (+3.0% on LLaVA-1.5, +4.3% on InternLM-XC2.5) while maintaining single-image performance. The ablations examining different components are thorough.

weaknesses:
However, I have several concerns that I think should be addressed:
My main concern is about the attention-based selection method. While using attention patterns to detect hallucinations is interesting, the threshold selection feels somewhat ad-hoc. The authors set different thresholds based on image counts (0.7/0.6/0.5/0.5 for 2/3/4/5 images), but there's limited justification for these specific values. I wonder if these thresholds could be learned automatically rather than manually set.
The relationship between augmentation strategies and model performance isn't fully explored. While the authors try three formats (sequence, grid collage, pic-in-pic), there's limited analysis of how different ratios of these formats affect results. Some discussion of which formats work better for what types of queries would be valuable.
The ablations, while informative, don't completely isolate the impact of different components. It would be interesting to see how model size affects the attention patterns and overall performance. Additionally, some analysis of failure cases would help understand the method's limitations.

questions:
Have you analyzed how sensitive the results are to the attention thresholds? What happens if you use uniform thresholds across different image counts?
For the post-selection filtering, how did you choose the specific metrics (perplexity, length ratio, edit distance)? Were other metrics considered?
Your method improves most on the Mantis benchmark (+11.1%). Any insights into why the gains are particularly large there?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730745257500

thread_type:
official_review

<thread_object>
creator: reviewer_epwn

<content>
summary:
The paper presents MIA-DPO, a multi-image DPO dataset construction approach for LVLMs without annotations from human or external models. The paper collects prompts from existing single-image dataset like LLaVA 665k, augmenting each data with unrelated images, which derives synthetic prompts of multiple images. Taking advantage of transformer's attention scores, rejected samples can be automatically filtered out to build pairwise preference answers for DPO. In addition, some post-selection methods are proposed for data cleaning. Experiment shows improvements on most multi-image and single-image benchmarks, and the capability to overcome hallucinations duo to the interference from unrelated images in prompts.

strengths:
- The paper is clearly written. Most details are provided, so it should be easy to reimplement. 

- The proposed method is well-motivated. Hallucination suppression is an important topic in LVLMs. The paper presents a straight-forward way to address hallucinations from unrelated images in multi-image scenario, which seems very easy to follow and does not rely on human annotations. 

- Experiments are well organized. Extensive evaluations show the effectiveness of the proposed method. An interesting phenomenon is, although the synthetic DPO prompts appear highly biased, improvements are still obtained on **general** single-image and multi-image evaluations.

weaknesses:
## Significance

My biggest concern is, the proposed synthetic method is designed to deal with the interference from unrelated images, however, which seems only a **corner case** in LVLMs. In practice, typically all images in the question prompt contribute to the final answer. It is a great challenge (but more important) to generate preference data for such *real* multi-image prompt at scale without much cost. The paper does not involve the topic. Therefore, I think the title "Multi-Image Augmented Direct Preference Optimization" may somewhat overclaim the contribution. 

To improve the work, I think the authors may defend their contributions by analyzing the root of the improvements on general benchmarks. For example, an interesting topic may be "does the proposed synthetic DPO dataset also help to suppress more general hallucinations other than those in Line 212?"; if yes, why?


## Experiment

I feel some empirical results are weak because: 
 
- According to Table 3, parts of the improvements are resulted from data cleaning (especially on MMMU and BLINK), which is not the unique technical contribution of the paper.

- In Eq 5, the NLL term plays a role of supervised fine-tuning with only chosen answers. It is an important baseline (i.e. fine-tuning with isolated NLL term) which can reflect the effectiveness of rejected samples, but not provided in the experiment.

questions:
* In Eq 4, please clarify how to calculate R(y) in details. Is the summation of attention values computed over all layers, attention heads and tokens subject to the given image/region?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732170146238

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal by Authors

comment:
Dear Reviewers,

Thank you for taking the time to review our manuscript and for providing thoughtful and constructive feedback. We are delighted to see that the reviewer acknowledged the paper is clearly written (ePWn), clear improvements and thorough ablations (mzdh), efficient pipeline (ZLYy), and clear motivation(DZHj). Below we summarize some major points we have addressed in the rebuttal. More detailed responses are provided individually for each reviewer.

**Image Coherence.** We analyzed the limitations of MIA-DPO and also conducted experiments to demonstrate that MIA-DPO can effectively handle the primary challenges of multi-image understanding. Furthermore, it achieves performance improvements on benchmarks that require understanding multiple related images simultaneously. (Reviewer ePWn)

**SFT Baseline(only the NLL term).** We added a baseline that directly uses the chosen data from MIA-DPO for SFT and compared it with other multi-image SFT methods. These comparisons validate the effectiveness of our approach. (Reviewer ePWn, ZLYy)

**Further Details on Calculating Attention Ratio.** To address the reviewer's concerns, we conducted a more detailed analysis of the formulas presented in the paper regarding the calculation of attention ratios.  (Reviewer ePWn, DZHj)

**Impact of Data Types and Ratios.** We further analyzed the impact of different data proportions, as well as the threshold settings and the results of the ablation experiments on thresholds.(Reviewer mzdh)

**Additional Experiments.** We supplemented our work with ablation experiments on model size to explore the performance of larger models. Furthermore, we added baselines for POVID and HA-DPO on the IXC2.5 model for comprehensive comparisons. (Reviewer mzdh, DZHj)

**VQA Test Set.** We constructed a 500-question multi-image VQA test set to evaluate model performance. Additionally, we created a subset consisting of 50 questions each for 4, 6, 8, and 10 images to explore the model's capabilities when handling a larger number of images. (Reviewer ZLYy)

In the point-to-point responses below, we have provided more detailed responses to each of the reviewers' concerns. ***All new material added to the revised manuscript has been highlighted in red text for better visibility.*** Please don't hesitate to let us know if there are any additional clarifications or experiments that we can offer！

Yours Sincerely,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732861982723

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
To All Reviewers:

I hope this message finds you well. We deeply appreciate the time and effort all reviewers have dedicated to our paper. However, we are embarrassed to say that we have not received any responses since the rebuttal period began, hence this reminder.

According to the ICLR 2025 Reviewer Guide, `The discussion phase at ICLR is different from most conferences in the AI/ML community. During this phase, reviewers, authors and area chairs engage in asynchronous discussion and authors are allowed to revise their submissions to address concerns that arise. It is crucial that you are actively engaged during this phase. Maintain a spirit of openness to changing your initial recommendation (either to a more positive or more negative) rating.`. 

This year, the ICLR organization community has made significant efforts to improve review quality. We sincerely thank PCs and ACs for sending reminder messages to reviewers who have not posted their responses. We sincerely thank the PCs that decided to extend the discussion period to October 2nd to facilitate more in-depth author-reviewer discussions. Several authors of this paper also served as reviewers. We diligently read rebuttals from other papers, ensuring that concerns were addressed and asking follow-up questions as needed.

We consider accepting a review invitation to be an honor that entails a responsibility to help the authors revise their manuscript.

We believe that as a reviewer, it is crucial to respond to rebuttal, regardless of whether you think our submission should be accepted or rejected. Even if our rebuttal does not alter your initial rating, it is important to communicate that you have carefully considered our comments.

We believe that even if you are very busy or on holiday, a short acknowledgment is at least necessary and will not take up too much of your time. In fact, being too busy to reply is not a reasonable excuse. During the around three weeks of discussion, we believe that PCs and ACs are often even busier than most reviewers. Despite this, they have made significant efforts to maintain the quality of author-reviewer discussions. We think all authors and reviewers should collaborate to be involved in the discussion period for our reputation and the benefit of the ICLR community.

We are committed to participating in the discussion period. **The discussion period deadline is approaching. We kindly want to follow up and ensure the reviewers provide your feedback.**

Regard,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper studies the problem of improving VLMs for image recognition. The authors propose to construct tasks where inspecting the model's attention on a sequence of images can be used as a signal to improve the model. The authors collect a preference dataset based on the synthetic data and then use DPO to improve the model. The resulting method is evaluated using models such as LLava and Intern-LM on standard image understanding benchmarks where it improves performance. 

Strengths
1. The paper proposes a simple idea of concatenating images into a sequence and use this multi-image task to get the VLMs to focus on the right image. The simplicity of this idea is a feature.
2. The experiments in this paper are well thought out and show that this work improves performance across multiple models and benchmarks.
3. The proposed method does not require creating a fresh dataset and shows that simple preference data collection can also improve performance.

Weaknesses
1. The method requires tuning the attention thresholds depending on the number of images. This parameter also seems to affect the final performance and makes the method brittle.
2. Studying the effect of multi-image DPO on larger models would be an interesting future direction since the effects of prior augmentation based training techniques typically diminishes with model size.


Justification
The paper proposes a simple, practically useful idea that is evaluated across multiple models and datasets. The primary concerns raised by the reviewers have been addressed by the authors.