MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models
Authors: Ziyu Liu
Yuhang Zang
Xiaoyi Dong
Pan Zhang
Yuhang Cao
Haodong Duan
Conghui He
Yuanjun Xiong
Dahua Lin
Jiaqi Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Vision Language Models
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Visual preference alignment involves training Large Vision-Language Models (LVLMs) to predict human preferences between visual inputs. This is typically achieved by using labeled datasets of chosen/rejected pairs and employing optimization algorithms like direct preference optimization (DPO).
Existing visual alignment methods, primarily designed for single-image scenarios, struggle to effectively handle the complexity of multi-image tasks due to the scarcity of diverse training data and the high cost of annotating chosen/rejected pairs.
We present Multi-Image Augmented Direct Preference Optimization (MIA-DPO), a visual preference alignment approach that effectively handles multi-image inputs.
MIA-DPO mitigates the scarcity of diverse multi-image training data by extending single-image data with unrelated images arranged in grid collages or pic-in-pic formats, significantly reducing the costs associated with multi-image data annotations.
Our observation reveals that attention values of LVLMs vary considerably across different images. We use attention values to identify and filter out rejected responses the model may have mistakenly focused on.
Our attention-aware selection for constructing the chosen/rejected pairs without relying on (i) human annotation, (ii) extra data, and (iii) external models or APIs.
MIA-DPO is compatible with various architectures and outperforms existing methods on five multi-image benchmarks, achieving an average performance boost of 3.0% on LLaVA-v1.5 and 4.3% on the recent InternLM-XC2.5.
Moreover, MIA-DPO has a minimal effect on the model's ability to understand single images.

Published as a conference paper at ICLR 2025
MIA-DPO: MULTI-IMAGE AUGMENTED DIRECT PREFERENCE OPTIMIZATION FOR LARGE VISION- LANGUAGE MODELS
Ziyu Liut.2, Yuhang Zang" Xiaoyi Dong", Pan Zhang", Yuhang Cao², Haodong Duan", Conghui He?, Yuanjun Xiong'. Dahua Lin2.3,6 Jiaqi Wang?.5 1 SITU, 2 Shanghai Al Laboratory, CUHK, MThreads, Inc, Shanghai Innovation Institute,
6 CPII under InnoHK
en, (zangyuhang, wangjiaqi J@pjlab. papilabborg.com org. Github: https:// bttps://github... wttpttwwwccmm//iiuuaaaaa.aaa///a
Suuziyu7essjtu...
ABSTRACT
Visual preference alignment involves training Large Vision-Language Models (LVLMs) to predict human preferences between visual inputs. This is typically achieved by using labeled datasets chosen/rejected pairs and employing op- timization algorithms like direct preference optimization (DPO). Existing visual alignment methods, primarily designed for single-image scenarios, struggle to ef- fectively handle the complexity of multi-image tasks due the scarcity of di- verse training data and the high cost of annotating chosen/rejected pairs. We present Multi- Augmented Direct Preference Optimization (MIA-DPO), a visual preference alignment approach that effectively handles multi-image in- puts. MIA-DPO mitigates the scarcity of diverse multi-image training data by extending single-image data with unrelated images arranged in grid collages or pic-in-pic formats, significantly reducing the costs associated with multi-image data annotations. Our observation reveals that attention values of LVLMs vary considerably across different images. We use attention values identify and filter out rejected responses the model may have mistakenly focused on. Our attention- aware selection for constructing the chosen/rejected pairs without relying on (i) human annotation, (ii) extra data, and (iii) external models APIs. MIA-DPO is compatible with various architectures and outperforms existing methods on five multi-image benchmarks, achieving an average performance boost of 3.0% LLaVA-v1.5 and 4.3% on the recent InternLM-XC2. Moreover, MIA-DPO has minimal effect the model's ability to understand single images.
INTRODUCTION
Recent progress in Large Vision Language Models (LVLMss) marks significant breakthrough in Al research. While proprietary models (e.g., GPT-4o (OpenAI, 2024)) excel at handling multi- image contexts, current open-source LVLMs (Liu et al., 2024b;a) yield promising results but are primarily focused on single-image visual question answering. In real-world environments, such as digital documents and web pages, multiple figures and texts are interleaved to convey complex information effectively. The ability to understand multi-image contexts is crucial direction for the future development of LVLMs.
LVLMs typically have three stages: (1) Pre- Tee-Trainnng (2) Supervised Fine- Tame-Tunig (SFT). and (3) Preference Alignment (i.e., Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) from AI Feedback (RLAIF) (Bai et al., 2022)). Correspondingly, to enhance the multi- image ability LVLMs, several recent multi-image pre-training (Awadalla et al., 2023a; 2024) and instruction fine-tuning (Jiang et al., 2024; Li et al., 2024a; Chen et al., 2024b; Liu et al., 2024d) datasets and evaluation benchmarks (Jiang et al., 2024; Fu et al., 2024: Song et al., 2024; Ma et al., 2024) have been proposed. Pre-training and SFT on multi-image data can enhance the model's ability to handle multiple images to some extent. Nevertheless, similar to ingle-image scenarios, hallucinations remain inevitable issue. Additionally, incorporating multi-image data during SFT

---

Published as a conference paper at ICLR 2025
Sompled Data (LLaVo668k)
UnislaUhrelated Images
Augmented Multi-Image Data
What
Ground Truthi There giraffes.
Sequence
Grid Collage
Pic-i--Pic
Prompt:
Single Imagelorg)
Imagel-IImage, Imoge2-Image.. Image3 -ineee Inageti-Insee What orinals are Image1?
MVBench
layer
Eratruction
gettod image
Riñi colcultiion
Ocear:
Imaget, thrr
Mcmppider SHARAA
Xieepisar2
Rejected
BLINK
NL VR2
LVLM
LVLM response: there thes two sheep.
ene ter degt and
Doer Lillt pay athertion
Months
MIA-DPO Overview
Benchmark Results
Figure I:
(a) Overview of MIA-DPO. We transform single-image data (e.g. LLaVA 665k) into multi -iati- data by adding noisy or unrelated images and using language descriptions to specify the target image. Attention values are then used to detect hallucinations in multi-image contexts, filtering out rejected data for DPO opti- mization. (b) Benchmark Results. MIA-DPO excels across five multi-image henchmarks while maintaining competitive performance on seven single-image benchmarks, demonstrating its robustness both single and multi-image tasks.
may adversely affect performance on single-image tasks. For example, previous work (Jiang al., 2024) shows strong results on multi-image tasks after multi-image SFT but suffers a 4% average drop in single-image benchmarks. Besides pre-training and SFT, another option preference align- ment. A series of Direct Preference Optimization (DPO) (Rafailov et al., 2024) approaches in the visual domain (Sun al., 2023; Yu et al., 2024a;b; Zhou al., 2024) have proven effective in miti- gating hallucinations in single-image scenarios. However, visual preference alignment remains little explored for multi-image fields.
Extending existing single-image preference alignment approaches to multi-image is non-trivial. A preference alignment data workflow consists of two key components: collecting question prompts, and selecting chosen/rejected response pairs. The transition to multi-image scenarios introduces the following challenges: (1) Limited Question Prompts. Multi-image image training data still emerging, with fewer instructions and less diversity than the extensive and varied inglle-maaee data. (2) High Construction Costs. Previous single-image ELHF/RLAIF approaches require high costs when con- structing chosen and rejected data pairs, such using human annotation (Sun al., 2023; Yu al., 2024a) or expensive GPT API (Zhao et al., 2023). Extending previous visual preference alignment data workflow to multi-image scenarios amplifies the associated costs.
To address the aforementioned challenges, we present multi-image visual preference alignment method, dubbed as Multi- Saati-mmag--Augmettte DPO (MIA-DPO). As shown in Fig. 1(a), to gather multi-image questions and answers, we extend single-image data to multi-image contexts by in- corporating unrelated images, and language description (e.g., in Imagel) to specify the target image. Additionally, we design three approaches to convert to data into multi-image format: se- quence, grid collage and pic-in-pic. This method uses existing single-image data, thereby reducing the costs associated with data collection and annotation, and is easily scalable for diverse data.
As for constructing chosen/rejected pairs, MIA-DPO eliminates the need for manual annotation or costly proprietary APIs. This is based our observation that when LVLMs process multiple images, the attention value distribution across different images varies significantly (see bottom left of Fig. 1). We perform Attention Aware Selection approach to filter out the rejected response that the attention values are mistakenly focused on irrelevant images. Our data construction method for DPO is also automated, cost-effective, and scalable for multi-image scenarios.
In summary, our key contributions are as follows:
(1) We first design multi-image visual alignment pipeline MIA-DPO. Our MIA-DPO requires no manual annotations and does not rely on APIs from larger models, offering a significant cost advantage compared to existing visual alignment approaches.
2

---

Published as a conference paper at ICLR 2025
(2) We contribute to the study of different types of multi-image hallucinations and propose to attention values as an indicator for detecting multi-image hallucinations.
(3) Extensive experiments (Fig. 1(b)) demonstrate that MIA-DPO is agnostic to different LVLM architectures (LLaVA-v1.5 (Liu et al., 2024a) and InternLM M-X2.55 (Zhang et al., 2024)), boosts the performance multiple multi-image benchmarks while maintaining the original single-image understanding capabilities.
RELATED WORKS
Large Vision Language Models (LVLMs), like GPT-4V (Achiam et al., 2023), signify a major breakthrough in the development of Large Language Models (LLMs) by incorporating both visual and textual data (Bai et al., 2023; Wang al., 2024a;b). LVLMs significantly enhance the quality of human-Al interactions, making these exchanges more intuitive and seamless. To enable LVLMs to handle multi-image tasks, several multi-image datasets suitable for pre-training and supervised fine-tuning (SFT) have gradually emerged (Jiang et al., 2024; Liu et al., 2024d; Song et al., 2024). However, due to the lag in the development of multi-image datasets, data and methods tailored for multi-image tasks during the RLHE/RLIIF phase remain unexplored. Therefore, designed dedicated MIA-DPO framework for multi-image tasks, aimed at improving the ability of LVLMs to handle multi-image scenarios.
Visual Preference Alignment is a multi-modal extension of preference alignment techniques with image inputs. Preference alignment aligns LLMs with human values and reduces hallucinations by collecting pairs of preferred and rejected data, using optimization techniques including PPO (Schul- man et al., 2017) and DPO (Rafailov et al., 2024) to guide the model's adjustments. Earlier ap- proaches, such as LLLaV--LLF (Sun et al., 2023) and RLHF-V (Yu et al., 2024a), required human labeling of preferred data, which incurs high labor costs. HA-DPO (Zhao et al., 2023) mitigates this by using GPT-4's API to generate the necessary DPO data, but it still faces high API costs. RLAIF-V (Yu et al., 2024b) employs a text-splitting approach to scoring individual text segments and using open-source LVLMs for data generation. POVID (Zhou et al., 2024) uses blurred images and GPT-4 to inject hallucinations to construct the DPO data. The previous approaches focus solely on single-image scenarios and require costly chosen/rejected data. Our MIA-DPO first enables vi- sual preference alignment for multi-image scenarios and achieves low-cost DPO data construction.
METHODS
We first introduce the background of visual preference alignment in Sec. 3.1. We analyze the multi- image hallucinations in Sec. 3.2. We present our MIA-DPO framework in Sec. 3.3.
3.1 PRELIMINARY
To enhance LVLMs' understanding multi-image inputs, we employ visual preference alignment. This section introduces the concept of visual preference alignment and highlights the DPO approach as a representative example.
Visual Preference Alignment Preference alignment aims to align model's preferences with human preferences. Representative approaches include Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) and Reinforcement Learning from AI Feedback (RLAIF) (Bai et al., 2022). Given a dataset D where each sample consists of an input prompt x, the chosen answers yuu and the rejected output yı- We represent 1 as follows: D (x., yhe 31 The input prompt x can be an interleaved sequence of images v and texts t. When an LVLM pro- cesses an input and generates an output y. a reward r(æ, y) is assigned. The reward model assesses the chosen (high value of r(x, y)) and rejected (low r(x, y)) samples. Visual preference alignment aims to maximize the reward r(x,
max E Ea- [r v)]
For simplicity, use single sample in our formulations, which can he easily extended to batch of samples.
3

---

Published as a conference paper at ICLR 2025
Imagei- Image> Imegei-Immge>
in ImageZ-Inage Inage3-Ingge's ImagetiInagere Image3h-Inger'w
What's s the mon doing Inagel?
Imagel, the stonding just observing troin.
possibly waiting for
Sequence Confusion
mistakerly idestifted Iraaged oddressing gaestion mainyy focuded en Inagef and Zraged nother than Tragel highlighting
The
this contusion.
Element Interference
Imagel -Imagerin Image2-IInage-nn Image2) The car in Image2 is red.
medel destified color mixing up the visual densils of the cur (Iraye with those the red natarcycle (Thige33 The aftertion
Image3 <IngeoI
distributedd
led interference.
Figure 2:
Examples of Iulti- Hallucinations. Top: Sequence Confusion that the model is confused about the order in which the images should be referenced. Bottom: Element Interference. The model incor- rectly identified the attributes due to visual element interference across different images. Attention values illustrate how the model's focus dispersed
different images, resulting in the hallucination response.
while 0, xy and <o(y)>) refer to the parameter, policy, and output distribution of LVLM, respectively.
To prevent over-fitting the dataset D, preference alignment approaches incorporate KL- divergence loss Dxr. to regularize the difference between the model's 's policy To (v(z)) and a reference model's 's policy mer(y)>)):
max E Ba-D.yrroou m) [r(z, v)1 - B DKL(To <Tr(y]l)))]
where the hyper-parameter 3 controls the influence of Kl--dvergeenc on the optimization objective. The reference model is the model's state prior to preference alignment.
Direct Preference Optimization (DPO) To optimize the preference alignment objective in Eq. (2), we can use either an online reward model (e.g., PPO (Schulman et al., 2017)) or pre- computed off-line chosen/rejected pairs (e.g., DPO (Rafailov et al., 2024)). Given its simplicity, DPO has been widely adopted in previous visual alignment works (Sun et al., 2023; Yu et al., 2024a; Zhao et al., 2023; Yu al., 2024b). We reformulate Eq. (2) as the loss function of DPO:
(Blog (1(vl))))) BBlog (ve((yl)r))
To (2r) Tee (y1 xx) )}
(Tro Ret)) (a. )~D
log
(3)
where (.) denotes the sigmoid function. As shown in Eq. (3), DPO-based alignment methods focus on constructing input prompts x (see Sec. 3.3.1), and selecting chosen yo and rejected yı pairs (see Sec. 3.3.2).
3.2 ANALYSIS ON MULTI-IMAGE HALLUCINATIONS
In this section, conduct various studies to analyze the characteristics of multi-image hallucina- tions in LVLMs and reveal that the attention mechanism is a proper indicator to determine when hallucinations occur.
Two-types of Multi-Image Hallucinations Some previous studies (Li al., 2023c; Ouali al., 2024) have explored different types of single-image hallucinations, such as object hallucination which means the model incorrectly describes objects that present in the image. Compared to
4

---

Published as a conference paper at ICLR 2025
MIA-DFO ChosendRejected Pairs
Past-Selection
Mrw.errpoocyy Bush To popiiauc
Prompt: mmggeh-emoge.. TraggEsueeggo ga2, Ece castion...
Selected Item
(lova6661)
Netry/ Uttellodd
natis: lersttggt claan't
Sequeue
Grid College Pic-in-Pic
Tmoges Types
Trogebileage TrgeZziinge Treege:: mmsper tragel-mage
Chesen:
o few pencis country
DFO
Edit Distance
eammerce capt For pronded irage.
Rejectedi A bottic country club bear.
Chacke of
Truahh Fer pencia with countryy
Rulls attention selection
LVLM
Generate
LVLM attiention awaree? Ratto/31g13/Rericct0)))
Original
Imagel
Imagez
Image:
Imagee
Resporse:
buttlcc countryy
boor
Ground
Figure 3: MIA-DPO Framework. We extend the single-image dataset multi-image datasets by inserting irrelevant images and using attention values to filter out the hallucination responses for rejected samples of the DPO algorithm.
single-image hallucinations, multi-image scenarios introduce more complex types of hallucinations. As shown in Fig. 2, we categorize multi-image hallucinations into two-types:
(1) Sequence Confusion. When presented with multiple images, the model may fail to identify which image the imput prompt refers to. For instance, in the top case shown in Fig. the question is directed at Image 3 (birds and sky), but the model responds based on Image 4 (a train on tracks).
(2) Element Interference. The presence of multiple images significantly increases the number of visual elements compared to a single image, leading to confusion between different elements by LVLMs. For example, in the bottom case of Fig. 2, the question "What color is the car in Image2?" should be answered with "white". However, the LVLM incorrectly interpreted the color attribute of the motorcycle Image as the color of the car in Image 2, resulting in an incorrect response.
Attention an Indicator for Detecting Hallucinations The attention mechanism reveals where the model is "looking" when making decision. We observe that the attention mechanism provides crucial clues for detecting multi-image hallucinations (Fig. 2). Ideally, attention values should focus on areas of the referred input image relevant to the question. If the attention values are scattered or not strongly focused on the correct visual element or region, suggests the model is experi- encing difficulty understanding multi-image sequences or distinguishing elements between different images. Based on our observation, we design an attention-awan selection that uses the attention val- ues to select the rejected sample that contains the hallucinations in the DPO algorithm (Sec. 3.3.2).
3.3 MIA-DPO FRAMEWORK
As illustrated in Fig. 3, MIA-DPO initially extends single-image prompts to multi-image prompts (Sec. 3.3.1), followed by attention-based filtering of rejected data and post-selection processing (Sec. 3.3.2). Finally, we apply the DPO algorithm (Sec. 3.3.3) to the constructed multi-image prompts and chosen/rejected pairs, resulting in a stronger model.
3.3.1 FROM SINGLF- PROMPTS MULTI-IMAGE PROMPTS
Rather than expending effort on collecting and annotating new multi multi-image prompts, we efficiently convert existing single-image datasets, such as LLaVA-665k (Liu et al., 2024a), by incorporating unrelated images. Our low-cost, scalable approach enriches data forms and allows us to comprehen- sively explore the various types of multi-image hallucinations that LVLMs might produce.
As shown in Fig. 4, we construct multi-image prompts in three formats: (I) Sequence: Multiple images are arranged sequentially, with questions targeting specific images. The number of images varies from 2 to 5. (2) Grid Collage: Multiple images are merged into a single image, each labeled with number description. Questions focus specific images based language descriptions. The number of images ranges from 2 to 9. (3) Pic-in-Pic: One image is resized and overlaid onto another, and questions are asked about the combined image.
5

---

Published as a conference paper at ICLR 2025
(a) Sequence
Teege bregelsamgggpiccheeeee Tranga deoge deoge-plooehttoe plooeeekeeee prowde enn-sentince for proviced Tengel:iingppccccheeee Tragskinnge-pooeeleee Twege--mge-peechhhle
Chesen: bomer Barne
Welland Centre.
Designed To oddress the istue of Sequerce Conlusion.
(b) Grid Collage
Pic Dec-in-pic
and input. Designed addiness tstue Elenont Dinerference
Collage Date: Merrge the indges into one and input. Designed
Prongt: sirage pioceholder Reger diny the imsge. a gaine being played
Blement Interfenence
Chezen: Yos, nomon playing gama.
Proept: vinge bbochskler Is Irage2, what typ structane antaall cornying people?
Rejected: No. there no game being played in mase imag
starding telkeng bie with partted piant on
Chesee:
carned large weeden elephant atnucture. Rejected: Emeget, there i lorgs elephent carrying people.
The
Figure 4: Multi- Images DPO Data Format. To address multi multi-image hallucinations mentioned in Fig. 2, we construct multi-iman image prompts in three formats: (a) Sequence. (b) Grid Collage. (c) Pic-in-Pic.
lmnger
Singgs
Iinget Tayeh
Incget
230
250
106
Numbeer
Nomber
100
30
Ratio
Ratio
Ratão
(a)
Sequence
(b) Grid Collage
(c)
Pic-in-pic
Figure 5:
Attention Ratio Statistic. We analyze the attention ratios distribution for different image
counts across various data types, and use dashed lines to indicate the thresholds for each data set.
These three data types are specifically designed to address the two types of multiple-image halluci- nations in Fig. 2. Sequence data extends the overall length of image tokens and introduces multiple unrelated images to confuse the LVLMs, challenging their ability to determine image order (Se- quence Confusion in Fig. 2). Grid Collage and Pic-in-Pic data stack multiple images, increasing the likelihood of LVLMSs confusing image elements and failing to accurately locate the content based on language descriptions (Element Interference in Fig. 2). Our diverse multi-image prompts enhance data richness, address various types of multi-image hallucinations, and provide a strong foundation for constructing chosen/rejected pairs.
3.3.2 ATTENTION-AWAR SELECTION FOR REJECTED SAMPLES
As we analyzed in Sec. 3.2, the model's attention values are clues for detecting multi-image hal- lucinations. Inspired by our observation, we present attention-aware selection mechanism for constructing the rejected samples of the DPO algorithm.
Given the input question x and set of generated answers (91, 92....) ~ To(y) For each answer Auge sample y, we compute the attention value metric R(y) = Aun where Aurget be the amount of attention directed toward the target defined in x, and Apoul be the total amount of attention values. By setting attention ratio threshold т, focus on the image or region specified:
select cases yr that the LVLMS did not correctly
{v To(y)>) and R(y) ST).
y
We use y as the rejected answer for the DPO algorithm. We use the ground truth of question rr as the chosen sample too Finally, construct the DPO pair data D {2r, yo v1 in Eq. (3).

---

Published as a conference paper at ICLR 2025
Determining the Ratio Threshold The sequence data include sets of 2, 3, 4, and 5 images. For each set, we calculated the proportion of attention focused on the image relevant to the question or instruction, relative to the total attention actoss all input images. Our statistical results are visualized in Figure S(a). As shown in Fig. 5(a), the average attention ratio decreases as the number of images increases, though the overall distribution trend remains consistent. Based attention ratio thresholds at 0.7, 0.6, 0.5, and 0.5 for sets of 2, 3, 4, and 5 images, respectively. Data below the thresholds marked as rejected. We applied the same statistical approach for grid collage data and pic-in-pic data, and visualized the results in Fig. 5(b) and 5(c). For grid collage image data with 2, 3, 4, and 9 images, we set the value of as 0.7, 0.6, 0.5, 0.4, and 0.4, respectively. For pic-in-pic data, we set 0.6.
findings, we set
Post-Selection for Data Cleaning Although our attention-aware selection is effective in con- structing the DPO data, small amount of noisy samples may be included and potentially causing detrimental effects. To filter out the noisy samples, we incorporate a post-selection step using the following three metrics: (1) Perplexity (PPL). The PPL metric measures the negative -ig-liillhood of the generated sequence. and is a common metric for data cleaning (Albalak et al., 2024). A high PPL value suggests that LVLMs have lower confidence and more likely contain hallucina- tions. We use the PPI. metric to filter out low confidence responses. (2) Length Ratio. Previous studies (Singhal et al., 2023; Dubois al., 2024) have shown that the reward model may favor lengthier content. To mitigate the length bias, we compute the length difference between the chosen and rejected data, excluding the samples where the difference value is too large. (3) Edit Distance. We observed that some samples may not contribute meaningfully to the optimization process. For example, the difference between "apple" (chosen) and "apples" (rejected) is minimal in terms of edit distance, which is less useful for distinguishing patterns. We use the edit distance to ensure the DPO process does not incorporate pairs with excessively small differences.
The post-selection approach will filter out approximately 5% of the data. We provide the ablation studies in Sec. 4.4 to demonstrate that our post-selection helps maintain the high quality of data.
3.3.3 OPTIMIZATION
As discussed in Sec. 3.3.1 and Sec. 3.3.2, we have outlined how construct multi-image input prompts and select chosen yo and rejected y1 pairs. By applying Eq. (3), we can update the policy ne.
To improve the stability of DPO training. following the approach in (Dubey et al., 2024; Pang et al., 2024), we add a negative log- (N. loss LNLL ) = log wo (Jwll))) We use a parameter y to balance the LDPO and LNLL. The final loss Lisua is defined in Eq. (5):
Liotll = LDOO (T; Teet) + (
(5)
EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Benchmarks We evaluate our method on the following representative benchmarks. First, we lect five multi-image benchmarks: MMMU (Yue et al., 2024), BLINK (Fu et al., 2024), Man- tis (Jiang al., 2024), NLVR2 (Suhr et al., 2018), and MVBench (Li al., 2024c). The MMMU benchmark includes questions involving both single-image and multi-image scenarios. Subse- quently, we also test the model on several single-image benchmarks: MMStar (Chen et al., 2024a), ScienceQA (Lu et al., 2022), MMVet (Yu et al., 2023), POPE (Li et al., 2023c), MMBench (Liu al., 2023), Math Mith (Lu et al., 2023), A12D (Kembhavi et al., 2016), and OCRBench (Liu al., 2024c). We evaluate our method on a diverse set of benchmarks, demonstrating its effectiveness across both scenarios. These evaluations confirm the model's 's improved performance, particularly in multi-image contexts.
We compare MIA-DPO with three preference optimization baselines. (1) LLa -LHF (Sun et al. 2023) improves model performance by augmenting GPT-4-generated data with existing human-written written image-text (2ta. HA-DPO (Zhao et al., 2023) uses GPT-4 to detect and correct hallucinations in the model's responses. (3) POVID (Zhou al., 2024) prompts GPT-4V
Baseline Methods

---

Published as a conference paper at ICLR 2025
Table 1: Main results multi-image benchmarks. We compare our MIA-DPO along with other DPO algorithms across five multi-image benchmarks. Our method brings significant performance improvements to both the classic LLLVV-vv1.5 55 and the recent MttemI.--XC.., In contrast, other single-image DPO methods perform poorly on multi-image benchmarks.
Models
Parameter MMMU BLINK Mantis NLVR2 MVBench Average
GPT-4V (Achiam et al., 2023)
56.8
51.1
62.7
88.8
43.5
60.6
LLaVA-v (Li i el al,, 2024h)
7B
35.8
39.6
45.6
58.9
40.9
44,2
Qwen-VL-Chan VL-Chat (Bai et al., 2023)
7B
35.9
31.2
39.2
58.7
42.2
41.4
LaVA (Lin et al., 2023)
38.9
56.5
44.3
Fuyu (Bavishi al., 2023)
27.9
27.2
51.1
30.2
34.6
Idefies2 (Laurençon
2024b)
8B
43.0
45.2
48.9
86.9
29.7
50.7
InstructBLIP (Dai al., 2023)
13B
30.6
42.2
45.6
60.3
32.5
42.2
C (Wang al., 2023)
17B
32.1
41.5
45.2
58.6
37.3
42,9
Ennu2-Caal (Sun et al., 2024)
37B
36.3
36.2
37.8
58.2
39.7
41.6
LL (Liu et al., 2024a)
7B
35.1
37.1
41.9
52.1
36.0
40.4
+ LLaVA-RLHF (Sun et al., 2023)
7B
34.6
40.8
30.4
51.8
38.0
39.1
HA-DPO (Zhao et al., 2023)
35.8
34.6
40.2
POVID (Zhou 2024)
35.2
37.8
21.4
39.4
30.7
MIA-DPO (Ours)
7B
36.3
42,9
44.2
54,2
39.5
43,4
+1.2
+5.8
+2.3
+2.1
+3.5
+3.0
Internl M-XC2.5 (Zhang al., 2024)
41.4
46.9
49.3
70.7
59.5
53.6
HA-DPO (Zhao et al., 2023)
7B
42.0
46.9
51.6
71.6
58.0
$4.0
POVID (Zhou et al., 2024)
7B
42.4
47.9
51.2
70.6
59.2
54.3
MIA-DPO (Ours)
42.6
47.7
60.4
75.2
63.6
57.9
4.1
+4.3
to inject plausible hallucinations into correct answers, followed by image distortion to provoke the LVLMs' inherent tendency towards hallucinations.
Implementation Details Our MIA-DPO is applicable to various LVLMs. We select two models in our experiments: the classic L.L. VA-v1.5 (Liu al., 2024a) and the recent MMMer2l.M-X2..55 (Zhang et al., 2024). The models eter (in Eq. 3) / = 0.1, and NLL loss coefficient (in Eq. 5) 7 = 0.1. For more experimental details, please refer appendix Sec. A.
trained on 3 epochs, with learning rate of 5e-5, temperature param-
4.2 RESULTS ON MULTI-IMAGES BENCIMARKSS
Results on VLaVA-v As present in Tab. 1, applying MIA-DPO to LLaVA-v1.5 achieves im- on five multi-image benchmarks, which demonstrates the effectiveness of MIA-DPO. As for the challenging MMMU benchmark that requires complex domain-specifie knowledge, MIA-DPO enables LLaVA-v1.5 to achieve a 1.2% improvement. The experimental results MMMU demonstrate that MIA-DPO enhances the LLa reasoning ability on multi-image problems. Additionally, on the BLINK dataset that includes multi-view and spatial relationship reasoning, MIA-DPO significantly boosts the performance LLaVA-v1.5 by 5.8%. Such an improvement highlights the effectiveness of MIA-DPO in enhancing the model's ability understand and reason under multi-image scenarios.
provements of 1.2%/5.8% 2tt.//3%//..1222
Comparison with Preference Optimization Baselines In Tab. 1, we compare MIA-DPO with three preference optimization baselines (LLaVA-RLHE, HA-DPO, POVID) LLaVA-v1.5. Thanks to our multi-image attention- based method for constructing the DPO data, MIA-DPO achieves sig- nificant advantages on the reported five multi-image benchmarks compared the baselines.
More LVLM Architectures We also applied MIA-DPO to other LVLM architectures, such as the recent Inte2 model. As shown Tab. 1, MIA-DPO boosts the performance of 1.2%/0.88 8.11. 1tt./...1//..///// 11..4%%%/// 5%%4.1% across the five benchmarks, resulting in average improvement of demonstrate that MIA-DPO is general and
4.3%. The results on LLaVA-1.5 and InternL--XCOS effective for different LVLM architectures. Notably, despite the Supervised Fine-tuning (SFT) phase of InternL--XC2 involving multi-ime multi-image data, our MIA-DPO still further boosts performance on multi-image benchmarks.
8

---

Published as a conference paper at ICLR 2025
Main results single-image benchmarks. We compare MIA-DPO with other DPO approaches across seven single-image henchmarks. MIA-DPO, which not only enhances multi-image performance but also maintains strong proficiency in single-image tasks.
Table 2:
Models
Parameter MMStar SQA MMVet POPE MMB Math A12D
OCR Average
LLaVA-v1 (Li ct al,, 2024b)
7B
37.6
87.5
40.2
70.3
69.8
31.5
67.0
53.7
57.2
Qwen-VI. Vh.-Chat (Bai et al., 2023)
7B
34.5
68.8
47.3
74.9
61.8
15.5
63.0
48.8
51.8
Idefics2 (Laurençon ct al., 2024b)
8B
49.5
88.7
34.0
86.2
75.7
51.4
72.3
OpenFlamingo (Awadalla ct al., 2023b)
9B
36.9
44.8 23.2
52.6
32.4
18.6
31.7
14.9
31.9
InstructBLIP (Dai et al., 2023)
13B
32.7
54.1 33.1
86.1
38.3
24.4
40.6
27.6
42.1
Cog CLM (Wang et al., 2023)
17B
39.9
66.2 54.5
88.0
65.8
35.0
63.3
59.0
59.0
Emu2-Chat (Sun et al., 2024)
37B
40.7
68.2 31.0
88.0
63.4
30.7
49.7
43.6
51.9
LLaVA-v1.5 (Liu et al., 2024a)
7B
32,9
66.6 30.5
85.9
64.3
25.4
55.5
31.8
49.1
LLaVA-RLHP Sun et al. (2023)
7B
31.6
64.0 27.8
80.8
60.1
23.5
47.9
28.0
45.5
HA-DPO (Zhao et al., 2023)
7B
33.5
67.3 29.1
84.3
64.9
25.8
53.9
31.1
48.7
POVID (Zhou et al., 2024)
7B
36,2
68.8 31.8
86.3
64.9
24.4
55.2
31.6
49.9
MIA-DPO (ours)
7B
32,9
67.6 32.1
87.2
63.1
24.4
54.7
30.5
49.1
IntemLM-XC2255 (Zhang et 2024)
7B
59.7
96.3 48.7
87.9
81.9
63.3
81.5
69.0
73.5
HA-DPO (Zhao ul., 2023)
7B
59.6
96.2 53.3
84.4
81.4
62.6
81.8
68.2
73.4
POVID (Zhou al., 2024)
7B
59.7
96.2 54.8
88.1
81.4
62.6
81.3
68.9
74.1
MIA-DPO (ours)
7B
61.1
96.2 46.7
86.9
80.4
61.7
81.6
67.4
72.8
4.3 RESULTS SINGLE-IMAGES BENCHIMARKS
While MIA-DPO is effective in multi- image scenarios, also report the performance single- image benchmarks. As shown in Tab. 2, MIA-DPO outperforms the LLaVA-v1.5 baseline and DPO methods, including LLaVA-RLHF and HA-DPO, in average results across seven single-image benchmarks. As for the InternL.-XX22.5 model, MIA-DPO achieves 1.4% increase on MMStar but performs slightly below baseline on average across all single-image benchmarks. The slight degradation in InternLM-XC2.55
single-image performance suggests that while the model benefits greatly multi-ine multi-image scenarios, there may be trade-off in optimizing for more complex, inter- leaved inputs. Overall, our findings highlight the robustness of our MIA-DPO, which not only excels in improving multi- multi-image performance but also preserves proficiency on single-image tasks. Our MIA-DPO serves as a strong candidate for real-world applications requiring versatile multi- modal abilities across both single and multiple image tasks.
4.4 ABLATION STUDIES
Ablation Studies on Post-Selection In our ablation study, we experimented with the post- selection process for DPO data. illustrated in Fig. 3, our post-selection process includes three components: perplexity (ppl), text length, and edit distance. We conduct ablation studies to compare the impact of whether to use the post-selection or not. In Tab. 3, the results show that while MIA- DPO without post-selection (row 1) still led improvements across multiple multi-image bench- marks, its performance was consistently lower than that MIA-DPO with post-selection (row 2). Our findings highlight that post-selection effectively removes outlier and low-quality data, further enhancing the overall quality of the DPO pair data and boosting model performance.
Ablation Studies on Data Types In the process of constructing multi-image DPO data for MIA- DPO, we created three types of data: Sequence, Grid Collage, and Pic-in-Pic Data. These three types of data work together to specifically eliminate the two types of multi-image hallucinations we identified: Sequence Confusion and Flement Interference. To study the impact of each data type on overall performance, trained the LLaVa-v1.5 model separately with 20k instances of each data type and summarized the results in Tab.
The experimental results indicate that using each data type individually for DPO on LLa yields similar average scores of 42 42.4, and 42.7 across five benchmarks. However, when com- bining all three data types, the model achieves higher average score of 43.4, shown in Tab. 1. This suggests that the three data types address different hallucination types, and their combination produces better results than using them separately.

---

Published as a conference paper at ICLR 2025
MMMU
BLINK
Mantis
NLVR2
MVBench Average
Ablation Studies. The top row refers to LLaVA-
Table 3:
35.1
37.1
41.9
52.1
36.0
40.4
w/o post sel.
35.3
38.7
44.2
53.7
39.4
42.3
v1.5 baseline. We conduct ex- periments about the impact of without (w/o) and with (w) post-
sel.
36.3
42.9
44.2
54.2
39.5
43.4
sequence
37.3
39.5
44.2
51.7
40.1
42.6
grid collage
37.1
40.4
44.2
51.0
39.4
42.4
selection techniques and dpo data types.
pic-in-pic
37.9
40.8
41.9
53.2
39.8
42.7
Inputs
Inputs
Inputs
Prompt: <imoge.
Image1,
Prompt: Imogelidmago. Inoge2-inago In Image1 what woother condition the mage?
Inagel-Tnap Imaye2 Innge, Imaye3 <Inugo What the
Original
MIA MIA-OPO -PO
Delta
Image
image 2
image
image
image
image 2
image
Figure 6: Attention Difference Before and After DPO. present the attention distribution the interme- diate layers for the original LLaVA-v1.5 (top row), MIA-DPO + LLaVA-v1.5 (second row), and the diffcrence value (bottom row), respectively.
4.5 VISUALIZATION OBSERVATIONS
We visualize the reasoning process of the LLaVA-v1.5 model before and after applying MIA-DPO on multi-image cases. In Fig. 6, we show the attention map of the generated text tokens relative to the input image tokens. The top and second rows display the attention distribution before and after applying MIA-DPO, respectively. The attention difference (delta value) in the third row indicates which areas receive increased attention due to applying our preference optimization process.
Using MIA-DPO, the LLaVA-v1.5 model adjusts its focus to specific image regions corresponding to the given instruction. both the first and second cases, we observe an increased focus on the instruction-targeted areas of Image after applying MIA-DPO. In the third case, attention gravitates more toward Image 2, which is specified in the language instruction. The visualization results in- dicate that MIA-DPO effectively improves the model's ability to correctly allocate attention to the relevant image regions, reducing the likelihood of multi-ime multi-image hallucinations.
5 CONCLUSION
Aligning models with human preferences is a critical goal. In this paper, we are the first to pro- pose multi-image DPO framework. We conducted an in-depth analysis of the differences be- tween hallucinations in multi-image and single-image reasoning for LVLMs, exploring the root causes of multi-image hallucinations through the lens of attention. Our findings reveal that a lack of attention-aware capabilities is key factor contributing to hallucinations in multi-image reason- ing. Based on these insights, we introduced MIA-DPO (Multi-Image Augmented Direct Preference Optimization). Results from tests five multi-image benchmarks and seven single-ima -mage bench- marks demonstrate that MIA-DPO significantly improves the model's performance imate-immae reasoning while maintaining its original single-image reasoning capabilities.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments
This project is funded in part by Shanghai Artificial Intelligence Laboratory. Shanghai Innovation Institute, the National Key R&D Program of China (2022D01660201), the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK. Dahua Lin is PI of CPII under the InnoHK.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08777 2023.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang. Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint arXiv:2402.1682 2024.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. OpenFlamingo: open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 000000900 2023a.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01399 2023b.
Anas Awadalla, Le Xue, Oscar Lo, Manli Shu, Hannah Lee, Etash Kumar Guha, Matt Jordan, Sheng Shen, Mohamed Awadalla, Silvio Savarese, et al. MINT-IT: Scaling open-source multimodal data by 10x: multimodal dataset with one trillion tokens. arXiv preprini arXiv:2406. 1271, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen- VL: A frontier large vision-language model with versatile abili- ties. arXiv preprint 2...v::2008.126666 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional AI: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077, 2022.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are on the right way for evaluating large vision-language 2024a.
models? arXiv preprint arXiv:2403.203:
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and genera- tion with better captions. arXiv preprint 240.::22200004225., 2024b.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision- language models with instruction tuning, 2023.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. arXiv preprint arXiv:2407.1 / 1691, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
11

---

Published as a conference paper at ICLR 2025
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint WwXiv:2404.044755 2024.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can but not perceive, 2024.
Drew Hudson and Christopher D Manning. GQA: A new dataset for real- world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6700-6709, 2019.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. MANTIS: Interleaved multi-image instruction tuning, 2024.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. diagram is worth a dozen images. In ECCV, 2016.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 2024a.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.022 2024b.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. 2023a.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. LLaVA-NeXT-Imterlaaw Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.0755 2024a.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleavee Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint N10.i::2407.009555 2024b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023b.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. MVBench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024c.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing. 2023c.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united
visual representation by alignment before projection. arXiv preprint arXiv:2311.1012 2023.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. pre-training for visual language models. In CVPR, 2024.
VILA: On
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr
Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. MMBench: Is your modti-modal model an all-around player? arXiv preprint arXiv:2307.06228. 2023.
12

---

Published as a conference paper at ICLR 2025
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. OCRBench: On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07899, 2024c.
Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, et al. Mmdu: A multi-turn multi-image dialog understanding benchmark and pnrruction-tuning dataset for Ivims. arXiv preprint arXiv:2406. 11833, 2024d.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song-Chun Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hanmanch Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Vista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma. Xiaoyi Dong, et al. MMLongBench- Doc: Benchmarking long-context document understanding with visualizations. arXiv preprint arXiv:2407.01523, 2024.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirhan Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019.
OpenAl. Hello gpt-4o, 2024. URL https: oppnaiienaa........oo.... com/ wwwte/ewweweeeeeeeeeeeeeeeeeeeet potoooppp-4////
Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. CLIP-DPO: Vision-
language models source of preference for fixing hallucinations in Ivlms. In ECCV, 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishknn, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404. 20..:2344419933. 2024.
Zhiliang Peng, Wenhui Wang, Li Dong. Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (141:0/777.66445 2017.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023.
Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Benchmarking milms in long context. arXiv preprint arXiv:2404.18532, 2024.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv: 1.00491, 2018.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In CVPR, 2024.
13

---

Published as a conference paper at ICLR 2025
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint 23X1:2300.115233 2023.
Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayi- heng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-VL: Enhancing vision-language model's perception the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. CogVLM: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03073 2023.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. RIHF-V: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In CVPR, 2024a.
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang. Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen Hc, Zhiyuan Liu, Tat-Seng Chua, et al. RLAIF-V: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.1722 2024b.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. MM-Vet; Evaluating large multimodal models for integrated capabilities. arXiv preprint ar0iv:2308. 02490, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, al. Aiternlm-xcomposer2222 A versatile large vision language model supporting long-contextual input and output. arXiv preprint urXiv:2407.033200 2024.
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hal- lucinations: Enhancing Ivlms through ballucination-aww aware direct preference optimization. arXiv preprint arXiv:2311 1..8192311.16899 16839, 2023.
Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.1 11411, 2024.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
In this appendix, we provide additional supporting materials to facilitate deeper understanding of work. First, in Sec. A, we further enrich the experiments, including ablation studies and experimental details. In Sec. B, we list all the models and benchmarks we used, along with statistical overview of the amount and ratio of data utilized MIA-DPO. In Sec. C. we present more examples of the three data types: Sequence Data, Grid Collage Data, and Pic-in-Pic Data. In Sec. D, we share our observations on the attention distribution in LVLMS multi-image reasoning, explaining the basis for attention aware selection.
A MORE EXPERIMENTS
A.1 ABLATION STUDIES
Ablation Studies on y and Epochs We perform ablation studies on the key hyper- asper-parameters including the NLL loss coefficient & and the number of training epochs. As shown in Tab. 4, we observe that larger value of negatively impacts the training process, while the number of epochs has minor effect on the final results. Based on the experimental results, we set epochs and 7 0.1 as the default values for the parameters.
GPT-44-mini Selection and MIA-DPO To validate the effectiveness of MIA-DPO, we introduce an ablation experiment using GPT-44-mini for DPO data selection. The process begins with the model generating answers to our multi-image questions, followed by presenting both the model's responses and the ground truth to GPT-4o-mini. GPT-4o-mini then assesses the accuracy of the model's 's responses and their similarity to the ground truth, assigning score between 0 and 10 based on various criteria. We classify responses with scores below construct the DPO data. The results are presented in Tab. 5. Our observations indicate that MIA- DPO not only offers a cost advantage over the GPT-4o mini-maseeeeee data selection method but also outperforms it across five benchmarks.
rejected data and use them to
The prompt we use to guide GPT-4o-mini in data selection is as follows:
Assume you are expert in evaluating the accuracy of answers. You will be provided with a question and two answers: : one is the ground truth, and the other is model model-generated response. You need to score the model's response based on its similammlai.meoee similarity to the ground truth, using a scale from 10. The specific requi are as follows:
The closer the model's response is to the ground truth, the higher the score.
1.If there are obvious ous errors and the model's response is completely different from the ground truth, score 0-3.
If there are errors and the model's s response is far from the ground truth, score 4-6.
I. there are some errors, and they have some negative impact on the overall response, score 6-8.
4. the model's answer is very close to the ground truth, score 9.
5. If the model's s response is identical to the ground truth, or even richer in content and better expressed, score 10.
Please return the score directly in the following format without any extra information, for example: "Score":
A.2 EXPERIMENTS DETAILS
All Lnnle-image experimental results presented in Tab 2 are obtained using the VLMEvalKit (Duan al., 2024). For the five multi-image benchmarks, MMMU (Yue al., 2024) is also tested using
15

---

Published as a conference paper at ICLR 2025
MMMU
BLINK
Mantis
NLVR2
MVBench Average
35.1
37.1
41.9
52.1
36.0
40.4
Ablation Studies. The top row refers to the LLaVA- v1.5 baseline. We conduct exper- iments about the impact of hyper- parameter and training epochs.
Table 4:
0.1
35.9
41.3
46.1
53.2
39,9
43.3
7-0.2
37.1
39.2
42.4
51.8
39.4
42.0
3
7-0.3
35.8
39.8
42.9
52.0
39.7
42.0
4
epoch
35.9
41.3
46.1
53.2
39.9
43.3
5
epoch-2 2
37.0
38.5
45.2
52.0
39.6
42.5
epoch 3
36.3
42.9
44.2
54.2
39.5
43.4
MMMU
BLINK
Mantis NLVR2 MVBench Average
Ablation Studies. The top row refers to LLaVA-
Table 5:
35.1
37.1
41.9
52.1
36.0
40.4
GPT-Sclection
36.3
41.7
42.9
53.0
39.5
42.7
v1.5 baseline. We conducted
MIA-DPO
36.3
42.9
44.2
54.2
39.5
43.4
an ablation study using GPT-4o-
0.0
+1.2
+1.3
+1.2
0.0
+0.7
mini for data selection.
VLMEvalKit, while the remaining four multi-image benchmarks, which are not yet fully supported by VLMEvalKit, are tested using the official evaluation code.
During the testing of Mantis, BLINK, and NLVR2, to avoid the model providing irrelevant answers, we add prompt suffix at the end of the question to guide the model to directly return the multiple- choice option. This makes it casier to extract the answer from the model's response. The prompts used are listed below: "Return the choice directly." or "Answer:("
Additionally, when testing multi-image benchmarks, we input the images into the model in se- quence. Since the input consists of image sequence rather than merged images, this significantly increases the length of the image tokens, posing a greater challenge to the model. For Mantis-Eve and MMMU, they already have dele-deee official evaluation codes, we used the official ones for testing.
B MODEL AND DATA SOURCES
B. MODEL SOURCES
For the experimental section, we present the testing results of multiple LVLMS several multi- image and single-image benchmarks. The models involved in the experiments are listed in Tab. 6 of the paper.
B.2 BENCHMARK SOURCES
The benchmarks involved in the experiments are diverse and include multi-image benchmarks and 7 single-image benchmarks. These benchmarks cover various domains, allowing for comprehen- sive assessment of the models' actual capabilities. We list all the benchmarks and their detailed information Tab. 7, along with further introduction to some of the benchmarks:
MMMU MMMU (Yue et al., 2024) is benchmark for assessing multimodal models college- level tasks that require advanced reasoning and domain-specific knowledge. It features 1,500 ques- tions across six disciplines and includes diverse image types. Initial evaluations show that even advanced model GPT-4V struggles, achieving only 56% accuracy, indicating substantial room for improvement. In addition, MMMU includes both single-image and multi-image test questions.
BLINK BLINK (Fu et al.. 2024) is benchmark for multimodal language models (LLMs) that tests core visual perception tasks solvable by humans "within a blink," like depth estimation and vi- sual correspondence. It reformats 14 classic computer vision tasks into 3,807 multiple-choice ques- tions with images. While humans achieve 95.70% 70% accuracy, top models like GPT-4V and Gemini perform significantly worse, highlighting gap in visual perception abilities among current LLMs.
NLVR2 NLVR2 (Suhr et al., 2018) is a dataset designed for joint reasoning involving natural language and images, focusing semantic diversity and visual reasoning challenges. It contains
16

---

Published as a conference paper at ICLR 2025
Table 6: Model Sources. We have compiled list of all the models involved in the experiments along with
their sources.
Models
Parameter Release Time
Source
GPT-4V (Achiam et al., 2023)
2023-09
Source Link: OpenA1
Kosmos2 (Peng et al., 2023)
1.6B
2023-06
Source Link: Kosmos2
VideoLLaVA (Lin et al., 2023)
7B
2023-11
Source Link: Video-LLaVa
Fuyu (Bavishi et al., 2023)
8B
2023-10
Source Link: Fuyu-8B
VIIA (Lin et al., 2024)
8B
2023-12
Source Link: VILA
Otter-Image (Li et al., 2023a)
2023-05
Source Link: Otter
Ideficsl (Laurençon et al., 2024a)
9B
2023-08
Source Link: Idefices
BLIP-2 (Li al., 2023b)
13B
2023-01
Source Link: BLIP-2
OpenFlamingo (Awadalla al., 2023b)
9B
2023-08
Source Link: OpenFlamingo
InstrucIBLIP (Dai al., 2023)
13B
2023-05
Source Link: InstructBLIP
Qwen-VL-Chat (Bai al., 2023)
2023-8
Source Link: Qwen-VL--CHe VL--hat
Emu2-Chat (Sun ct al., 2024)
37B
2023-12
Source Link: Emu2-Chat
CogVLM (Wang ct al., 2023)
17B
2023-10
Source Link: CogVLM
Idefics2 (Laurençon et al., 2024b)
8B
2024-04
Source Link: Idefices2
LLaVA-v1.6 (Li al., 2024b)
7B
2024-01
Source Link: LLaVa-Nextl
LLaVA-v1.5 (Liu et al., 2024a)
2023-10
Source Link: LLaVa-v1.5
Inteml.M-XX2.55 (Zhang al., 2024)
7B
2024-07
Source Link: Intern M-X22d5
Table Benchmark Sources. We have included information and links for all the multi- image and single- image benchmarks tested in the paper in the table.
Setting
Models
Evaluation Metric
Number
Source
MMMU (Yuc al., 2024)
Multiple Choice
1,050
MMMU
BLINK (Fu et al., 2024)
Multiple Choice
3,807
BLINK
Multi-Images
NLVR2 (Suhr et al., 2018)
Multiple Choice
6,967
NLVR2
Benchmark
Mantis-Eval (Jiang et al., 2024)
Multiple Choice
217
Mantis-Ei
MVBench (Li et al., 2024c)
Multiple Choice
4,000
MVBench
MMStar (Chen et al., 2024a)
Multiple Choice
1,500
MMStar
Sci-QA (Lu et al., 2022)
Multiple Choice
4,241
ScienceQA
Single-Image Benchmark
MM (Yu et al., 2023)
Subjective Questions
218
MM-Vet
POPE (Li al., 2023c)
Yes/No
9,000
POPE
MMB (Liu al., 2023)
Multiple Choice
1.164
MMBench
Math (Lu et al., 2023)
Multiple Choice
6.141
MathVista
A12D (Kembhavi et al., 2016)
Multiple Choice
3,090
A12D
107,292 examples of English sentences paired with web photographs, where the task is determine the truth of a caption regarding a pair of images.
Mantis-Eval Mantis-Eval (Jiang et al., 2024) comprises 217 reasoning examples involving mul- tiple images, addressing various topics like size perception and weight comparisons. Curated by annotators, the dataset features images sourced from Google Search, accompanied by questions that necessitate a thorough understanding of the image content. includes both multiple-choice and short-answer formats.
MVBench MVBench (Li al., 2024c) is dataset that converts static tasks into dynamic video tasks, requiring diverse temporal abilities, from perception to cognition. It automates the genera- tion of multiple-choice questions from public video annotations, ensuring efficient creation and fair evaluation using ground- -ruth data. The dataset features 20 examples of temporal tasks.
MMStar MMStar (Chen et al., 2024a) is a high-quality benchmark for evaluating multi-modal performance, addressing issues of unnecessary visual content and data leakage in training. It com- prises 1,500 carefully selected samples from initial pool of 22,401, focusing six capa- bilities with 18 detailed dimensions. Each capability features 250 balanced samples, ensuring a comprehensive assessment of multi-modal models.
17

---

Published as a conference paper at ICLR 2025
Table 8: DPO Data Statistic. We listed in the table the data volume used for DPO with 1.L.Va-11.5 and M-XC2. along with the proportion of each type of data.
Models
Total
Sequence
Grid Collage
Pic-in-Pic
LLaVa-v1.5 (Liu et al., 2024a) MnC2mLM-XC2d5 (Zhang al., 2024)
28.9k
15.1k
9.3k 7.8k
4.5k
23.1k
11.7k
3.6k
ScienceQA ScienceQA (Lu et al., 2022) is a newly collected dataset designed for science question answering, comprising 21,208 multiple-choice questions. Each question includes multimodal con- text, the correct option, general background knowledge, and a specific explanation, enabling models demonstrate multi-step reasoning and interpretability. This dataset addresses the limitations of existing resources by providing detailed explanations alongside the answers.
MM MM-Vet (Yu et al., 2023) is designed to evaluate the capabilities of versatile models that integrate various core visual language (VL) functions for solving complex tasks. It defines six key VL abilities rebiitie--cconniiion. OCR, knowledge, language generation, spatial reasoning, and mathe- matical computation-ann examines 16 interesting combinations of these functions. The evaluation employs large language model (LLM)-basse based open-output assessor, which produces unified scor- ing metric across different question types and answer styles.
POPE POPE (Li et al., 2023c) is a dataset designed to evaluate large vision language models (LVLMss by first extracting ground-truth objects from input images using human annotations or automatic segmentation tools. It then conducts negative sampling for non-existent objects under various settings, including Random, Popular, and Adversarial. Finally, these objects are organized into question templates to assess the models' performance.
MMBench MMBench (Liu et al., 2023) is a benchmark designed to evaluate vision-language (VL) models by addressing the limitations of traditional evaluation methods. It features approxi- mately 3,000 multiple-choice questions across 20 fine-grained ability dimensions, enabling a more comprehensive assessment of model performance. By utilizing ChatGPT to match model predic- tions with question choices, MMBench ensures robust evaluations that are less biased and more reproducible.
Math Math (Lu et al., 2023) is a benchmark designed to assess the mathematical rea- soning capabilities of large language and multimodal models visual contexts. It includes 6,141 examples sourced from 28 existing multimodal datasets and three newly created datasets: IQTest, FunctionQA, and PaperQA. The tasks in Vista require fine-gran fine-grained visual understanding and compositional reasoning, posing significant challenges for state-of-the-ee models.
AI2D A12D (Kembhavi et al., 2016) a dataset focused on diagram interpretation and reasoning, addressing the challenge of understanding complex diagrams and their relationships. It includes over 5,000 diagrams and 15,000 annotated questions and answers, providing extensive annotations of diagram constituents and their semantics.
B.3 MIA-DPO DATA STATISTIC
In constructing our MIA-DPO dataset with three types of multi-image data (Sequence Data, Grid Collage Data, and Pic-in-Pic Data), we used the VL6V56 (Liu et al., 2024b) dataset as the founda- tional single-image data. The LLaV6665k dataset contains 665k training samples, with a negligible amount of pure text data, sourced from TextVQA (Singh et al., 2019), COCO (Lin et al., 2014) GQA (Hudson & Manning, 2019), OCRVQA (Mishra al., 2019), and others. By directly using the existing LLaV5665k dataset, we avoided the high costs of building a multi-image dataset from scratch for MIA-DPO.
18

---

Published as a conference paper at ICLR 2025
Prompt: Emagel-inage ImageZ Imsge3-imagee Imige4-imager
In Imagel, provide ene-sontence oepti the provided image.
Imagel
Image2
Imaged
Imaged
Chosen: A bosl with a solod and half drunk gloas le of sorsi sorsi
Rejected: A cell phone with word "Facebook" the
Prompt: ImageI-imager cimgo Image3rrimag Imagedc: image>
Imagel, provide coption image.
Imagel
Image2
Image3
Image4
Chosen: Severd standing green, basketbal with wall them fouring club hanried.
Rejected:
book page soys
Prompt: Imogeeioinege Image2:innge
Imaget,
dog doing image?
Chosen: In the the stonding toilet, nd the sitting counter cobinet The deg sniffs the both bathroom.
the table front computer.
sitting
Imagel
Image2
Prompt: ImageIIiingge Imagee
In Imagel, what type of objects present in this image?
MAIN
The imgg features a couple of stuffed to bears and little dell, which appear to be obando ned and have moss vegetation They are leaning aysinst stone wall, cutsiiee.
Chesen:
ST STEPHENSON 100
on them.
Rejected: In Imagel, there, ore two street signs, free, and group teddy bears
Imsgel
Imoge2
Prompt: ZingeLsimage Imoge2 vimages Iniege3iimmge In Imagež. provide ore-sentence coption for the prov ded image.
Chosen:
Bockside of silver rolex watch that shows the log the chain.
Rejectedi
Codax
Imagel
Image2
Tmoge3
Imagel
Image2
Imoge3
Prompt: Imagel>imege> ImageZzimage Image3 speinn Image2, provide one-dentence caption the provided image.
Chasen: boseboll player on the field with the rumber 27.
Rejected: The number 27 on the box.
Figure Sequence Image Data Cases. The image displays several examples Sequence data.
Considering that DPO requires only small amount of data, randomly sampled portion of the LLaV5665k dataset to construct the three types of data and employed attention selection in MIA- DPO for filtering. The final data volume used for DPO is summarized in Tab. 8.
19

---

Published as a conference paper at ICLR 2025
Imagel
Image2
Prompt: <mager Imagei, what color is the airplane in the image?
Chosen: The cirplane in the image is blue and white.
Rejected: In Imagel, the airplone in the white.
Imagel
Inoce2
Prompt: vinage: In Imege2. what lind of gothering could this zoenarie likely represent?
Chesen:
while playe standin. represent
gathering party omong friends family gathering usunlyy drinks, enjoying sharing
together in cozy, familiar environment. presence refriggereto, nicrowave, en, wine glasses, chairs home apartment where pe opls come together for these types cosuci muskc
Rejected: ImageZ, mon woman are standing kitchen, refrigerstor, microwave, presence penguin suggests that this gothering could likely represent themed partty or costume party. ssed up costumes, and the penguin costume adds un and quirky element to the event: The tctheen setting and the presence of the TV indicate that the party might be hald home or casucl venue where people can ming! and enjoy the festivities.
kitchen sugs
est
dre
vents,
touch ambione entertairment to the gathering, emphos zing the personal connections and the relaxed atmosphere.
Prompt:
smage Inagei, image?
prinary activity person
Chosen:
The primary activity of the person the inage flying lite.
Rejected: In Inagel, the primary activity of the person in the age playing cricket The person is holding a bet and is in the process of swinging it, likely attempting to hit ball.
Pe
Imoget
Image2
Prompt:
omages In Image2. provide one sentence caption for the provided image.
Chosen: that Vote No.
front with shirt
Rejected: A man block and white shirt is standing on a street.
Prompt: cimages In Imagel, are the people the image holding plat
Imagel
Inays2
people image holding them, specifically
Rejected: No, the people in Imagel not holding plates.
cimaye ImageZ, what device beiny ChargedD
the
Image2
Besides small device being charged, which olde r model white iPod connected through USB cable, there is cell phone and remote control nearby the image. The presence of these items sugyests that a typical space where electronic d evices are conmorty used, stored, or charged.
Chosen:
Rejected: Image2, besides device being charged, there couple
pumplón nearby. The apples placed left side of the device, while the pumplón located
Figure 8: Grid Collage Data Cases with Two Images. present some examples of Grid Collage Data, which consists of images created by stitching together 2 to pictures. Here, we showcase examples of images that combine two pictures.
From Tab. 8, we can see that InternLM -C2.55 has inherent multi -itte data in its pre-training set, which enhances its multi-image capabilities compared to I.La As result, InternI M-XC2.5
20

---

Published as a conference paper at ICLR 2025
Prompt: omage Regording the centrol subplot, what ore the the pho to doing?
The people in the photo gathered oround a fenc ed-off perimeter, woiting to watch event to somet hing, They are stonding behind e gate and oppear to onfici patting something happoning within the centrol
Chosen:
Rejected: In the photo, man talking on a cell phone while standing in crowded crea. The crowd consists of several pe ople, including little girl wearing Krispy Krene hat, The scene appeart to be buiy location, possibly restourant public event, where people gothered and engaged in lous octiveties
Prompt: <inoge: With regards to middle subplot, what is setting inage?
Chosen: image features on fashioned estone streets, cent of scones. focodes, balconies uldings and there's an alleyway catching some sunlight.
chorning
Rejected: overall setting of image namrow alleyw oy between two buildings, gressy area
Prompt: sinago In terms of the centrol subplot, what type of workspace is shown in the inage?
Chasen:
cluttered ofe warkspace with The image desk. There computer tiered
ybcord,
memos, books, catebooks, pope pers, electronic paraphartalio
The image shows wer workspace with computier, ke yboord,
Rejected:
well stack of books donu setting for
The wor workspacc situsted kitchen, which
workspace.
Prompt: simages Concerning the plat at the center, what kin d of video gomt ore the mon and the woman playing?
playing boxing ome together, likely using motion-sensttive characters on-soneen.
Chesen: and
Rejected: The mon and the womon are racing video game, they are both holding Wii controllers and ed it that involves racing cars.
Prompt: When to the midole subplot, what focus of the image?
Chasen: The main focus of the image is the bright and color ful umbrelias and lights at a festival, along with a lively ofme ophere filled with people enjoying the event.
Rejected: The main focus of the image the colorful umbre llo, which deconated with faces and placed in a field.
Prompt: simager With respect to the centrol image, what the main subjectt of the image?
Chosen: The main subject image is truck with Furne life-size sculptune "Lunchtine on Skysoroper* displaye d on it.
Rejected: subject image is playe standing on baseball
Figure 9: Pic-in-Pic Image Data Cases. The image displays several examples of Pic-in- Pic data.
exhibits better attention- aware abilities, leading to a smaller amount of DPO data selected through attention-aware filtering compared to V.
21

---

Published as a conference paper at ICLR 2025
C MIA-DPO DATA CASES
C.I SEQUENCE IMAGE DATA
Sequence Image Data is the first type of MIA-DPO data we constructed, where multiple images are combined into a sequence, and questions are posed about a randomly selected image within that sequence. The number of images included in Sequence Image Data ranges from to 5. This approach increases the difficulty answering questions for LVLMs by adding interference from other images beyond the indicated in the instructions. Additionally, imputting multiple images in sequence significantly increases the length of image tokens, posing greater challenge for LVLMs. At the same time, the Sequence Image Data type is primarily designed to address the Sequence Confusion type of multi-image hallucination, while also mitigating the Element Interference type of hallucination some extent. We provide several examples of Sequence Image Data in Fig.
C.2 GRID COLLAGE IMAGE DATA
Grid Collage Image Data is the second type of MIA-DPO data we constructed, where multiple im- ages are stitched together, and each image is assigned a label such as 'Imagel' to indicate which image the instructions refer to for the LVLMs. The number of images in the Grid Collage Data ranges from 2 to 9, forming a large image composed of to 3 rows or columns of smaller images. By combining multiple images, Grid Collage Image Data mixes vast array of visual elements and details, posing high demands LVLMs. The instructions for Grid Collage Data involve question- ing specific visual elements within the image, with other visual elements serving as interference factors. This data type primarily targets the Element Interference type of hallucination, while the numbered labels for each sub-image also assist the model in addressing the Sequence Confusion type of hallucination. We provide several examples of Grid Collage Image Data in Fig. 8.
C.3 PIC-IN-PEL IMAGE DATA
Pic-in-Pic Image Data is the third type of MIA-DPO data constructed. We randomly select two images, resizing one to about half the size of the other, and then paste the smaller image in the center of the larger one. The instructions for Pic-in-Pic Image Data involve questioning the central image, while the background image adds numerous visual elements and details that serve as interference. LVLMs need to carefully distinguish the relationships between these images and integrate the correct visual information to generate answers. Pic-in-Pic Image Data is primarily designed to address the Element Interference type of hallucination. We provide several examples Pic-in- Image Data in Fig. 9.
D MORE OBSERVATION
D. ATTENTION OBSERVATION
In the MIA-DPO architecture, a key step is the selection of chosen and rejected data based on attention. The core idea filter data according to the attention-aware capability of LVLMs. For Sequence Data, we assess the ratio of attention between the instructed image and all images. For Grid Collage Data, we evaluate the attention ratio between sub-images and the larger image. For Pic-in-Pic Data, we analyze the attention ratio between the central area of the image and the entire image.
To ensure the smooth execution of attention- filtering, we visualized the image attention distri- bution at each layer of the LVLMs, as shown Fig. 10. The attention distribution of images varies dynamically across different layers. In the early layers of the LVLMs, there are distinct features in the attention distribution of images, and the same is true for the later layers. However, in the middle layers, there are significant differences in attention distribution among different images. At this point, we can observe where the LVLMs' attention is focused and filter out rejected data where the LVLMSs did not attend to the correct areas.
22

---

Published as a conference paper at ICLR 2025
Imogel
Imoge2
Image3
Image4
Imoge5
ImageI=Imege>Ve ImageZ2Imoge>>W
Image4:<magg: In Imagebi<Zmage>Mn
What's the daing Imagel?
Layer4
Front Layer
Layer4
Loyer5
Layer5
Layer6
Loyer6
Layer10
Layer11
Layer12
Layerz
Middle Layer
Layer15
Loyer16
Layer16
Layer17
Layer17
Layer20
Layer21
Layer22
Loyenea
Layer28
Back Layer
Layer29
loyent
Layer30
Layer30
Attention Observation. We studied the multi-layer attention of LVLMs and found that the atten- Figure 10:
tion of images is most pronounced in the middle layer.
D.2 HALLUCINATIONS OBSERVATION
In the context of multi-image reasoning, the types of hallucinations that LVLMs may produce are more diverse and varied. Therefore, during the data construction process, we need to specifically analyze these hallucination types. In addition to hallucinations that may occur in single-image tasks, such existence, attributes, and relation Hallucination. cinations may exist in imatieimage tasks: Sequence Confusion and Element Interference. These two types of hallucinations primarily caused by excessive number of input images that the
believe that two unique types of hallu-
23

---

Published as a conference paper at ICLR 2025
Table 9:
Comparison of only LNLL multi-image henchmarks.
Models
Parameter
MMMU
BLINK
Mantis NLVR2
MVBench
Average
LLaVA-11.5
7B
35.1
37.1
41.9
52.1
36.0
40.4
CNLL
7B
36.1
39.1
43.5
51.8
39.9
42.1 (+1.7)
Cnco + LNLL 7B
36.3
42.9
44.2
54.2
39.5
43.4 (+3.0)
Table 10: Comparison of only use Lxu on single-image benchmarks.
Models
Parameter MMStar SQA MMVet POPE MMB Math A12D
Average
VA-v1.5
7B
32.9
66.6
30.5
85.9
64.3
25.4
55.5
51.6
+ ENLL
7B
33.7
67.6
28.2
76.9
61.8
24.9
56.3
49.9 (-1.7)
Copo LNL 7B
32,9
67.6
32.1
87.2
63.1
24.4
54.7
51.7 (+0.1)
LVLMs cannot follow in sequence, well the overwhelming number of image tokens and visual elements.
In the process of constructing DPO data, take hallucination types starting point and thor- oughly consider solutions for these two types of hallucinations. More hallucination cases are already presented in Fig. 7, Fig. 8, Fig. 9.
E FURTHER EXPLORATION
E. COMPARISON WITH LNLI ONLY
The LNLI term Eq. (5) plays role of supervised fine-tuning (SFT) with only chosen answers. We compare the baseline of fine-tuning with only the LNLL. and results are presented in Tab. 9 and 10 for multi-image and single-image benchmarks, respectively.
On multi-image benchmarks (Tab. 9), fine-tuning merely with the CNLL (second row) yields per- formance improvement over the LLaVA-v1.5 baseline. However, MIA-DPO (third row) consistently outperforms the -oly-only baseline, demonstrating the significant contribution of negative samples to model improvement.
On single-image benchmarks (Tab. 10), the LxLI--ony baseline (second row) leads to performance degradation compared to ILLVA-v1.55 highlighting the potential risks incorporating multi-image data during SFT may adversely affect performance single-image tasks. By contrast, MIA-DPO (third row) maintains performance parity with LLaVA-v1.5, thanks to the KL-divergence loss con- straint Eq. (3). This further demonstrates the advantages of MIA-DPO over the LNLL-only base- line.
E.2 COMPARISON WITH SET ON MULTI-IMAGE DATA WITHOUT DPO
We conducted experiments comparing MIA-DPO with existing multi-image SFT methods including Mantis (Jiang et al., 2024) or MMDU (Liu et al., 2024d). The results, presented Tab. and 12, demonstrate that MIA-DPO effectively improves multi-image performance without compromising performance on single-image benchmarks. In contrast, direct SFT on multi-image data can lead to slight degradation single-image performance. This highlights the advantage of MIA-DPO in maintaining balance between both tasks.
E.3 ABLATION STUDIES ON THRESHOLD AND DATA RATIOS
Threshold We conduct multiple ablation experiments with different threshold ranges, Tab. 13 and Tab. 14. We observe that our MIA-DPO is relatively robust to different threshold ranges, and default choices (0.7/0.6/0.5/0.5)) perform slightly better than other choices.
Additionally, we present the baseline that uses uniform thresholds, and results are shown in the fourth row of Tab. 13 and Tab. 14. The experimental results demonstrate that using uniform
24

---

Published as a conference paper at ICLR 2025
Table 11: Comparison of SFT on multi-image data with MIA-DPO multi-image henchmarks.
Models
Parameter
MMMU
BLINK Mantis NLVR2
MVBench Average
LLaVA-v1.5
7B
35.1
37.1
41.9
52.1
36.0
40.4
+ SFT on MMDU
7B
36.1
39.2
42.9
53.7
39.1
42.2
SFT on Mantis
7B
35.9
39.5
45.6
52.8
40.9
42.9
MIA-DPO (Ours)
36.3
42.9
44.2
54.2
39.5
43.4
Table 12: Comparison of SFT on multi-image data with MIA-DPO on single-image benchmarks.
Models
Parameter MMStar SQA MMVet POPE MMB Math A12D Average
LLaVA-11.5
7B
32.9
66.6
30.5
85.9
64.3
25.4
55.5
51.6
SFT MMDU
7B
32.5
65.7
30.4
85.9
62.3
25.6
54.7
51.0
+ SFT Mantis
7B
32.9
65.9
29.8
63.5
25.9
54.4
50.9
+ MIA-DPO (ours)
7B
32.9
67.6
32.1
87.2
63.1
24.4
54.7
51.7
threshold can negatively impact performance. This observation is due to the attention ratio distribu- tions varying significantly depending on the number of images (see Fig. 5), which indicates that one-size-fits-all threshold is not optimal. By adjusting the threshold based the number of images, we can improve the model's ability to generate accurate and coherent responses.
Data Ratios We present ablation studies on different data proportions in Tab. 13 and Tab. 14. We adjust the proportions of the three data types and evaluate the results. The results indicate that the model's 's performance fluctuations. This demonstrates the robustness our model across different data proportion settings. In conclusion, none of the data types showed particularly significant advantage others. When used in combination, the three data types achieve better performance than any single type used individually.
both multi-image and single-image tasks remains stable, with only minor
E.4 ABLATION STUDIES ON MODEL SIZE
To explore the effectiveness of larger models, we apply MIA-DPO with the LLaVa-v1.5-13B model. We observe that the attention distribution patterns of the 13B model are largely consistent with those of the 7B model. The results of MIA-DPO + LLaVa-v 1.5-v11..111B benchmarks are presented in Tab. E.7 and Tab. 17. These results further validate our conclusions about the effectiveness of MIA-DPO, demonstrating that it consistently improve the perfor- mance of larger-size models.
multi-image and single-image
E.5 VQA TEST SET AND LARGER IMAGE NUMBERS
We construct VQA test of 500 questions using images and questions from LLLVV-665kk but are mutually exclusive with the MIA-DPO training data. This test set includes questions with 5 images per question, allowing to directly assess improvements in sequence confusion and element interference. By evaluating the pre- and post-DPO versions of LLaVA and IXC2.5 on this test set, we observed accuracy improvements of 5.8% and 1.9%, respectively (see Tab. 18). These results further validate the effectiveness of MIA-DPO in enhancing multi-image understanding. We plan to release this VQA test the final version our paper to facilitate future research this area.
Additionally, We constructed a new VQA test set (the construction steps keep the same) consisting of 50 questions each for 4, 6, 8, and 10 images. We report the performance of IXC 2.5 + MIA- DPO Tab. 19. Our MIA-DPO consists of improving the multi-image understanding abilities as the number of images increases. However, the performance of LVLMs, such as IXC2.5, on extremely large numbers of images will also be limited by factors like context window size. As a result, the performance gains from MIA-DPO will gradually diminish with an increasing number of images. These findings show potential for future research on long-context abilities, such as ROPE extrapolation LVL.Ms.
25

---

Published as a conference paper at ICLR 2025
Table 13: Ablation studies of attention threshold and data ratio on multi-image benchmarks.
Models
Setting
MMMU
BLINK
Mantis
NLVR2
MVBench Average
LLaVA-v1.5
35.1
37.1
41.9
52.1
36.0
40.4
0.860.7/0.60066
threshold
37.5
40.5
42.4
54.3
40.3
43.0
5.750.6
threshold
36.3
42.9
44.2
54.2
39.5
43,4
3
0.6/0.5/0.4/00.
threshold
36.2
41.1
44.7
$4.1
39.9
43.2
0.6/0.6/0.60066
threshold 36.2
40.6
41.9
53.7
39.2
42.3
2:3:1
data ratio 36.1
41.5
42.9
55.0
39.4
43.0
2:1:3
data ratio 37.1
39.8
46.0
52.4
40.2
43.2
3:2:1
data ratio 36.3
42.9
44.2
54.2
39.5
43.4
Table 14:
Ablation studies on threshold and data ratio single-imas benchmark.
Models
Setting
MMStar SQA MMVet POPE MMB Math AI2D
Average
I.LVA-v1.5
32.9
66.6
30.5
85.9
64.3
25.4
55.5
51.6
0.8/0.7/0 7/0.6/0.6 threshold 33.2
67.7
31.1
86.8
25.0
55.1
51.7
2 2..700.6005/000 threshold 32.9
67.6
32.1
87.2
63.1
24.4
54.7
51.7
0.6/0.5/0.4/00 threshold 33.1
67.3
32.2
87.1
63.2
54.7
4 0.6/0.6/0.600
threshold 33.0
67.1 32.3
86.4
62.5
25.5
54.7
5 2:3:1
data ratio 32.2
67.0 32.6
87.1
63.2
24.5
$4.6
51.6
6 2:1:3
data ratio 32.1
67.5
32.0
87.4
63.3
24.7
54.7
51.6
7 3:2:1
data ratio 32.9
67.6 32.1
87.2
63.1
24.4
54.7
51.7
E.6 MORE BASELINES OF INTERNLM- INTERNLM-XCOMPOSR22
Since the authors of POVID and HA-DPO have open-sourced their trained models based LLaVa, we directly tested their released models and report in Tab. 1 and Tab. 2. Additionally, we conducted additional experiments by applying DPO IntemLLM-X22.5 using the datasets provided by POVID and HA-DPO. As shown in Tab. and Tab. 2, the performance improvements for applying POVID and HA-DPO on multi-image benchmarks were limited. This suggests that while these previous papers are designed for single-image scenarios, their direct application to multi-image settings may not yield significant gains. We believe that the limitations observed these experiments highlight the unique challenges posed by multi-image hallucinations and the need for specialized techniques like our proposed MIA-DPO approach.
E.7 ANALYSIS OF FAILURE CASES
We provide some failure cases in Fig. 11. The cases presented are from the multi-image QA of the MMMU benchmark. We observe that although MIA-DPO has improved the model's multi- image understanding and reasoning capabilities, the model may still make errors when encountering questions from out-of-domain knowledge (e.g., fine-grained plant classification, medical image pro- cessing), which is not present in our training data. We believe extending the training data to more diverse domains will alleviate these failure cases.
26

---

Published as a conference paper at ICLR 2025
Table 15: Ablation Study on Different Components of Post-Selection. We conducted ablation experiments
different components of post-selection explore their respective roles and contributions.
Models
MMMU
BLINK
Mantis NLVR2
MV Bench
Average
LLaVA-v1.5
35.1
37.1
41.9
52.1
36.0
40.4
1 w/o post sel.
35.3
38.7
44.2
53.7
39.4
42.3
2 w ppl
35.6
40.4
44.2
53.8
39.5
42.7
3 w length ratio
35.7
40.6
44.3
53.7
39.4
42.7
w edit distance
35.6
40.0
44.2
53.9
39.4
42.6
5 MIA-DPO
36.3
42,9
44.2
54.2
39.5
43.4
Table 16: Study on Model Size for Multi-Image Benchmarks. We conduct MIA-DPO on larger model LLaVa-v1.5-13B (Liu et al., 2024b).
Models
Parameter MMMU
BLINK
Mantis
NLVR2
MVBench Average
LLaVA-v1.5
7B
35.1
37.1
41.9
52.1
36.0
40.4
MIA-DPO (Ours)
42.9
44.2
43.4
+1.2
+3.5
+3.0
LLaVA-v1.5
13B
37.0
40.9
47.0
40.6
45.6
MIA-DPO (Ours)
13B
38.8
42.4
46.5
64.5
42.0
46.8
+1.8
+1.5
-0.5
+2.0
+1.4
+1.2
Table 17: Study on Model Size Single-Image Benchmark. conduct MIA-DPO on larger model LLaVa-vI.5-133 (Liu et al., 2024b).
Models
Parameter MMStar SQA
MMVet POPE
MMB Math
A12D
Average
LLaVA-v1.5
7B
32.9
66.6
30.5
85.9
64.3
25.4
55.5
51.6
+ MIA-DPO (ours)
7B
32.9
67.6
32.1
87.2
63.1
24.4
54.7
51.7
LLaVA-v1.5
13B
34.3
71.2
35.6
86.7
68.5
27.7
61.1
55.0
+ MIA-DPO (ours)
13B
33.4
69.7
36.9
85.1
68.2
25.1
59.4
54.0
Accuracy Image Numbers Vavv1- IXC2.5
Table 18: Results on Our
Original MIA-DPO
75.8
Own VQA Test Set. Our VQA consists of 500 questions, each involving to images.
81.6
90.3
A
2-5
+5.8
+1.9
Models
4-Images 6-Images 8-Images 10-Images
Table 19: Results with more images. MIA-DPO continues to improve multi-image under- standing abilities the number images increases. up the limit of the context window.
IXC2.5
74
10
MIA-DPO
82
72
12
3
A
+10
+8
+4
+2
27

---

Published as a conference paper at ICLR 2025
Insgel=Inage Imopé--imgeeee group par mistaken negrowth fallowing glyphasa blachberry to little leaf' symn commenty witnessed pott-ghyhoostte trealment? Herr some choice. Nematodes, Fungi, toplatam After chooseing the unswer. the restons.
The Nematades Nematodes snall, elanget cylindrical onim that lockberry plants that symptoms commoniy vittessed postt glrphosate treatm ent. They can infecct the rests of the piant, leasing tunted growth. yellowing leaves, and other symptons. N enctoder one often mistoken for regrowtt following gly phosote irsalimntt dus to their similor sppwor sace.
hobit idenh the leaf
Phytopiasmis
Imogel-Imaggell Imagee-dd This hwo-year-o boy wis brought to C-PP with S-week history of occip heodaches.
sleeps week prior presenting to
extended During he vomited day. nourologic exon charges, diegnosis? Subeependymotra Gieblostomo, Anaplostic ependymoms After chooseing the explon the reatons
goit
choice. Pilocytic astracytomo,
answer Pilecyric estrocytons, type tunor charocteniaed dlow growth and besign It typically young composied star-shoped called rocroctss Pilacytic astrecylomo: ametines therapy.
surgury
Figure 11: Analysis of Failure Cases. Although MIA-DPO has improved the model's multi-image understanding and reasoning capabilities, the model may still make errors when encountering ques- tions from out-of-domain unseen training data (e.g., fine- ane-grained plant classification, medical image processing).
Is there red double-decker bus <imogel> and <Imog edx2
Ophore:
(A) There is red double decker bus in image but not
image
(8) There is red double decker bus in image but not (C)
red decker in both images
The questtion pertains to the prisience red mages. mage, bus option 2 the second image
descrip
red double
Ground Truth,
cken
the time shown clack or watch both .Imagel on IInege:> the same??
Opt Sariars
(A) Yes, they are bath clock
Yes, they are
o'clock
Ground Trufh:
(C) they show different
The watcckes both images ane set the soma time. In the top image, the watch the peor shows the time oporoximctely 9 d'clock. In the bottom maye, the woman's watch diso showel the time ea upproximately 9 clock. Th. enefore, the correct answer to the question is (A) Yes, they are both of 9 clock
Figure 12: Analysis of Mantis Benchmark. Original IXC2.5 provides lengthy descriptions instead of concise answers for multiple-choice questions.
28


Reviews:
[{"review_id": "3HoSnK1elI", "reviewer": "Reviewer_ePWn", "summary": "The paper presents MIA-DPO, a multi-image DPO dataset construction approach for LVLMs without annotations from human or external models. The paper collects prompts from existing single-image dataset like LLaVA 665k, augmenting each data with unrelated images, which derives synthetic prompts of multiple images. Taking advantage of transformer's attention scores, rejected samples can be automatically filtered out to build pairwise preference answers for DPO. In addition, some post-selection methods are proposed for data cleaning. Experiment shows improvements on most multi-image and single-image benchmarks, and the capability to overcome hallucinations duo to the interference from unrelated images in prompts.", "questions": "* In Eq 4, please clarify how to calculate R(y) in details. Is the summation of attention values computed over all layers, attention heads and tokens subject to the given image/region?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly written. Most details are provided, so it should be easy to reimplement. \n\n- The proposed method is well-motivated. Hallucination suppression is an important topic in LVLMs. The paper presents a straight-forward way to address hallucinations from unrelated images in multi-image scenario, which seems very easy to follow and does not rely on human annotations. \n\n- Experiments are well organized. Extensive evaluations show the effectiveness of the proposed method. An interesting phenomenon is, although the synthetic DPO prompts appear highly biased, improvements are still obtained on **general** single-image and multi-image evaluations.", "weaknesses": "## Significance\n\nMy biggest concern is, the proposed synthetic method is designed to deal with the interference from unrelated images, however, which seems only a **corner case** in LVLMs. In practice, typically all images in the question prompt contribute to the final answer. It is a great challenge (but more important) to generate preference data for such *real* multi-image prompt at scale without much cost. The paper does not involve the topic. Therefore, I think the title \"Multi-Image Augmented Direct Preference Optimization\" may somewhat overclaim the contribution. \n\nTo improve the work, I think the authors may defend their contributions by analyzing the root of the improvements on general benchmarks. For example, an interesting topic may be \"does the proposed synthetic DPO dataset also help to suppress more general hallucinations other than those in Line 212?\"; if yes, why?\n\n\n## Experiment\n\nI feel some empirical results are weak because: \n \n- According to Table 3, parts of the improvements are resulted from data cleaning (especially on MMMU and BLINK), which is not the unique technical contribution of the paper.\n\n- In Eq 5, the NLL term plays a role of supervised fine-tuning with only chosen answers. It is an important baseline (i.e. fine-tuning with isolated NLL term) which can reflect the effectiveness of rejected samples, but not provided in the experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yN7RgR8gHP", "reviewer": "Reviewer_mzdh", "summary": "This paper introduces MIA-DPO, a visual preference alignment approach for LVLMs that handles multi-image tasks. The authors propose extending single-image data with unrelated images and using attention patterns to detect hallucinations, allowing construction of chosen/rejected pairs without human annotation or external APIs. The method shows promising results on both multi-image and single-image benchmarks.\nOverall, I believe this is a solid paper that addresses an important and under-explored problem. It makes meaningful contributions to visual preference alignment for multi-image scenarios, though there are some aspects that need clarification.", "questions": "Have you analyzed how sensitive the results are to the attention thresholds? What happens if you use uniform thresholds across different image counts?\nFor the post-selection filtering, how did you choose the specific metrics (perplexity, length ratio, edit distance)? Were other metrics considered?\nYour method improves most on the Mantis benchmark (+11.1%). Any insights into why the gains are particularly large there?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* While there's been significant work on visual preference alignment for single images, multi-image alignment remains largely unexplored despite its real-world importance. The authors identify key challenges like data scarcity and high annotation costs.\n* Rather than collecting new multi-image data (which would be expensive), they augment existing single-image data through sequence, grid collage, and pic-in-pic formats. Using attention patterns to select rejected samples is an elegant way to avoid costly human annotation or API calls.\n* The authors test on 5 multi-image benchmarks and 7 single-image ones, showing clear improvements (+3.0% on LLaVA-1.5, +4.3% on InternLM-XC2.5) while maintaining single-image performance. The ablations examining different components are thorough.", "weaknesses": "However, I have several concerns that I think should be addressed:\nMy main concern is about the attention-based selection method. While using attention patterns to detect hallucinations is interesting, the threshold selection feels somewhat ad-hoc. The authors set different thresholds based on image counts (0.7/0.6/0.5/0.5 for 2/3/4/5 images), but there's limited justification for these specific values. I wonder if these thresholds could be learned automatically rather than manually set.\nThe relationship between augmentation strategies and model performance isn't fully explored. While the authors try three formats (sequence, grid collage, pic-in-pic), there's limited analysis of how different ratios of these formats affect results. Some discussion of which formats work better for what types of queries would be valuable.\nThe ablations, while informative, don't completely isolate the impact of different components. It would be interesting to see how model size affects the attention patterns and overall performance. Additionally, some analysis of failure cases would help understand the method's limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wz3c34X37P", "reviewer": "Reviewer_ZLYy", "summary": "In this paper, the authors address the performance limitations of existing Large Vision Language Models(LVLMs) in multi-image tasks, identifying the difficulty of constructing multi-image preference data at low cost as a critical constraint. To tackle this issue, the authors propose Multi-Image Augmented Direct Preference Optimization (MIA-DPO), which can efficiently scale single-image training data to multi-image data without relying on human annotations or external APIs. In terms of data construction, MIA-DPO designs three expansion methods—Sequence, Grid Collage, and Pic-in-Pic to extend single-image data into multi-image version. Additionally, to address the common visual hallucinations of sequence confusion and element interference in  multi-image tasks, a hallucination detection mechanism based on attention values is designed for the constructed multi-image data. The authors conducted ablation experiments on LLaVA-v1.5 and InternLM-XC2.5, and the results demonstrate that the proposed MIA-DPO can enhance performance on multi-image tasks such as MVBENCH without compromising single-image metrics like MMMU.", "questions": "1. How does the model's performance change on other single-image or multi-image metrics before and after applying MIA-DPO? (Refer to Weaknesses)\n2. Can MIA-DPO still provide stable benefits when the number of images is expanded? (Refer to Weaknesses)\n3. Can experiments be conducted on more advanced open-source large vision-language models or those explicitly supporting multi-images, such as the Qwen-VL series and Tarsier?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Research on visual hallucinations in large vision-language models is significant as it can improve the general capabilities of these models.\n2. This paper presents an efficient pipeline for constructing and filtering multi-image preference data, which can generate large amounts of multi-image preference data at a low cost, making it highly applicable.\n3. The paper analyzes the prevalent sequence confusion and element interference phenomena in large vision-language models for multi-image tasks from the perspective of attention, identifying that the key influential factor of these multi-image hallucinations is incorrect attention distribution.", "weaknesses": "1. The authors mention at the beginning of the paper that existing multi-image SFT methods can degrade performance on single-image tasks while improving multi-image performance. However, they do not compare their approach with these methods in the experimental section.\n2. Although the MIA-DPO method can easily convert data into SFT format, the authors do not compare the performance changes of the model after SFT using the same source data.\n3. When evaluating the model's single-image performance, only MMMU is used, without assessing performance on other widely recognized benchmarks such as MMStar, MMBench, and OCRBench.\n4. For evaluating the model's multi-image performance, the authors could use the Sequence, Grid Collage, and Pic-in-Pic methods to construct multi-image VQA test sets from existing single-image VQA data, directly assessing improvements in sequence confusion and element interference.\n5. While handling more images is crucial for large vision-language models, the paper does not analyze whether MIA-DPO can still provide stable benefits when the number of simultaneously appearing images increases significantly (e.g., 10+ images).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Jl8XvCxHU", "reviewer": "Reviewer_DZHj", "summary": "This paper proposes MIA-DPO, a method for improving the Large Vision Language Modle (LVLM) with multiple images. The main idea of MIA-DPO is finding the rejected prompts by using attention from pretrained LVLM and using the DPO training to fine-tune the LVLM for multiple image settings. Through the proposed MIA-DPO, the paper achieves better performance on multiple image settings than the baseline LVLM, without needing 1) The data for the multiple image setting and 2) external sources such as human/ChatGPT for creating feedback in the RLHF/RLAIF methods.", "questions": "All the concerns and questions are listed in the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The motivation of this paper is clearly stated, which helps the reader a lot in understanding the idea.\n\n- The proposed **Attention Aware Selection** can alleviate the problem of multiple image prompt datasets requirement and the need for external human/module in RLHF/RLAIF approaches.\n\n- The experiments show the effectiveness of the method against the baselines.", "weaknesses": "**Clarification**:\n- Can the authors provide a more specific way to compute $A_{target}$ and $A_{sum}$ in L306? In L306-L307, the paper states that \"$A_{target}$ be the amount of attention directed toward the target defined in $x$\", however, as defined in L150, $x$ consists of $v$ and $t$, and as mentioned in Figure 2, only the attention on the given images $v$ is considered. \n- Which are the layers used to extract the attention $A_{target}$ and $A_{sum}$?\n- In section 3.3.3, what are the parameters we need to optimize in the proposed method?\n\n**Experiments**:\n- In Table 1 and Table 2, why do the authors not include \"RLHF\", \"HA-DPO\" and \"POVID\" in InternLM-XC2.5?\n- In Table 1, why LLaVA-RLHF, HA-DPO, and POVID are worse than LLaVA?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)