LLaVA-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
Authors: Feng Li
Renrui Zhang
Hao Zhang
Yuanhan Zhang
Bo Li
Wei Li
Zejun MA
Chunyuan Li
Venue: ICLR 2025 Spotlight (2025)
Keywords: large language model
multimodal learning
interleaved image-text
Primary area: foundation or frontier models, including LLMs

Abstract: Visual instruction tuning has made considerable strides in enhancing the capabilities of Large Multimodal Models (LMMs). However, existing open LMMs largely focus on single-image tasks, their applications to multi-image scenarios remains less explored. Additionally, prior LMM research separately tackles different scenarios, leaving it impossible to generalize cross scenarios with new
emerging capabilities. To this end, we introduce LLaVA-Interleave, which simultaneously tackles Multi-image, Multi-frame (video), Multi-view (3D), and Multi-patch (single-image) scenarios in LMMs. To enable these capabilities, we regard the interleaved data format as a general template and compile the M4-Instruct dataset with 1,177.6k samples, spanning 4 primary domains with 14
tasks and 41 datasets. We also curate the LLaVA-Interleave Bench to comprehensively evaluate the multi-image performance of LMMs. Through extensive
experiments, LLaVA-Interleave achieves leading results in multi-image, video,
and 3D benchmarks, while maintaining the performance of single-image tasks.
Besides, our model also exhibits several emerging capabilities, e.g., transferring tasks across different settings and modalities.

Published as a conference paper at ICLR 2025
LLAVA-INTERLEAVE: TACKLING MULTI-IMAGE, VIDEO, AND 3D IN LARGE MULTIMODAL MODELS
Feng Li Renrui Zhang 13xx Hao Zhang 1.2x Bo Li Li' Zejun Ma', Chunyuan Li
Yuanhan Zhang 1.4
ByteDance 2 HKUST CUHK NTU
Core contributor
Dtrpa://Llavas www.vvl..ggiteee 2014://////2220--0 06-0066111 -ll-va-aaaeee interleave
LLaVA-NeXT
Multi-Image SOTA
Multi-frame SoTA
Multi-view SoTA
Averago Score (86
76.7
91.
26.6
3.59
4.13
16.
3.9
3,19
66.4
3.1
20.3
40.5
33.3
Q-Banch
NLVR2
Montis
BLANK
MMMMU uny
Spot Diff
Story Telting
Tort-rich VGA
Mulli-inage
vDD
VcG.co
VCG-D0
VCG-CU
VCG-TU
VCG-a
AUFRED
nuscenes
Tamk Decomp
aD Dialoque
VOA
Multi-image
Multi-Frame (Video)
Multi-view (3D)
Figure Performance comparison three interleaved scenarios, including multi-image, multi- frame (video), and multi-view (3D). Our LLaVA-Interleave model achieves SoTA performance across a variety of evaluation benchmarks.
ABSTRACT
Visual instruction tuning has made considerable strides in enhancing the capa- bilities of Large Multimodal Models (LMMs). However, existing open LMMs largely focus single-image tasks, their applications multi-image scenarios remains less explored, Additionally, prior LMM research separately tackles dif- ferent scenarios, leaving it impossible emerging capabilities. To this end, we introduce LLaVA-Interleave which si- multaneously tackles Multi-image, Multi-frame (video), Multi-view (3D), and Multi-patch (single-image) scenarios in LMMs. To enable these capabilities, we regard the interleaved data format as general template and compile the M4- Instruct dataset with 1,177.6k samples, spanning 4 primary domains with 14 tasks and datasets. We also curate the LLaVA--nterleave Bench to compre- hensively evaluate the multi-image performance of LMMs. Through extensive experiments, -LaVA--nterlcaav achieves leading results in multi-image, video, and 3D benchmarks. while maintaining the performance of single-image tasks. Besides, our model also exhibits several emerging capabilities, e.g., transfer- ring tasks across different settings and modalities. Code is available at https: //github com/ LLVVA VT/LLaVA- -NXXT.
generalize
scenarios with new
INTRODUCTION
Recent advancements in Large Multimodal Models (LMMs) (37; 26; 43; 12; 64; 11; 66) have show- cased impressive capabilities diverse multimodal contexts, advancing the pursuit of artificial gen- eral intelligence. With extensive vision-language data (46; 47), they empower Large Language

---

Published as a conference paper at ICLR 2025
Isttr malli VIDAS
Splii insepaaches
Eusalmulii franes
Multi-image
Multi-Frme (Vidoo)
Multi-Vie View (3D)
Multi-Patch (Single-imsge))
(b)
(c)
(d)
Tasks our M4-Instruct. (a) showcases an example of interleaved multi-imas scenarios (visual story telling). (b), (c), and (d) indicate that video, 3D and single-image data can also be organized as the interleaved data format for unified processing.
Figure 2:
Models (LLMs) (52; 53; 8; 5) with visual modality by aligning vision encoders (44; 9; 45). This in- legration has propelled forward the field of AI, enabling complex image and language understanding tasks to be performed with unprecedented accuracy.
However, most open-source LMMs (24; 11; 34; 36) have primarily focused on pushing the perfor- mance limit of the single-image scenario, the more complex multi-image scenarios remain largely less explored. This oversight is significant given that many real-world applications demand multi- image capabilities, such as comprehensive multi-image analyses. Traditionally, researchers have approached these challenges by training separate task-specific models for each application scenario, e.g., multi-image (19; 27; 1), video (7; 29; 67), and 3D (14; 58; 15). This both labor-intensive and time-consuming, resulting in fragmented methodologies that are inefficient and often unscalable. Considering the diverse range of computer vision settings and data formats, there a pressing need to develop general framework for LMMs that can operate effectively across these varied contexts.
In this paper, we observe that the image-text interleaved format can naturally serve as a general data template to unify different scenarios, e.g., single-image multi-image special cases, video as multi-frames, and 3D as multi-views, as illustrated in Figure 2. Therefore, we present LLaVA- Interleave, an all-around LMM that extends the model capabilities to various real-world settings such as Multi-image, Multi-frame (videos), Multi-view (3D) while maintains the performance of the Multi-patch (single-image) performance. We denote the four settings M4.
The core innovation of our approach lies in the perspective to leverage an image-text interleaved format as a universal data template capable of accommodating different scenarios, and construct the related visual instruction-followe across various domains, but also allow the model to emerge new capabilities due to cross-domain task composition.
data. This perspective not only simplifies the training process
Our contributions are summarized below:
Interleave data format unifies different tasks. We convert multi-image, vidco, 3D, and single-image data all into an interleaved training format, which unifies different tasks in single LMM.
New dataset and benchmark. We compile high-quality training dataset, M4-Instruet, with 1177.6 samples to empower LMMs with the M4 capabilities, which spans primary domains (multi-image, video, 3D, and single-image) with 14 tasks and 41 datasets. We also curate I.LaVA-Interleave Bench, a diverse set of benchmarks to evaluate the multi-image performance, including 7 newly collected and 13 existing in/out-domain benchmarks.
SoTA performance. With single model, LLaVA-Interleave can achieve leading results across different multi-image tasks compared to the previous SoTA, while maintaining the single-image performance, exemplified in Figure 1.
Emerging capabilities with cross-task transfer. By jointly training on diverse set of tasks, our model showcases emerging capabilities to transfer tasks across different settings and modalities. e.g., from spotting differences between images to videos.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Interleaved Image-text Training Data. As a more general format, interlcaved image-text data can enable LMMs with two distinctive capabilities: multimodal in-context learning (ICL) capability and instruction-followii
capability in real-world multi-image application scenarios. The former in-context scenarios interleave several image-text examples within the prompt as task demonstra- tions, adapting LMMs to new tasks in the inference stage in a few-shot manner. Flamingo (1) is first model to demonstrate this capability, and thus is considered as GPT-3 moment for multimodal community. Typically, the multimodal ICL ability is emerged after pre-training on web-scale raw interleaved image-text sequences. In the open-source community, MMC4 (68) introduces a public 101.2M interleaved dataset spanning everyday topics, OBELICS (22) also presents a filtered dataset comprising 141M interleaved web pages. 1 (18) curates a 71M multimodal corpora, in- cluding arbitrarily interleaved documents. To explicitly enable the ICL capability, MIMIC-IT (25) proposes an automatic pipeline to create 2.8M multimodal samples in the instruction-tune suttiinnnttiinggg stage. On the other hand, the latter multi-image scenarios aim to tackle diverse real-world applications scenarios that involve multi-images. The training data of VPG-C (27) collected 4 new datasets with ChatGPT. Kantis-Instruce (19) compiles existing 11 interleaved datasets and creates 4 new datasets.
The proposed M4-Instruct (19) compiles existing 41 interleaved datasets and creates 6 new datasets, covering a much higher scenarios diversity than
Interleaved LMMs. As representative closed-soure source LMMs, both GPT-4V (42) and Gemini (12) support real-world -ulti-image application scenarios with leading performance. With various public datasets aforementioned, the community has developed open-source LMMs equipped with remark- able multi-image proficiency. The ICL. performance is typically considered to evaluate multimodal pre-training, which has been adopted in several known LMMs, such as OpenFlamingo (2), IDEFICS series (22; 23), VILA (33) and MMI (41), Emu2 (51). Otter (25) is initialized from OpenFlamingo, and is fine-tuned on the MIMIC-IT dataset to further improve ICL ability with instruction -unnnng- In contrast, the use of instruction-tun tunnin in LMMs for various real-world multi-image applications has been less explored, despite of Mantis (19). The proposed LLaVA VIInteerleave not only broad- ens the multi-image scenario itself as demonstrated by the improved experimental results, but also generalize the settings to diverse scenarios with one model, e.g., video, 3D, and single-image. The cross-scenario training leads to emerging capabilities, achieving zero-shot task composition in new multi-image contexts.
Interleaved Benchmarks. To assess the interleaved multi-image capabilities of LMMs, there have been several -ugaltyultty benchmarks in various scenarios. The ICL benchmarks (20; 49) for LMMs comprehensively evaluate their interleaved skills from few-shot to many-shot settings. For the more challenging multi-image scenarios, previous works mainly focus on a specific domain for evalua- tion, including NLVR2 (50) for daily-life VQA, MMMU (61) for colleague-level problem-solving. Verse-my (65) and SeiVersemmy (13) for mathematical and scientific reasoning, BLINK (10) to challenge LMMs, and Mantis-Eval (19) for multi-image understanding. To further evaluate LMMs on collection of multi-image scenarios, DEMON (27) is the first benchmark that compiles dozens of datasets with 477K samples. With the large amount of data and high diversity, DEMON lays good foundation for multi multi-image research. Unfortunately, it also inherits significant amount of low-quality data samples from existing datasets. To facilitate evaluation, the proposed V.L Interleave Bench curate high-quality samples, comprising both specific (synthetic, mathematical, low-level) and general (daily, real-world, text-rich) multi-image scenarios. With 9 newly curated and 13 existing datasets, we categorize them into in-domain (12.9K) and out-domain (K) schemes. Con-current multi-image evaluation benchmarks include MuirBench (54) and ReMI (21).
3
INTERLEAVED MULTI-IMAGE TASKS & DATA
3.1 TASK OVERVIEW
We observe different computer vision scenarios can be generally represented by the interleaved multi-image format, such as video, 3D, and single-image data. Therefore, to endow LLaVA- Interleave with diverse capabilities, as shown in Figure 2, we adopt the interleaved multi-image format to unify the data input of the following four tasks:
3

---

Published as a conference paper at ICLR 2025
Spot the fference
Vicual Story Telling
Image Edit Instruction
& mabe mepad
Luiid the grea ('ugae giota
tentle rage to lurget irige
Searcs
Torget
mags right unieen our coomi Dre intorm ascer ande
Hoow rullon
UaetA MATT
betting nomer
Media
Lou-level Comparison
Video Description
Grounding
Tocter pat prosiey The loving naged
is hou durpstiu the don cheeee
Sivanin derabae Aviiig vides
(1N96-92CC-MM:: in
conketter frn the even aioneatig
Valeyy The Hornng coeee with briws
Surper
Figure 3: Task examples of M4- containing diverse scenarios in multi-image, multi-frame (video), and multi-view (3D).
Multi-image scenarios include visual instructions incorporating interleaved vision-language input with multiple images. This setting covers 12 challenging real-world tasks included in train- ing data, such as spotting the difference, visual story telling, image editing instruction generation, interleaved multi-image dialogue, multi-image puzzle, low-level multi-image assessment, etc.
Multi-frame scenarios refer to taking video as input data by sampling it into multiple frames, preserving temporal visual cues across the multi-image sequence. We mainly focus on 2 tasks: video detailed captioning and video VQA.
Multi-view scenarios depict 3D environments by multi-view images from different perspectives, where the visual correspondence and disparity can indicate spatial information in the 3D world. For 3D perception, we include 2 tasks: embodied VQA (dialogue and planning), and 3D scene VQA (captioning and grounding).
Multi-patch scenarios represent the conventional single-image tasks. With the design of 'any resolution' in LLaVA-NeXT (36), we divide high-resolution image into multiple low-resolution patches for efficient visual encoding, compatible with our interleaved multi-image format.
3.2 M4-INSTRUCT
To empower all-round multi-image capabilities, we meticulously curate a comprehensive training dataset including 1177.6K instances, termed M4-Instruct, widely spanning multi-image, multi- frame, and multi-view scenarios with 14 tasks and 41 datasets, along with multi-pate data to pre- serve basic single-image performance. We showcase different task examples in Figure
We exhibit data overview of M4-Instruct in Figure 4, and the detailed data statistics in Table 15. For the multi-image data, most of the datasets are collected from previous public efforts and rigorously converted into our unified format with task-specific instructions, some inspired by DEMON (27) and Mantis (19). On top of that, we also utilize GPT-4V (43) to annotate 3 new tasks to enable more di- verse capabilities, i.e., Real-world Difference, Synthetic Difference, and Twitter Post. For the video data, we collect a 255K subset from LLVV--Hound (63), including 240K video VQA and 15K video detailed captioning. We also include NExT-QA (57) and STAR (55) expand our video training data. For the 3D data, we widely gather the training set from nuScenes QA (6), ALFRED (48), ScanQA (3), and 3D-LLM (16), covering both outdoor and indoor scenarios. For the single-image
4

---

Published as a conference paper at ICLR 2025
Deliveen
Invee Edits Ieceraation
Imaage Palte
Spen
lomorvttion
Vicual Story Tellig
Video VQA
3D
Multi- image 500.8K
The na Voa
BLINK
Oui-domais
1.1K
Nort
Carated Benchmarks
In-domsin
Newly
Sep viA
YUA
Video
Multi image VQA
Haistnng
Benchmarks
5K
Pazzle
Q-Bench
Single-
In In-domain Existing Benchmarks
image
Lowe level Coropatioon waan
307K
loage repies
8K
40%
LlaVA-
NeXT SPT Data
NLVR2
Figure 4: M4-Inst training data statistics.
Figure 5: LLL Bench statistics.
data, we randomly sample 40% of the stage-2 fine-tuning data from LLaVA-NeXT (24), which aims to preserve the single-image capacity.
To comprehensively evaluate the interleaved multi-image performance, we introduce the LLaVA- Interleave Bench for LMMs, consisting of 13 challenging tasks with 17K instances. We present data overview of the benchmark in Figure 3, and the detailed data statistics in Table 16. In detail, we categorize multi-image tasks into two classes:
In-domain Evaluation includes tasks that have been 'seen' during our training, designed to verify the model performance within familiar scenarios. We adopt 5 newly curated multi- image tasks corresponding to training datasets, and 2 existing benchmarks, Q-Bench (56)
and NLVR2 (50), with 12.9K in total.
Out-domain Evaluation involves tasks that don't overlap with training scenarios, aiming reveal the generalization capacity of LMMs. We construct 2 new tasks for multi-image mathematical (Math Verse (65)) and scientific (SciVerse (13)) comprehension, and utilize existing benchmarks. Mantis-Eve (19). BLINK (10), and MMMU (60), with 4.1K in total.
INTERLEAVED VISUAL INSTRUCTION TUNING
In this section, we introduce several key techniques during the interleaved visual instruction tuning of
For architecture designs, we follow LLaVA-NeXT (24) to adopt the most general framework, i.e., vision encoder (62), an intermediate projector, and an LLM (4). Then, we consider the following techniques to achieve improved multi-image performance.
Technique 1: Continue training from single-image models. The interleaved multi-image tasks flexible in formats and challenging
be regarded as extension of single-image scenarios, in reasoning. Therefore, to better leverage the pre-trained single-image proficiency, we adopt an off-the-shelf LLa VA-eeeT--lmaaa caption pre-training and stage-2 single-image fine-tuning. On top of this model, we perform the interleaved multi-image instruction tuning with our M4 Instrucct dataset.
(24) as the base model, which has gone through a stage-l image-
Technique 2: Mixed Interleaved data formats during training. We adopt two format choices for the positions of image tokens during the interleaved multi-image training. The first to place all the image tokens in front of the prompt, while maintaining the placeholders (image) within the text, denoted as tett-hefon format'. The second preserves the interleaved format to put image tokens in the place they originally in, i.e., the positions of (image), denoted as 'interleaved format' In this way, LLVVA-Intellavve supports more flexible inference modes, exhibiting robustness to different input formats.
5

---

Published as a conference paper at ICLR 2025
Table 1: Results on our LLaVA-Interleave Bench. SD: Spot the Difference, IE: Image Edit In- struction, VST: Visual Story Telling, TRVQA: Text-rich VQA, MIVQA: Multi-image VQA, QB:
Q-Bench, SQ: ScanQA, Math: Math Verse-my, Sei: Sci Versermmm
Model
Inddenain Evalastion
Oztelommin Exslutioon
Ang
SD
IF
VST
TRVQA
MIVQA
Pazslc
QRI
NEVR?
Ang
Math
Sei
Mantis
BLINK
MMMII
GPPLA (45)
99.2
12.5
11.0
10:9
54.5
$2.0
17.1
76.5
68.8
57.8
80.3
66.9
62.7
51.1
47.9
MLaVA-NeNTAnaap (16)
10.4
12.0
13.3
10.1
59.5
90.4
9.0
51.0
58.0
20.4
13.5
12.2
46.1
41.3
13.5
VIG-CC (28)
35.8
27.8
15.2
21.5
36.9
46.8
2.4
57.6
73.2
14.5
24.3
23.1
53.4
43.1
2944
Masos (78) (199
29.5
17.5
11.2
12.5
45.2
52.5
25.7
69.9
$7.4
39.3
27.2
29.3
59.3
46.4
34.1
LLAVA Inteiiooee
0,58
43.9
34.3
21.6
29.7
63.9
54.8
35.4
52.0
57.8
33.1
13.3
12.2
45.6
39.1
28.6
87.5
95.0
76.7
32.7
52.1
Combining different data scenarios improves individual task performance. Most existing works conduct supervised fine-tuning with only one type of data source, e.g., multi- image tuning of Mantis (19) and multi-frame tuning of ALaMA-VID (31). Instead, we utilize the M4-Instruct to simultaneously conduct instruction tuning with four different tasks (multi- amage/frane//iee/pacch)) With a unified interleaved format, distinct data scenarios have the po- tential to provide complementary semantics and instruction-follwiiii capabilities.
Technique 3:
EXPERIMENTS
In Section 5.1, first introduce our evaluation schemes and implementation details. Then, in Section 5.2, we report and analyze the quantitative results in four interleaved multi-image scenarios.
5.1 SETTINGS
Evaluation Schemes. We evaluate LLaVA-Interleave on four real-world interleaved scenarios, ie., multi-image, multi-frame (video), multi-view (3D), and multi-patch (single-image).
For multi-image evaluation, adopt the proposed LLaVA-Interleave Bench covering comprehensive in-domain and out-domain tasks.
For video evaluation, we utilize the existing NExT-QA (57), MVBench (30), Video De- tailed Description (VDD) (67), and ActivityNet-OU QA (Act) (59). For ActivityNet-QAA we present both the accuracy and GPT score (Acc/Score). We also evaluate on VideoChat-GPT (VCG) (39) with five metries: CI (Correctness of Information), DO (Detail Orientation), CU (Context Understanding), TU (Temporal Understanding), and CO (Consistency).
For 3D evaluation, select ScanQA (3), two tasks from 3D-LLM (16), i.e., 3D-assisted Dialogue and Task Decomposition, and also curate two new test set from nuScenes VQA (6) and ALFRED (48).
Implementation Details. Following the same architecture in LLaVA-NeXT (24), our LLaVA- Interleave adopts Qwen 1.5 (5) as the base LLM with 0.5B, 7B and 14B parameters, SigLIP- 400M (62) 384)) the vision encoder, and a two-layer MLP as the projection layer.
5.2 MAIN RESULTS
As reported in Table 1, the average multi-image performance of LLaVA- Interleave surpasses previous open-source models in both in- and out-domain benchmarks. For in- domain evaluation, our model demonstrates significant advantages across various tasks as expected, due to the multi-image instruction tuning with M4-Instruct. For out-domain evaluation, LLaVA- Interleave also showcases superior generalization capacity within novel scenarios, e.g., comparable to GPT-4V on Mantis-Eval and BLINK.
Multi-image Results.
Multi-frame (Video) Results. Compared with previous video-based LMMs under similar model sizes, LLaVA-Interleave achieves superior results many benchmarks in Table 2, though not

---

Published as a conference paper at ICLR 2025
Table 2: Results on multi-frame (video) benchmarks. VDD: Video Detailed Description. CI (Cor-
rectness of Information), DO (Detail Orientation), CU (Context Understanding), TU (Temporal Un- derstanding). and CO (Consistency).
VideoChat-
Model
NEXTQA
MVBench ActivityNel-QAA
VDD
GPT-4V (43)
4,00
3.88
4.37
VideoChulGPT (7B) (40)
35.2/2.70
2.40
2.52
2.62
198
2.37
Video-LLLaV (7B) (32)
45.3/3.30
2.87
2.94
3.44
2.45
2.51
VISTA-LLaMA (7B) (38)
48.3/3.30
2.31
2.64
3.18
2.26
VideoChar2 (7B) (29)
68.6
49.1/3.30
3.51
LLaMA-VID (31)
50.2
47,4/3.30
2.84
3.01
2.97
3.54
2.53
2.60
LLaVA-NeXT-Visko (7B) (67)
53,5/3,20
3.32
3.39
3,29
3,92
2.60
3.12
LLaVA-NeXT-Video-DDO (7B)
60.2/3.50
3.72
3.64
3,45
4.17
2.95
4.08
LLaVA-NeXT-Viiee--DOO (34B)
64.4/3.60
3.84
3.55
4.24
4.12
LLaVA-Intericave
0.5B
59.5
45.6
48.0/2.84
3.25
3.12
2.97
3.62
2.36
3.27
7B
78.2
53.1
55.3/3.13
3.57
3.51
3.28
3.89
2.77
3.68
14B 7B
79.1
54,9
56,2/3.
3.59
3.65
3.37
3.98
2.74
3.67
77.9
55.0/3.13
3,90
4.24
Table 3: Results multi-view (3D) benchmarks. 3D-assisted Dialogue and Task Decomposition are evaluation tasks from 3D-LLM.
DDassisted Dialogue
Task Decomposition
ScanQA (val)
nuScenes
Model
Avg
ALFRED
VQA
Flamingo (1)
20.5
27.9
33.2
31.1
5.3
4.9
GPT-4V (43)
31.2
35.4
32.6
10.3
63.7
Point-Bind & I.LM (14)
22.5
38.3
35.8
34.6
0.6
3.3
3D-LLM (17)
22.9
39.3
37.8
35.7
1.4
0.4
Mantis (7B) (19)
18.7
2.60
14.7
16.1
14.0
46.2
-nterllveee
0.5B
57.0
7B
58.2
69.3
51.4
32.2
61.6
76.5
14B
59.2
70.6
52.2
34.5
62.0
76.7
specifically designed for video tasks. We also follow LLaVA-Hound add DPO training after our M4-Instruct tuning. After adding DPO, our 7B model attains SoTA performance on VDD and VideoChat-GPT benchmarks, surpassing the previous LLaVA-NeXT-vin strates the effective temporal understanding and reasoning capabilities of our model across sequen- tial frames. Note that we calculate the average scores by multiplying a weight of 10 times by the score of Video Detailed Description and VideoChat-GPP
(34B). This demon-
Multi-view (3D) Results. For 3D perception in Table 3, model also obtains leading results for both indoor and outdoor scenarios on five in-domain benchmarks. Compared to 3D-LLLM and only accepts multi-view images
Point-LLM with additional point clouds input, to interpret the 3D world, attaining significantly higher scores in challenging 3D scenarios.
Multi- (single-image) Results. We also add 307k (40%) of original LLaVA-NeXT single- image data, which makes our model capable of doing single-image tasks. We use the anyres training for single-image data, which divides image into multiple patches, forming another multi-image setting. As shown in Table 5, we maintain the single-image performance of LLaVA-NeXTT Image. As single-image data is of high quality and diversity, adding single- image data also improves the cnstruction-followinn ability and enables task transfer from single-image to multi-image which is demonstrated in Section 6.

---

Published as a conference paper at ICLR 2025
Table 5: Results on multi-patch (single-image) benchmarks with different LLM sizes. 'Single' and "Iterleavee' denote LLaVA-NeXT-Imma Image and our model, respectively.
Model
LLM
Avg
A12D
CharlQA Doc MME
SciQA
POPE
Single 0.5B
59.8
51.7
50.2
59.1
52.8
60.0
85.4
Interleave
60.5
52.2
52.2
59.2
52.0
60.6
86.8
Single
72.3
72.7
66.3
75.6
61.0
71.1
86.9
7B
Interleave
73.3
73.9
67.2
75.7
63.5
72.6
86.8
Single
77.2
77.5
72.1
80.0
67.7
78.9
87.3
14B
Interleave
76.4
76.5
71.2
78.9
66.2
77.4
87.9
Table 6: Ablation on whether continue training from single-image models. QB: Q-Bench, Act: ActivityNet-O QA, MVB: MVBench, VDD: Video Detailed Description, MME*: Throughout our paper, we convert MME's score to accuracy by summing up the perception and cognition scores and dividing 2800, SQA: Scienceqa-IMG.
Continne
Malt-
Multi drnme
Malti-view
Single-image
Marns
BLENK
Q6
NUVR2
Act
MVB
VDD
ScantJA
A120
CharQAA
DeeVQA
MME
POPE
SQA
From pre-training
41.0
37.6
47.0
56.0
44.9/2177
43.0
2.96
21.7
46.3
3833
42.1
8544
59.4
From single image models
45.6
392
520
67.
48.02.84
45.6
3.25
29.3
522
522
59.2
320
368
50.5
5.3 ABLATIONS OF PROPOSED TECHNIQUES
We study the effectiveness of the three proposed training techniques in Section 4 below.
In Table 6, we compare training strategies. It is seen that initialization from a good single-image model checkpoint (from Stage-2) consistently enhance the interleaved multi-image performance, than directly from a Stage-1 model checkpoint.
In Table 7, our mixed-format training benefit the results of both input formats.
In Table 8, we progressively incorporate single-image and multi-image data upon the video data. The integration of more sources contributes to enhanced performance, compared with models from individual visual scenarios.
6 EMERGING CAPABILITIES
In this section, show some example to demonstrate the emerging capabilities of our model. Emerging capabilities means the capabilities not trained during training but demonstrated when inference. We mainly showcase the emerging capabilities from three aspects:
1. Task Transfer from Single-image to Multi-image: The capability to reason over one image and tell the funny part is initially observed single-image models (35), and not cluded in our multi-image training. As shown in Table 9, our model is capable of analyzêng the fun part within multiple images. This new task is probably emerged by the composition of the single-image capability and multi-image VQA training.
2.
Task Transfer from Image to Video: We only include the multi-image Twitter post task in the M4-Instruct training, while our model can directly perform the witter post from video, as shown in Table 10. This new task is probably composed by the training data of imale Twitter post and video VQA tasks.
3. Real-world Applications: In Tables 11 and 12, we showcase one real-wort woaldworld scenario that is not explicitly contained in our interleaved training data, which multi- multi-image painting style recognition and PPT summary & QA. This demonstrates our generalization potentials broader spectrum of applications. More examples are in Appendix.
8

---

Published as a conference paper at ICLR 2025
Table 7: Ablation on mixed interleaved data formats. We select several impor- tant tasks within our LLaVA-Interlean Bench for the ablation study.
Table Ablation on the improvement of com- bined data scenarios for video tasks. CI (Cor- rectness of Information), DO (Detail Orienta- tion), CU (Context Understanding), TU (Tem- poral Understanding), and CO (Consistency).
Spatt::
Sao
NEXT-QA
VDD
VideoChuGPP
hepfecice
$2.0
15.
21.
10.1
140
CI
DO
CU
TU
CD
154
51.6
318
374
963
15.0
Valeo
Video Single-image
67.7
42.6
3.49
3,46
3.46
3.47
3.30
3.27
3.87
3.85
2.71
2.74
3.60
3.61
<70
283
324
761
150
Valan Both
Video Mulii image
72.7
78.2
3.50
3.58
3.50
3.50
3.31
3.27
3.90
3.87
2.70
2.77
3.63
3.68
Table 9: An example of emerging capability: multi-image fun part analyze.
Task Transfer from Single-image Multi-image
WHERE HEADE??
STUL FHEL tHe with TROSE
somng to paEc onl
WHAT IS COMMIATTIGG REGLLT CONSEIRESEESS HIIH END?
User
What is fun about the images?
Ours
The humor in these images comes from the anthropomorphized hirds engaging in human-like activities and conversations. In the first image. one bird is asking about going get some fries on the pier, which is human activity. The second image shows a bird asking about the "culminating result of consciousness." which a philosophical question rather than a literal onc. The third image has a expressing its desire to do something with fries on the pier. again referencing human activity. The anthropomorphism of birds adds layer humor by attributing human thoughts and desires animals.
CONCLUSION
In conclusion, our research highlights the transformative potential of LLaVA-Interlaave in unifying and advancing the capabilities of Large Multimodal Models (LMMs) across diverse visual tasks. By leveraging the interleaved data format, we effectively integrate multi-image, video, 3D, and single- image scenarios, offering cohesive approach to handling thwoese varied challenges. The introduc- tion of the comprehensive M4-Instruct dataset and the LLa VA-Interleave Bench provides solid foundation for training and evaluating LMMs, ensuring high-quality performance across multiple domains. Our extensive experiments validate that -ntal--ne not only sets new state-of-the- art benchmarks in multi-image tasks but also maintains exceptional performance in single-image tasks. Furthermore, the model cxhibits promising emerging capabilities, such as cross-task transfer, showcasing its versatility and potential for broader applications. This work sets new precedent in the field, paving the way for future advancements in multimodal Al and complex visual understand- ing tasks.

---

Published as a conference paper at ICLR 2025
Table 10: An example of emerging capability: Twitter post for videos.
Task Transfer from Image to Video
User
llelp me write twitter to describe this video.
Ours
"Feeling the chill in the air. but the cherry blossoms sight behold! Walking down the street, each person bundled up against the winter chill. The snow-covered trees and market stalls add to the screne beauty of this winter wonderland. #Cherry Blossrss #WinterWalk #CityScenes".
Table 11: An example of emerging capability: recognizing artists' painting styles.
Real-World Applications
User
Do kown who draw these paintings?
Ours
Yes, these paintings are the renowned artist Vincent Gogh. The first painting is his famous work "The Starry Night," which depicts swirling night sky with bright crescent moon and stars. The second painting is his self-portrait, which shows him smoking pipe and wearing a hat. The third painting is also self-portrait, but features a different expression and hairstyle. The fourth painting is by another artist, not Gogh, and depicts person seated with flowers background.
Table 12: An example of emerging capability: PPT summary & VQA.
Real-World Applications
Veniductory
Oyeans Aired
USA Crepell
Worlffl Most
Fire
User
The images are the PPT of iPhone 15 review. can you summarize the main information?
Ours
The PPT features a sleek design with black background and includes text and images related to the iPhone 15. highlights the phone's 's innovative features such as a custom dual glass hack cover, a dynamic island display for delivery tracking, and USB-C compatible charging cable. The color options are showcased of pink, green, gray, and onyx. The presentation also emphasizes the phone's powerful processors, including the A16 Bionic chip and A17 Pro chip.
gradient
User
What the price of iPhone 15 plus?
Ours
The price iPhone 15 plus Rs. 89,000 mentioned the PPT slide.
User
Why the dynamic island uscful?
Ours
The dynamic island useful for delivery tracking. as allows the user know when their order will arrive and there are any delays. also provides new way to interact with phone's 's interface, making easier navigate various features.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprins, 2022.
[2] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308. 01390, 2023.
[3] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 19129-19139, 2022.
[4] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin
Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin. Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang. Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv: 230,,69999,,6 2023.
[5] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. 2.0..:260,.,29666,, 2023.
[6] Ankan Bansal, Yuting Zhang, and Rama Chellappa. Visual question answering image sets. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28,
2020, Proceedings, Part XXI 16, pages 51-67. Springer, 2020.
[7] Guo Chen, Yin-Dong Zheng, Jiahao Wang, Jilan Xu, Yifei Huang. Junting Pan, Yi Wang. Yali Wang, Yu Qiao, Tong Lu, et al. Videollm: Modeling video sequence with large language models. arXiv preprint arXiv:2305. /3292, 2023.
[8] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1wt.vv:8100.4000000 2018.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11909, 2020.
[10] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390 2024.
[11] Peng Gao. Renrui Zhang, Chris Liu, Longtian Qiu. Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.05520 2024.
[12] Google Gemini Team. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 /805, 2023.
[13] Ziyu Guo, Renrui Zhang, Hao Chen, Jialin Gao, Peng Gao, Hongsheng Li, and Pheng-Ann 2024.
Heng. Sciverse. htpp::/seiverse-cubkgggittt
[14] Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.0061
2023.
[15] Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tuning. 2023.
arXiv preprint 2.XX::2309.03990
[16] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482-20485
2023.
[17] Yining Hong. Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models, 2023.
[18] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao
Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need:
11

---

Published as a conference paper at ICLR 2025
Aligning perception with language models. ArXiv, abs/2302. 14045, 2023.
[19]
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi- instruction tuning. arXiv preprint arXiv:2405.01 0005.01433, 2024.
[20] Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H Chen, and Andrew Y Ng. Many-shot in-context learning in multimodal foundation models. arXiv preprint arXiv:2405.08 097998, 2024.
[21] Mehran Kazemi, Nishanth Dikkala, Ankit Anand, Petar Devic, Ishita Dasgupta. Fangyu Liu, Bahare Fatemi, Pranjal Awasthi, Dee Guo, Sreenivas Gollapudi, et al. Remi: A dataset for reasoning with multiple images. arXiv preprint arXiv:2406. 091 2024.
[22] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang. Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelies: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024.
[23] Hugo Laurengon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.0m 27X2:2200.002246, 2024.
[24] Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal capabilities in the
wild, May 2024.
[25] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint
arXiv:23066. 05425, 2023.
[26] Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision- laiiuae understanding and generation. In International Con- ference on Machine Learning, pages I 19888-12900. PMLR, 2022.
[27] Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji. Wengiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Fine-tuning multimodal Ilss to follow zero-shot demonstrative instructions. In The Twelfth International Conference on Learning Representations. 2023.
[28] Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang. Hanwang Zhang, and Yueting Zhuang. Fine-tuning multimodal Ilms to follow zero-shot demonstrative instructions, 2024.
[29] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo. Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint
arXiv:23305.00 00635, 2023.
[30] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding bench- mark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- lion, pages 22195-22206, 2024.
[31] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprini arXiv:2311. 17143, 2023.
[32] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning. Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection, 2023.
[33] Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689-26699 2024.
[34] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embed- dings for multi-modal large language models. arXiv preprint arXiv:2311.07575. 2023.
[35] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved basclines with visual instruction tuming. 2023.
[36] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[37] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. In NeurIPS, 2023.
[38] Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reli- able video narrator via equal distance to visual tokens, 2023.
[39] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt:
Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024. [40] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models, 2024.
12

---

Published as a conference paper at ICLR 2025
[41] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre-training. arXiv preprint arXiv:2403.09611, 2024.
[42]
OpenAI. Gpt-4 technical report, 2023.
[43] OpenAL. GPT-4V(ision) system card. 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khali- dov. Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint prXiv:2304.07193 2023. [45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[44]
[46] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02111 2021.
[47] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 2556-2556 2565, 2018.
[48] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10740-10749, 2020.
[49] Mustafa Shukor, Alexandre Rame, Corentin Dancette, and Matthieu Cord. Beyond task perfor- mance: Evaluating and reducing the flaws of large multimodal models with in-context learning- arXiv preprin! ar3iv: 23X00:23/0.00647, 2023.
[50] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv:1811 111111110099 1.00491, 2018.
[51] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in- context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14398-14400 2024.
[52] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[53] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.0920
2023.
[54] Fei Wang, Xingyu Fu, James Y Huang. Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding, arXiv preprint arXiv:24406009 09411.991111.0. 2024.
[55] Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A bench- 2024. [56) Haoning Wu, Zicheng Zhang. Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang. Chunyi
mark for situated reasoning in real-world videos. arXiv preprint arXiv:2405.0977
Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: A benchmark for general-purpos foundation models on low-level vision. arXiv preprint arXiv:2309. 1e..239991111.... 2023.
[57] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777-9786 2021.
[58] Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. arXiv preprint XrXiv:2308. 16911, 2023.
[59] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In P Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127-9134,
2019.
[60] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-diseipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF
13

---

Published as a conference paper at ICLR 2025
Conference Computer Vision and Pattern Recognition, pages 9536 05676995672200 2024.
[61]
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin. Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipling multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311. 1.6502,, 2023.
[62] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for lan- guage image pre-training. In Proceedings of the IEEE/CVF International Conference Com- puter Vision, pages -1975-1986, 2023.
[63] Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng. Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference op- timization of video large multimodal models from language model reward. arXiv preprint WiXiv22404.01255 2024.
[64] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. LLaMA-adaateed Efficient fin-ttnnig of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representa- lions, 2024.
[65] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal I1m truly see the diagrams in visual math problems? arXiv preprint arXiv:2403. 14624, 2024.
[66] Renrui Zhang. Xinyu Wei, Dongzhi Jiang, Yichi Zhang. Ziyu Guo, Chengzhuo Tong, Jiaming
Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, et al. Mavis: Mathematical visual instruction tuning. arXiv preprint arXiv:2407 087399 2024.
[67]
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024.
[68] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang. and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36, 2024.
14

---

Published as a conference paper at ICLR 2025
A DATA STATISTICS
The detailed data statistics of M4-Instruct is shown in Table 15.
The detailed data statistics of LLaVA- LLaVA-Interleave Bench is shown in Table 16.
B ABLATION STUDY
B.1 POOL vs NOT POOL VISION TOKENS FOR VIDEO TASKS.
Similar to LLaVA-NEXT-Videw we adopt a "Pooling to 1/4" strategy for which we pool the width and heighs of feature maps 1/2 therefore reducing the number totals 1/4. We study the im- pact of image token pooling. We train and infer our model under two settings: pooling to 1/4 and not pooling with ShareGPTVidee Caption+ttaaSSSS CiniiPPViiee-Caiioon--AAAAA QA(255K) data. Pooling to a 1/4 setting similar to LLLVA--EXTE Video, which uses the pooling technique trade-off between the number of image tokens and the number of frames. In our experiment, we find that not pooling yields better perfor- mance under similar #image tokens. During training, sample 10 frames for videos. In this table, we also observe that adding more frames (from 10 to 16) during inference improves performance.
B.2 IMPACT OF VIDEO DPO TRAINING ON OTHER TASKS.
In Table 14, we compare the results of doing video DPO on other tasks. Though DPO significantly improves the video performance as shown in Table 2, it slightly impacts the performance of other tasks.
Table 13: Ablation to compare pooling and not pooling.
Training
Inference
#frames
# Image tokens
Act
Avg
VDD
VideoChanGPT
CU
TU
CO
Pooling 1/4
Pooling 1/4
40
40x729x 1/4-)/1///1100
52.8/3.53
3.35
3.38
3.46
3.25
3.87
2.59
3,57
Pooling 1/4
Pooling 1/4
16416799
52.7/3.53
3.86
3.55
Not Pooling
Not Pooling
10x729
52.9/3.48
3.38
3.46
3.43
3.26
3.85
Not Pooling
Not Pooling
16
16x729
54.4/3.51
3.41
3.46
3.48
3.28
3.87
2.74
3.62
Table 14: Ablation on the impact of video dpo on the performance of other tasks. QB: Q-Bench,
Act: ActivityNet-QU QA,, MVB: MVBench, VDD: Video Detailed Description, MME*: Throughout our paper, we convert MME's s score to accuracy by summing up the perception and cogmition scores and dividing 2800, SQA: Scienceqa-IMG.
Multi-image
Multi-viow
Single-image
Setting
Mantis
BLINK
NIVR2
ScanQA
AEDD
ChariQA
DocVQA
MME
POPF:
SQA
Before Video-DPO
62.7
52.7
73
89.1
32.2
73.9
67.2
75.7
63.5
85.4
72.6
Afer Video DPO
60.8
51.7
86.8
87.7
25,5
72.2
56.1
73.1
62.6
86.6
71.7
15

---

Published as a conference paper at ICLR 2025
Table 15: M4-Instruct detailed datasets.
Task
Dataset
Scenario
Sumples
Multi-Image Scenarlos
Spot the Difference(42.66)
Real world Difference
Realistic
6.7K
Synthetic Difference
Synthetic
7.0K
Spot-the tiifiiff
Surveilance
10.8K
Birds-to- Words
Birds
14.2K
CLEVR CLEVR-Change
Solids
3.9K
Image Edit Instruction667.76 (
HQ-Edit
Synthetic
50K
MagicBrush
Realistic
14.2K
IEdit
Realistic
3.5K
Visual Story (7.5))
AESOP
Cartoon
6.9K
F
Cartoon
22.3K
PoromSV
Cartoon
12.3K
VIST
Realistic
26K
Text-rich VQA(2133K)
WehQA
Wehpage
9.3K
TQA
Textbook
OCR-VQA
OCR
1.9K
DocVQA
Document
1.9K
Multi-image VQA(153.5K)
NLVR2
Realistic
86.4K
TTT-StatesSStateCoiiee
General
MIT-SSaaes.PropettyCCeeeee
MIT-States PropertyCoherence
General
1.9K
RecipeQA ImageCoherence
Recipe
8.7K
VISION
Industrial
9.9K
Multi-VQA
General
5K
IconQA
34.6K
Low-level Comparison(65..9g KK)
Coinstruct
Low-level
50K
Dreamsim
Low-level
15.9K
Image-caption Comprehension (41.8)
ImageCoDe
General
16.6K
Contrast Caption
General
25.2K
Daily Scenarios (5.7K)
MMChat. MMChot_TWitter_Pon
General
5.7K
Multi-image Puzzle (35K)
Raven
Abstract
35K
Multi-frame (Video) Scenarios
Video QA(246.KK)
NEXT-QA
Generul
3.9K
STAR
General
ShareGPTVidoo-QAA
General
240K
Video Detailed Captioning (15K)
ShareGiPTViddo Caption
General
15K
Multi-view (3D) Scenarios
Scone VQA(45.4K)
Nuscenes
Outdoor
9.8K
ScanQA
Indoor Realistic
25.6k
3D-LLM-Soene
Indoor Realistic
10K
Embodied VQA(62.5K)
ALFRED
Indoor Synthetic
22.6K
3D-LLM-Dialoge
Indoor Realistic
20K
3D-LLM-
Indoor Realistic
19.9K
Single-image Scenarlos Single-image (307))
Randomly sampling 40% SET data VEXTTT
General
307K
16

---

Published as a conference paper at ICLR 2025
Table 16: LLaVA-Interleave Bench detailed datasets.
Task
Dataset
Scenario
Samples
In-domain Evaluation Newly Curated Benchmarks
Spot the Difference(O.3K)
Spot-the-Did Diff
Surveilance
0.IK
Birds-t--Word -oosdd
Birds
0.1K
CLEYR-Change
0.1K
Image Edit Instruction(2K)
HQ-Edit
Sythentic
MagicBrush
Realistic
0.9K
IEdit
Realistic
0.1K
Visual Story Telling(O.4K)
AESOP
Cartoon
0.1K
FlintstoneSSV
Cartoon
0.1K
PororoSV
Cartoon
0.1K
VIST
Realistic
0.1K
Text-rich VQA(0.4K)
WebQA
Wehpage
0.1K
TQA
Textbook
0.1K
OCR-VQA
OCR
0.1K
DocVQA
Document
0.1K
Multi-image VQA(0.4K)
MTT-States StateCoherencc
General
0.1K
MtTeStates Property Coherence
General
0.1K
RecipeQA ImageCoherence
Recipe
0.1K
VISION
Industrial
O.1K
Puzzle (1.4K)
Raven
Abstract
1.4K
In-domain Evaluation Existing Benchmarks
NLVR2 (7K)
NLVR2
Realistic
7K
Q-Bench (IK)
Q-Bench
Low-level
IK
Out-domain Evaluation Newly Curated Benchmarks
MathVerse-mn (0.8K)
MathVerse (Vision Dominant) SciVerse (Vision Dominunt)
Math Diagram
0.8K
SciVerse-mv (0.4K)
Scientific Diagrum
0.4K
Out-domain Evaluation Existing Benchmarks
Mantis-Eval (0.2K)
Mantis Eval BLINK
General
0.2K
BLINK (1.9K)
General
1.9k
MMMU-mv (test) (0.8K)
MMMU
Scientific Diagrum
0.8K
17

---

Published as a conference paper at ICLR 2025
C
MORE REAL-WORLD APPLICATIONS
Table 17: An example of emerging capability: multi-doc VQA.
Real-World Applications
Oprah Winfrey
STEVE JOBS
PROS DOPGRENCE
MISSION
STRUGGLES LEARNED FROM
Oyych DOvE
coppinn and aneo
ENTREPRENEURAL EXPERIENCE
Chinweree OLD
D01E
SAIEA 121
MOST PROLO OF
Hest Proc
Nelond
Carine
User
Hi, there two candidates, you provide brief description for each of them for me?
Ours
Certainly! Here's brief description of cach candidate: 1. Oprah Winfrey: Professional Experience: Known for her talk show, "The Oprah Winfrey Show," and her work on "COVID-19" as presenter on Apple TV. Skills: Emphasized on emotional intelligence, interpersonal skills, and creativity. Honors Awards: Notable achievements include the Golden Globe Award Best Television Series and the Golden Globe Award for Best Television Special. Steve Jobs: Professional Experience: Co-founder Apple Inc.. where he led the development of iconic products like iMac. iPod. iPhone. Skills: Highlighted thinker with focus on innovation and problem-solvvng. Honors & Awards: Recognized with numerous accolades, including the Arthur Fellowship and the NeXT Computer Science Award. Please note that these descriptions are bused on the information provided in the images and do not include any personal details or specific accomplishments mentioned in the images.
User
Which of them better suits the role of the Congressman of the US? Why?
Ours
The first resume, fcaturing Oprah Winfrey, would better suit the role of Congressman because it highlights skills and experiences that are relevant to political office, such us leadership. public speaking, and media ex- perience. These qualities are essential for politician to effectively represent their constituents engage legislative processes. second resume, while could suitable creative entreprencurial role, does specifically mention skills experiences that align with the responsibilities of Congressman.
18


Reviews:
[{"review_id": "5zjm5yTt7H", "reviewer": "Reviewer_bKSe", "summary": "The authors collect a interleave-formatted dataset from existing multi-image, video, multi-view, and image datasets. After that, they finetune a LMM from an existing single-image LMM to support multiple multi-image scenarios based on their collected dataset. Moreover, they curate a multi-scenario interleave benchmark to comprehensively evaluate LMM’s interleave capabilities.", "questions": "See weakness above", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In addition to the clear writing and logical flow of this work, I have outlined its strengths below:\n\n1. This work presents a solid benchmark for comprehensively evaluating the interleaving capabilities of LMMs. This does help the LMM community.\n\n2. The experiments in this work are thorough, providing an excellent baseline for future research in this field.", "weaknesses": "In my view, the technical innovation of this work is limited. I have outlined more detailed reasons below.\n\n1. The proposed M4-Instruct dataset is merely a combination of several existing datasets, lacking sufficient technical innovation in its construction process.\n\n2. The model proposed in this paper adopts the architecture and even checkpoints of previous methods, with only fine-tuning applied to the collected dataset. Technical innovation in the training process is also quite limited.\n\n3. Although this work primarily follows a path of data collection and model fine-tuning, it only validates the data's effectiveness on a single model, LLaVA-Next. The authors should supplement their study by applying the M4-Instruct data to other 7B-sized single-image LMMs, such as InternVL-1.5, Cambrian, and MiniCPM-V2.5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "39sJPdOUvJ", "reviewer": "Reviewer_5CHq", "summary": "The paper proposed llava-next-interleave, which unifies the muliple image, multiple frame (video), mulitple view (3D), and Multiple patches (single-image) scenarios in one MLLM, which is very interesting.\nOne dataset namely M4-Instruct dataset and the LLava-interleav Bench is proposed, which is very helpful for pushing the progresses of the MLLM research.\nDetailed ablation studies are provided, where some useful and meaningful insights are provided.", "questions": "Check detailed information in the Weakness part.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper makes attempt to unify the visual format, such as  multiple images, multiple frames, multiple views, and mulitple patches into one signal MLLM, which can facilitate the application of MLLM and improve generalization ability of MLLM on processing different format visual signals. With such unification, different visual formats only need to be processed with the corresponding formate, which can thereby be learned and tuned on the MLLM.\n\nThe composed dataset and benchmark are very useful for the research of MLLM.\n\nSome ablation studies are conducted which are very interesting, such as the mixed interleaved data formats during training, as well as combining differenet data scenarios improves individual task performance.", "weaknesses": "Some insights may be further studied. \n\n1. Mixed interleaved data formats can help improvign the perfromances. I am wondering the underlying reasons. One reason may be the different formats increases the diversity of the data. One reason may be the deficiency of the data, with mixed formats, the data are trained in two rounds (if I understand correctly), the performances are improved. The authors are highly suggested to performed detailed studies.\n\nPlease specify the experiment settings and also provide the results with the data with both formats. In this case, we can find with the data sufficiency matters for the performance gain.\n\n\n2. The constructed data are trained in the continual training stage. I am wondering if we want to improve the ability of MLLM for handling different visual formats. Should we add more data (different types of visual formats) in the stage-i pretraining, besides the single image-text pair. Or we only perform the training in the continual training stage, the MLLM ablity on handling on multiple images, patches, frames, views can be well improved.\n\nIf possible, add the data in the pretraind stage. I am wondering whether there are still performance gain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vqdhMqSBwf", "reviewer": "Reviewer_jqfW", "summary": "This paper introduces LLaVA-NeXT-Interleave, a large multimodal model (LMM) designed to unify and enhance capabilities across multi-image, video, and 3D tasks while maintaining performance on single-image tasks. The authors propose using an interleaved image-text data format as a universal template to represent different computer vision scenarios. They compile a comprehensive dataset called M4-Instruct, consisting of over 1 million samples spanning four primary domains with 14 tasks and 41 datasets. Additionally, they create the LLaVA-Interleave Bench, a diverse set of benchmarks for evaluating multi-image performance. Extensive experiments demonstrate that LLaVA-NeXT-Interleave achieves state-of-the-art results across various benchmarks and exhibits emerging capabilities such as cross-task transfer.", "questions": "Data Curation Process: Can the authors provide more details on the data collection and curation process for the M4-Instruct dataset? Specifically, how were data quality and diversity ensured, and what measures were taken to mitigate potential biases?\n\nComputational Resources: How does LLaVA-NeXT-Interleave perform in terms of computational efficiency compared to other state-of-the-art models? Are there any optimizations implemented to handle the increased complexity of processing multiple modalities?\n\nScalability: Have the authors explored the scalability of the model when trained on even larger datasets or when applied to more complex tasks? Are there any observed limitations or degradation in performance?\n\nGeneralization to Unseen Tasks: While the model demonstrates emerging capabilities, how well does it generalize to completely unseen tasks or modalities not included in the training data?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Innovative Unification Framework: The paper presents a novel approach by using an interleaved image-text format to unify multi-image, video, and 3D tasks within a single LMM framework. This unification is both practical and innovative, addressing a significant gap in the current research.\n\nComprehensive Dataset and Benchmark Creation: The compilation of the M4-Instruct dataset and the LLaVA-Interleave Bench provides valuable resources for the community. These datasets are extensive and cover a wide range of tasks and scenarios, enhancing the reproducibility and applicability of the research.\n\nState-of-the-Art Performance: The proposed model achieves leading results across various benchmarks in multi-image, video, and 3D tasks while maintaining strong performance on single-image tasks. This demonstrates the effectiveness of the approach.\n\nEmerging Capabilities: The model exhibits emerging capabilities such as cross-task transfer and zero-shot task composition, indicating strong generalization and adaptability to new scenarios.\n\nDetailed Experimental Analysis: The paper includes extensive experiments and ablation studies that thoroughly evaluate the model's performance and validate the proposed techniques.", "weaknesses": "Limited Details on Data Curation: While the paper introduces a large and diverse dataset, it provides limited information on the data curation process, quality control, and potential biases. More details on how data quality and diversity were ensured would strengthen the work.\n\nComputational Efficiency Concerns: The paper does not thoroughly discuss the computational efficiency or resource requirements of the proposed model, especially when handling multiple modalities. Comparisons in terms of model complexity and inference speed with existing models would be beneficial.\n\nScope of Evaluation Metrics: The evaluation focuses mainly on quantitative performance metrics. Incorporating qualitative analyses or user studies could provide additional insights into the model's real-world applicability and limitations.\n\nScalability and Practical Deployment: Potential challenges related to scaling the model to larger datasets or deploying it in practical applications are not fully explored. Discussion on these aspects would enhance the paper's impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)