mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models
Authors: Jiabo Ye
Haiyang Xu
Haowei Liu
Anwen Hu
Ming Yan
Qi Qian
Ji Zhang
Fei Huang
Jingren Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: multimodal large language model
long sequence
efficient multimodal understanding
Primary area: foundation or frontier models, including LLMs
TL;DR: A effective and efficient multimodal large language model for various image-text interleaved scenario with hyper-attention

Abstract: Multi-modal Large Language Models have demonstrated remarkable capabilities in executing instructions for a variety of single-image tasks. Despite this progress, significant challenges remain in modeling long image sequences. In this work, we introduce the versatile multi-modal large language model, mPLUG-Owl3, which enhances the capability for long image-sequence understanding in scenarios that incorporate retrieved image-text knowledge, multimodal in-context examples, and lengthy videos. Specifically, we propose novel hyper attention blocks to efficiently integrate vision and language into a common language-guided semantic space, thereby facilitating the processing of extended multi-image scenarios. We conduct evaluations on 21 benchmarks that cover single/multi-image, and short/long video understanding. mPLUG-Owl3 achieves competitive performance with the state-of-the-art methods while reducing inference time and memory usage by 87.8\% and 48.5\% in average. Moreover, we propose a Distractor Resistance evaluation to assess the ability of models to maintain focus amidst distractions. mPLUG-Owl3 also demonstrates outstanding performance in distractor resistance on ultra-long visual sequence inputs. We hope that mPLUG-Owl3 can contribute to the development of more efficient and powerful multimodal large language models.

Published as a conference paper at ICLR 2025
MPLUG-OWL3: TOWARDS LONG IMAGE-SEQUENCE UN- DERSTANDING IN MULTI-MODAL LARGE LANGUAGE MOD-
ELS
Jiabo Ye!
Haiyang Xu
Haowei Liu
Anwen Hu
Ming Yan
Qian
Ji Zhang
Fei Huang
Jingren Zhou
Alibaba Group
(yejiabo.yy yib, shuofeng. ym119608 -alibaaa-iiccaaa.aaaaaaaaaa
https://github.com// http://ggttbb..com/t-o/////////t
Retrieved image-text knowledge
Pac3abize DTTd
multimodal knowledge retrived
Despdeeent
Eisnchi dM LLA newe soereaes
Image I:
is Irish ferrier. Image 2
West Highland
ferrier.
Image 3:
Kerry blue ferrier. Image 4: is Gordon
setter:
9.22
Quastion: Based your deservation photographs above, can you elassify the dog shown
3.5
3.00
1.97
Give me the reason
Marber tepttimgges
dog photographs,
Image classified
dog Image
described as Irish ferrier, and the dog in Image shores similer physical characterisfics with the dog image such as having wiry coot, pointed
distinctive factal structure. other dogs
images, a West Highlend while have different physical features that do not match the dog in image 5.
Kerry herrier,
Pardi
efficiency performance comparison.
Small-scale multi-image scenario: augmented generation.
Operation History
Query: Open Eventbrite. Search for "Language Learning".
Thought: om currenfly leamiing Operation:
Thought: lopped on the search -. "Language Learning". Operations
Thought:
Thought: entered the search ferm "Longuage Leorning" Operation:
successfully show only online events.
Operation: Type (Language Learning)
Tab (877, 492)
(449,
(c)
image internction with
2 hours movie
0:11:40
1:44:36 "1445:55
At the beginning of the mevie, what does the policemen wear their Faces
the post-production segment of the film, color the car lifee by the bulldozer?
At the beginning of the movie, the policamen wear meaks on their faces
lifeed by
(d) large-sccll multi-inage sccnario arii long video understanding
Figure 1: (a) mPLUG-OW13 demonstrates leading performance on video and multi-image under- standing. (b,c,d) Examples of mPL.UG-O113 on handling different scale of multi-image scenarios.
Equall contribution
Corresponding author

---

Published as a conference paper at ICLR 2025
ABSTRACT
Multi-modal Large Language Models have demonstrated remarkable capabilities in executing instructions for a variety of single-image tasks. Despite this progress, significant challenges remain in modeling long image sequences. In this work, we introduce the versatile modti-modal large language model, mPLUG-Ow13, which enhances the capability for long image-sequence understanding in scenarios that incorporate retrieved image-text knowledge, multimodal in-context examples, and lengthy videos. Specifically, we propose novel hyper attention blocks to efficiently integrate vision and language into a common language-guide semantic space, thereby facilitating the processing of extended multi-image scenarios. We conduct evaluations on 21 benchmarks that cover single/multi-iman -mmall, and short/long video understanding. mPLUG-OwI3 achieves competitive performance with the state- of-thhean art methods while reducing inference time and memory usage by 87.8% and 48.5% in average. Moreover, we propose Distractor Resistance evaluation to assess the ability of models to maintain focus amidst distractions. mPL.G- Ow13 also demonstrates outstanding performance in distractor resistance on ultra- long visual sequence inputs. We hope that mPLUG-OWl3 can contribute the development of more efficient and powerful multimodal large language models.
INTRODUCTION
Recently, Multimodal Large Languages Models (Liu et al., 2023a; Ye et al., 2023b; Liu et al., 2024a; Ye et al., 2024; Chen et al., 2024d) have achieved rapid advancements, demonstrating strong lingle-image understanding capabilities. The current approaches primarily rely on vast amounts of image and text data to align Large Language Models (Zheng et al., 2023; Touvron et al., 2023a;b) with visual encoders, thereby extending multimodal capabilities.
More advanced image- sequence understanding capabilities are required in practical applications, such as Multi-Image Reasoning (Suhr et al., 2018; Lu et al., 2021; Jiang et al., 2024), Multimodal RAG (Chen et al., 2022; Lin al., 2024), Video Understanding (Xiao et al., 2021: Li et al., 2023c; Fu et al., 2024a; Wu et al., 2024), Multi-modal Agents (Wang et al., 2024; Zhang et al., 2024a), and Multi-Doc QA (Tito et al., 2023; Van Landeghem et al., 2023). The existing methods are primarily based on interleaved image-text web data for pre-training (Laurençon et al., 2023; Laurençon et al., 2024) to extend multi-image capabilities or focused on the in-context abilities (Alayrac al., 2022; Awadalla et al., 2023; Zhao et al., 2023; Yang et al., 2023; Peng et al., 2024) within multi-image scenarios. However, these methods have not explored the in-depth comprehension or the efficiency of multi-images sufficiently, which makes it hard to support long image sequences. For example, LLAVAA-Next-nnerllee (Li et al., 2024) and IDEFICS2 (Laurençon et al., 2024) insert raw or compressed visual features into textual sequences. As shown in Figure (a), the inference latency increases dramatically as the number of images increases. This design also costs more memory and hinders these models from processing more visual inputs.
To address this challenge, we introduce mPLUG-OWl3, a new general-purpont multi-modal foun- dation model. mPLUG-Owl3 is designed to handle long image sequences both effectively and efficiently. mPLUG-OW13 integrates innovative hyper attention blocks in the language model to achieve efficient interleaved vision-language semantic alignment. Specifically, Hyper Attention in- troduces cross- aosse-attnnionn parallel to the self-attention in the transformer block. The language query reused select and extract visual features from a lengthy visual sequence, allowing for adap- tively obtaining complementary visual information that the language model lacks, based on textual semantics.
As shown in Figure/ 1 (b,c.d), the mPLUG-OWl3, characterized by its high inference efficiency, also capable of handling a variety of tasks involving interleaved text and image modalities, including multimodal retrieval-augmennted generation, multimodal in-context examples, and long video understanding. mPLUG-OWl3 also can handle inputs containing up to 1,000 images at an extremely fast speed surpassing the existing models. We evaluate mPLUG-OW13 with total of twenty-one benchmarks, spanning single-image, multi-image, short, and long video scenarios. Among models of the 8B-level size training publicly available data, mPL achieves
2

---

Published as a conference paper at ICLR 2025
ffte-of-the-art results in 15 out of 21 benchmarks. Besides existing benchmarks, also propose a challenging long visual sequence evaluation named Distractor Resistance. It is designed to assess the ability of models to maintain focus amidss distractions. mPLUG-Owl3 demonstrates outstanding performance in handling ultra-long visual sequence inputs while also maintaining extremely high execution efficiency. can also reduce inference time costs by 87.8% and memory usage by 48.5% in average compared to concatenate-based method. The superior performance of the new architecture in mPLUG-Owl3 implies trend for future multimodal large language models.
MPLUG-OWL3
As illustrated in Figure 2, mPLUG-Ow13 comprises visual encoder, a linear projection layer, and a decoder-only language model. This architecture is commonly employed in recently proposed Multi-modal Large Language Models. Unless specified otherwise, we use Siglip-400m (Zhai al., 2023) as the visual encoder and Qwen2 (Yang et al., 2024) as the language model. First, we provide detailed information about efficient architecture and its handling of various lengths of visual inputs in Section 2.1. Additionally, we introduce the Hyper Attention module in Section 2.2. It is lightweight extension designed to enhance the transformer blocks of the language model by enabling cross-attention capabilities for adaptive visual sequence utilization.
2.1 CROSS-AITENTION BASED ARCHITECTURE
Popular MLLMs (c.g., features into the sequence of embeddings, which easily exhaust the language model's context window, resulting in significant memory and computational overhead. This kind of disadvantage hinders these MILMss modeling the long vision input such multiple images, videos and multiple pieces high-resolution images.
(L., 2024), Intern (Chen et al., 2024d)) insert visual
consider cross-attention for feeding the visual information into the language model. Specifically, given a interleaved multimodal input S = [Ti, J, T2, I2, Ta] (the format can be adapted to various text-image organizational structures), mPLUG-OWL3 first extract visual features of the input images and use a linear projection to align the dimensions of visual features be the same of the language model. The projected visual features are denoted by Hing = [71 I2] E L. The text sequence are Stezt = [T. Timg. T2, Ting Ts] E R Dj where Timg plain text <limage|> indicate the original place of the image. We feed the sequence into the word embedding to obtain text feature Htext-
Therefore, mPLUG-Owl3
In the language model, we fuse the visual features Hing into the text features H R.XD, of the ith layer through cross-attention operator. Different from Flamingo (Alayrac al., 2022) and EVLM (Chen et al., 2024b) that insert an additional layer into each layer of transformer layer, we sparsely extend small number of transformer blocks the network to perform cross attention parallel with self-attention. We name this the Hyper Attention Transformer Block (HATB). We discuss the design HATB detail in Section 2.2. HABT can significantly reduces the number of additional training parameters and facilitates model convergence. Besides, we observe that having fewer HATBs does not degrade the model's performance; instead, it offers the advantages of low memory consumption and high inference efficiency during inference. For language model consisting of Ñ layers, we start from layer O and uniformly extend K layers to HATB.
2.2 HYPER ATTENTION TRANSFORMER BLOCCK
In this section, we specifically introduce the Hyper Attention Transformer Block used in mPLUG- Ow13. The cross-attention structure employed in Flamingo, as shown in Figure 3 (a), has been widely utilized constructing MILMMs (e.g., IDEFICS (Laurençon et al., 2023), EVLM (Chen et al., 2024b)). However, this structure presents three main drawbacks: it introduces a large number of additional parameters, which results in significant memory and computational overhead: the knowledge learned by the language model cannot benefit the understanding of visual inputs; the cross attention does not fully take into account the original positions of images in the interleaved sequence, which limits the performance of these models in multi- scenarios. In response to these issues, we propose a lightweight Hyper Attention Transformer Block, illustrated in Figure (b). This block introduces small number of parameters and extends self- self-attention capabilities
3

---

Published as a conference paper at ICLR 2025
During travel, they plane, the beach, took boaf ride, and went snorkeling. They also stayed the boat for the night and had great time. wee
airport, took
went lunch,
Hyper Attention Transformer Block
Large Language Model
Original FFN
Transformer Block
Hyper Attention Transformer Block
Hyper Attention
Transformer Block
Original Layernorm
Transformer
Visual inputs
Hyper Attention Transformer Block
Sequence Patch Features
Sequence Textual Features
dimage_
eiaage_22
cimage_ 3b
ciraage 4
Linear Projection
Word Embedding
Visual Encoder
Video
Tell me what they did this video climagel> during travel order. Replace with piace holders
Tell what they did this video
during the travel in time order.
Interleaved multimodal Inputs
Single-Image
Multi-Iman
Step dimagel Step dimagel>
Step 3: <limagel Step 4; <limagel> Search the results for todayš
<limagel> Does the man wear a hat?
Lakers and then create note to write news for this results.
Figure 2: An overview of mPLUG-OW13.
perform both intra-text self-attention and inter-modal cross-attention between text and images in parallel. It also introduces Multinodal-Interlenwee Rotary Position Embedding (MI-Rope) to maintain the position information of images. The extended modifications are detailed below:
Shared Input Layernorm. The visual feature Himg and the 7th layer's text features H although sharing the same dimensionality, originate from different distributions. Hence, both sets of features initially normalized using LayerNorm module. Our findings indicate that employing the LayerNorm module already integrated within the transformer block results in better convergence compared training separate layer normalization module specifically for the visual features.
SSpelity-Speciffc Key-Value Projection. In cross-attention, the Query is derived from textual data, while the Key and Value are extracted from visual features. Inspired by Ye et al. (2024), construct a weight matrix W K&V E RDDxD to generate the Key and Value for the visual features. This matrix initialized using the weights from the language model's 's KV (Key-Value) projection. Furthermore, the query vector from the self-attention mechanism is repurposed as the Query in the cross-attention. The computation procedure for self-attention remains unchanged. This design beneficial as it preserves more specific visual information and allows for the adaptive supplementation of visual information that the language model lacks, based on textual semantics.
Visual Position Modeling in Attention. For models that process multiple images, positional en- coding is essential to correctly understanding interleaved image-text input. Existing cross-atte cross-attention models, such as Flamingo (Alayrae et al., 2022) and IDEFICS (Laurençon al., 2023), do not assign
4

---

Published as a conference paper at ICLR 2025
Flamingo Tronsformer Block
Hyper Attention Transformer
FFN
Seff-Attention
Hyper Attention
4
Adeotive Gate
11
FFM
Aftientoon Leperttor
Artantion
operetor
Attention
operetor
Wissi
imputs
loyernore
layernorm
Visual Inputs
Hong
Heext
Staxt
Timg
T2.
Tangs
(a) Flamingo Transformer Block
(b) Hyper Attention Transformer Block
(6) Causal Attention Mask of Cross-Attentin
Figure 3: Comparison between Flamingo Transformer Block (a) and Hyper Attention Transformer
Block (b). Pink indicates that the module is additionally introduced. (c) presents the causal attention mask strategy of cross attention in Hyper Attention image-text interleaved scenario. The gray block denotes the attention score is ignored. Timg denotes the token of plain text <limage|>.
position embeddings to visual inputs, leading to suboptimal performance in scenarios involving mul- tiple images. To accurately represent the original positions of images in interleaved sequences, we develope eantimoddl-Inteeleaaeedd Rotary Position Embedding, which we name MI-Rope. Specifi- pre-record the position index of its placeholder T Timg
cally, for each visual feature In of image the interleaved sequence Stent All patches of I, share T, 's positional encoding to obtain the rotary embedding. This ensures that the positional encoding of the image not only reflects the order among images but also reveals its position in the textual context. We also use a causal attention mask in cross attention. As shown in Figure (c), for text sequence S = [Ti, Timg: T2, Ting: T3}, each text token can only attend the visual features that precede it. Then, HATB simultaneously performs cross-attention and self-attention, denoting the resulting hidden states as Hi and A².
Adaptive Gating Existing implementations of cross-attention utilize a learnable scale to regulate the extent of information transfer from the image to the language model. However, the semantics of language are ignored. Consequently, we introduce an adaptive gate that obtains the gate value based on the textual features. Gate value of each token is obtained by: g = Siggnidd ( gate ft'). Then, the multimodal features are fused: H Hiext g Hi text (1 (l-g). g). The H is passed to the FFN and fed to the next layer of transformer.
3 IMPLEMENT DETAILS
3.1 TRAINING PARADIGM
We adopt three-stage training approach for mPLUG-OW13. Initially, pre-train mPLUG-OW13 using image-text pairs to achieve robust multimodal alignment. In the second stage, we leverage diverse datasets that include image and video captions enhance the model's ability to understand multiple images. Finally, we fine-tune mPLUG-OWl3 using a mixture of supervised data, encom- passing tasks involving both single and multiple images, to ensure comprehensive performance. The statistics the datasets, the training settings and data processing details can be found in Appendix A. We train three sizes of mPLUG-Ow13, based on Qwen2 with sizes of 0.5B, 1.5B, and 7B, All three models share the same visual encoder.
3.1.1 PRE-TRAINING
We follow the mPLUG-Ow12 (Ye et al., 2024) to collect the pre-training datasets and randomly sample subset consists of 41 million image-texi pairs for pre-training During pre-training, only the newly introduced modules are trainable, which include the linear layer following the vision encoder, the visual KV projection, and the Adaptive Gate in the Hyper Attention Transformer Block.
5

---

Published as a conference paper at ICLR 2025
Model
Param
VQAv2
OK-VQA
GQA
VizWizQA
TextVQA
CogVLM
17B
82.3
64.8
70.4
EVLM-Chat
32B
81.9
64.4
47.3
67.5
Flamingo
81.3
50.6
57.2
54.7
8e-level MLLMs
Idefics1
9B
68,8
50.4
39.3
Flamingo
9B
51.8
44.7
46.3
mPLUG-OWl2
8B
79.4
57.7
56.1
54.5
58.2
LLAVA-1.5
78.5
62.0
50.0
58.2
LLAVA-Next
8B
81.8
64.2
57.6
64.9
VILA-1.5
8B
80.9
61.9
58.7
66.3
Idefies2
8B
80.8
53.5
51.1
58.5
70.4
Mantis SigLIP
74,9
55.4
56.3
42.2
59.2
1B
69.2
40.3
55.7
34.8
53.1
mPLUG-OW13
79.3
53.2
61.0
56.6
62.6
82.1
60.1
65.0
63.5
69.0
Table I: Performance comparison on visual question answering. The accuracy is reported. We use bold mark the highest score and underline mark the second highest 8B-level MLLMs.
3.1.2 MULTI-IMAGE PRE-TRAINING
In the multi-image pre-training stage, we collected three types of data enhance the model's multi- image understanding capabilities: (1) Interleaved data. We utilize sources such as MMDU (Liu et al., 2024c) and M4-Instruction (Li et al., 2024) for multi-image data. Additionally, from LLaVA- Recap 558K, we randomly sample 3 to 6 images and combine their image-caption pairs into an interleaved format create Interleaved Captions. We also consider sampling 4 images and requiring a description of one among them to form Selective Captions.
(2) Text-rich data. We use text reading and key point generation data proposed by UReader (Ye et al., 2023a), enabling the model to reconstruct the text and extract key points contained within text-rich images. (3) Video data. We adopt annotated data from ShareGPTVideo (Zhang et al., 2024b), which includes 900K caption entries and 240K question-answeet caption data from VATEX (Wang al., 2019). During multi-image training, modules except visual transformer are trainable.
instances. We also incorporate Chinese and English video
3.1.3 SNTERVISED-FINETUNNII
In Snpervised-Finctuning stage, mPLUG-Ow13 is trained with an extensive and diverse assembly of instruction tuning datasets aimed at enhancing its instruction-folloiing capability. The datasets include LLVA-SFF-665L (Liu et al., 2024a), The Cauldron (Laurençon et al., 2024), Mantis (Jiang al., 2024), M4-Instruction (Li al., 2024), ALLaVA (Chen al., 2024a), ShareGPTVideo QA 240K (Zhang et al., 2024b), Video Instruct 100K (Maaz et al., 2023), MSR-VTT (Xu et al., 2016) and MSVD Caption (Chen Dolan, 2011). We keep the same training setting as the Multi-image Pre-training stage.
4 EXPERIMENTS
4.1 VISUAL QUESTION ANSWERING BENCHMARKS
We conduct experiments on VQAv2 (Goyal et al., 2016), OK-VQA (Marino et at., 2019), GQA (Hud- son & Manning, 2019), VizWizQA (Bigham et al., 2010), and TextVQA (Singh al., 2019). The detail introduction can be found in Appendix H.1. These datasets are strategically selected to thor- oughly evaluate model's ability to understand and reason across various visual contexts and knowledge domains.
Table presents the comparison results between mPLUG-Owl3 and State-of-the-Art multimodal large language models trained on publicly available data. mPLUG-OWI3 outperforms 8B-level language models in VQAv2, OK-VQA, GQA, and VizWizQA. Furthermore, it surpasses the 32B-

---

Published as a conference paper at ICLR 2025
Model
# Param
MMB-EN
MMB-CN
MM-Vet
POPE
A12D
CogVLM
17B
65.8
69.8
52.8
88.0
63.3
EVLM --Chtt
32B
76.9
76.9
89.7
76.0
8B-level MLLMS
LLAVA-L.5
8B
64.3
58.3
31.1
85.9
S5.5
OpenFlamingo
32.4
14.4
24.8
31.7
mPLUG-Owi2
64.5
36.2
55.7
LLAVA-Next
67.4
60.6
43.9
86.5
66.6
LLAVA-Interae Interleave
8B
39.1
86.8
73.9
VILA1.5
8B
72.3
66.2
38.3
84.4
Idefics2
8B
75.7
68.6
34.0
86.2
72.3
Cambrian
8B
74.6
67.9
48.1
86.4
74,6
MigLiP
8B
68.7
63.1
39.3
84.5
67.3
1B
51.4
44.8
21.4
85.3
46.7
mPLUG-Ow13
2B
65.7
61.8
36.3
87.4
62,6
8B
77.6
74.3
40.1
88.2
73.8
Table Zero-shot multi-modal evaluation on multi-modal benchmarks. The overall scores are reported for evaluation. We use bold to mark the highest score and underline to mark the second highest of 8B-level MLLMs.
parameter EVLM in GQA and VizWizQA. In TextVQA, although mPLUG-OWI3's performance slightly lower than that of Idefics2, it still exceeds that of other 8B models. noteworthy that, despite having 8B parameters, mPLUG-OWI3 exhibits superior inference speed and memory efficiency compared to models of the same scale, thanks to the introduction of Hyper Attention.
4.2 GENERAL MLLM BENCHMARKS
We evaluate mPLUG-Owl3 various single-image general multimodal large language model bench- marks including MMBench (Liu et al., 2023b), MM-Vet (Yu et al., 2023), POPE (Li et al., 2023d) and A12D (Kembhavi al., 2016). Please refer the Appendix H.l for more details.
Table shows that mPLUG-OW13 achieves state-of-the-art performance on MMBench-EN, MMBench-CN, MM-Vet and POPE across 8B-level models. It also matches or surpasses the performance of larger models such as CogVLM (Wang et al., 2023) and EVLM-Chat (Chen et al., 2024b). mPLUG-Owl3 does achieve state-of-the-at performance on the A12D dataset. Due to limited training resources, we do not fine-tune the vision encoder, which restricts its performance in scenarios rich text.
4.3 MULTI-IMAGE AND VIDEO BENCHMARK
We also evaluate the performance of mPLUG-OW13 on video and multi-image benchmarks (in- cluding NexIQA (Xiao al., 2021). MVBench (Li al., 2023c) VideoMME (Fu al., 2024a), Long VideoBennh (Wu et al., 2024) and MLVU (Zhou et al., 2024)), as it is capable of processing multiple images with an interleaved format. The introduction of benchmakrs and sample setting can be found in Appendix H.2.
The results of video evaluation is shown in Table On benchmark that measure the short video understanding performance, mPLJU-OW11 achieves performance comparable to state-of-the-ae models. And on long video benchmarks, mPLUUGOWW3 significantly outperforms existing models and present more significant advantage in understanding long videos. The results demonstrates that mPLUG-OWI is highly suitable for understanding videos with various durations. We also present qualitative results video understanding in Appendix G.2.
Table 4 presents the the evaluation results on multi-image understanding. The evaluation benchmarks consist of NLVR2 (Suhr et al., 2018), Mantis-Eval (Jiang et al., 2024), Math (Li et al., 2024), Sci Veiveere-mm mn (Li et al., 2024), BLINK (Fu et al., 2024b) and Q- Bench2 (Zhang et al., 2024c)). We provide detail introduction in Appendix H.3.

---

Published as a conference paper at ICLR 2025
Model
Param
NexIQA
MVBench
VideoMME w/o sub
Long MLVU
VideoChat2
8B
68.6
51.9
43.8
36.0
47.9
Video-LLaMAL
8B
54.6
47.9
48.5
Video-ChatGPP
8B
32.7
31.3
ShareGPT4Video
8B
39.9
39.7
46.4
PLLaVA
8B
46.6
40.2
Idefies2
8B
29.7
25,9
49.7
49,8
SigLLP
8B
50.2
44.0
47.0
49.3
LLAVA-Interleave
8B
78.2
53.1
48.7
48.8
56.4
65.3
47.1
41.8
52.7
mPLUG-OW11
2B
74.5
48.4
8B
78.6
53.5
63.7
Table 3: Multi-modal evaluation on video understanding benchmarks. The overall scores are reported for evaluation. We use bold mark the highest score and underline mark the second highest.
mPLUG-Owl3 surpasses existing models in both NLVR2 and Mantis-Eval. On Math Verse-mv and SciVerse-mv, it can be observed that mPLUG-Ow13 significantly outperforms Interllavee However, on BLINK, mPLUG-OwI3 performs weaker than dataset requires models to possess low-level visual perception capabilities for fine details images, and mPLUG-Owl3's ability may be limited due to the vision encoder being frozen during training. On the Q-Bench2, which evaluates a model's s ability to discern low-level visual differences across multiple images globally, mPLUG-OWl3 achieves performance comparable to the state-of-the-au We also conduct fine-graines multi-image experiments and present the result Appendices and C. mPLUG-Ow Gw333 exhibits stronger capability for following external multimodal knowledge. Addition- ally, we also present qualitative results the multi aalti-image understanding scenarios Appendix G.1.
We note that this
Model
# Param
NLVR2
Mantis Eval
Verse-ere
SciVerse-mi my
BLINK
Q-Bench2
CogVLM
17B
58.6
45.2
41.5
53.2
VideoLLaVA
8B
56.5
35.9
38.9
45.7
VILA
8B
76.5
51.2
39.3
45.7
Idefies2
8B
86.9
48.9
45.2
57.0
Mantis-SigLIP
8B
87.4
59.5
46.4
69.9
LLAVA--nterlcavve
8B
88.8
62.7
32.8
31.6
52.6
74.2
1B
75.8
39.6
23.9
26.4
40.6
50.6
mPLUG-OwL3
2B
85.5
54.4
42.5
30.0
43.0
65.8
8B
90.8
63.1
65.0
86.2
50.3
74.0
Table 4: Multi-modal evaluation multi-image understanding benchmarks. The overall scores are reported for evaluation. We use bold to mark the highest score and underline to mark the second highest.
4.4 ABLATION STUDIES
In this section, we investigate the integration method and design of Hyper Attention. We also validate the integration density and the results presented in Appendix For ablation study experiments, we adopt the training methods of LLaVA-1.5 (Liu et al., 2024a) using the same datasets to conduct our ablation study. Additionally, we employ the Qwen1.5 7B as our language model. To validate the single-image understanding capabilities of our structures. we use datasets such as GQA and TextVQA (with OCR). We conducti zero-shot evaluations on benchmarks including MvBench, VideoMME, NLVR2, and Mantis-Eval to examine the generalization capabilities in multi-image understanding and video comprehension.
4.4.1 CROSs ATTENTION LNTEGRATION
There are two primary methods to integrate Cross-Attention into the transformer block: one method positions it prior to the self-attention (referred Pre-Cross-Attentionn), while the other places it
8

---

Published as a conference paper at ICLR 2025
Attention Structure
Elapsed Time (ms)
VRAM (GB)
GQA
TexVQQ
MvBench
VideoMME
NLVR2
Mantis-Bval
Concalenate
2761.7
40.6
59.0
51.6
22.4
25.1
55.7
38.7
Pre Cross Attention
384.3
21.2
53.8
45.2
43.0
38.9
55.3
44.7
Soat-CrossAAteattie
397.2
21.2
48.9
40.9
38.3
37.0
54.0
47.0
Hyper Attention
59.5
51.6
Table 5: Comparison between different attention structure. Concatenate means direct concatenate visual and text feature sequences. We use bold to mark the highest score.
Adaptive Gating
Shared LayerNorm
MI-Rope
GQA
TextVQA
MvBench
VideoMME
NLVR2
Mantis
53.3
44.6
40.2
38.1
52.7
41.9
55.7
49.3
43.2
40.1
53.4
47.9
58.1
49.7
42.8
38 4
54.9
46.1
57.6
50.0
42.8
39.4
59.5
51.6
Table 6: Ablation on the Adaptive Gating, Shared LayerNorm and MI-Rope.
subsequent to the self-attention (referred as Post-Cross-Attention) We analyze both configurations and compare them to the concatenate- based method and our novel Hyper Attention in mPLUG-Ow13.
model which directly embeds image features into the input sequence of the language model, has the best performance in single-image understanding, On the other hand, utilizing Bost-Cross-Attentii results in the worst performance. Comparatively, Pre- Cross-Attente performs better but still incurs achieves comparable performance with concatenate-basse model. In evaluations involving videos and multiple images, observe that the concatenate-base model demonstrate poor performance. This is attributed to the inadequate training of inter-image attention, which significantly disrupts the model's hidden states. Conversely, both single images and multiple images share the same paradigm when performing cross attention with text, which allows its multi-image capability to be better generalized from single-image training. the Hyper Attention design stands out as particularly effective in balancing the model's capabilities for handling both single and multiple images, showcasing superior generalizability. also reduces inference time costs by 87.8% and memory usage by 48.5% in average compared to concatenate-be method.
Table 5 shows that the concatenate-
performance loss. Hyper Attention, however,
4.4.2 DESIGN OF HYPER ATTENTION
To further investigate the impact of the structural design of Hyper Attention on model performance, we start with a basic hyper attention model and gradually introduce adaptive gating, shared layer- norm, and MI-Rope. The Table 6 shows that when incorporate adaptive gating, the single-image understanding performance is significantly improved. And if we use a shared layernorm, perfor- mance is further improved. In video scenario, notice that even without any inter-image position encoding, the performance video understanding is also improved, suggesting the temporality in- herent in visual content also be implicitly modeled by the model with the help of adaptive gating. However, when evaluating models with multiple images, the contextual position of the images is crucial and cannot be implicitly modeled. Therefore, can be observed that incorporating adaptive gating and shared layernorm does not lead to performance improvement on multi-image benchmarks. However, with the introduction of MI-Rope, the metrics for various multi-image benchmarks have demonstrated significant improvement.
4.5 DISTRACTOR RESISTANCE IN LONG VISUAL CONTEXTS
We develop a challenge evaluation method to assess the distractor resistance of multimodal models in long visual contexts. Specifically, based on MMBench, where N = 1, 5, 10, 20, 50, 100, 200, 400 and measure the model's accuracy. The details of the experiment setting and analysis can be found in Appendix D.
randomly insert N Distractor images,
In Figure 4, LLaVA and Mantis struggles to consistently answer the questions accurately when 50 distractors images are present, resulting low accuracy rate. In contrast, mPLUG-OwIs only drops to a performance level of 43.09%. As the images increases to 400, the performance mPLUG-Owl decreases to 28.58%. It is still challenging task and valuable for future models evaluate.

---

Published as a conference paper at ICLR 2025
Quen-VtCChet
MPLUG OwiZ
60.00%
Mantielpeffes2
50.00%
LLWAA-Nextinterreese
MPLLIG-OWA Owi3
30.00%
20.00%
10.00%
11020
50
150
200
400
Number of Input images
Figure 4: The performance of interference resistance with long visual context across LLa VexNext- Interleave 7B, Mantis-Idefics2, Qwen-VI,, mPLUG-Owl2 and mPLUG-Ow13.
5 RELATED WORK
5.1 MULTIMODAL. LARGE LANGUAGE MODELS
Multimodal large language models (MLLMs) can perceive visual contents, conduct visual reasoning, and engage in multimodal dialogue with humans.
Models like LLaVA (Liu et al., 2023a) use an MLP to map visual features into the representation space of the language model, and directly concatenate them with the text sequence. These methods can preserve fine-grained visual information but consume a large number of tokens which slows down both training and inference. To reduce the number of tokens, models such mPLUG-Owl (Ye et al., 2023b) and IDEFICS2 (Laurençon et al., 2024) adopt structure similar to Q-Former (Li al., 2023a), compressing the token count proposed embedding cross-attention layers into the language model, integrating visual features into the intermediate representations of the language model. IDEFICS (Laurençon al., 2023) and EVLM (Chen et al., 2024b) have also trained MLLMs based on this structure. This method avoids occupying the context window of the LLM. However, it introduces more parameters and may interfere with the intermediate representations of the pre-trained language models, making the performance of such models often sub-optimal compared to mainstream models.
fixed size. Flamingo (Alayrac al., 2022) first
5.2 MULTIMODAL MODELS WITH INTERLEAVED SUPPORT
Recent research are expanding the capabilities of multimodal models process multiple images inputs.
Video is a special form of multi-image existence, and MLLMs related video understanding treat frames as multiple images with temporal correlation as input. These models such as VideoChat2 (Li al., 2023b) and Vid2 (Cheng al., 2024) introduce extra module to compress temporal frames. ShareGPT4Videe (Chen et al., 2024c) proposes to improve the video understanding by introducing GPT-4 annotated video caption as pretrain data. In general multi-image scenario, including in-context learning, cross image reference, comparison, and reasoning. Flamingo (Alayrac et al., 2022) demonstrates limited in-context learning capabilities, while Idefics2 (Laurençon al., 2024) and capabilities by constructing more refined multi multi-image training data.
(Li et al., 2024) further enhance the model's multi-iman -mmge understanding
6 CONCLUSION
In this paper, we present mPLUG-OW13, a multi-modal large language model that significantly advances the state-of-the-art in handling both single-image, multi-image and video tasks. The intro- duction of novel Hyper Attention enables the mPLUG-OWl3 to maintain the fine-grained visual input and effectively fuse visual and textual information, leading to superior performance across various benchmarks. We also propose a challenging long visual sequence evaluation named Distractor Re- sistance. Notably, mPLUG-Owl3 excels in managing ultra-long visual sequences and demonstrates strong performance in evaluation. We believe that mPLUG-Owl reveals a direction for building efficient and effective modlil large language models.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022. 2, 3, 4, 10
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.013990 2023.
2
Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software
and technology, pp. 333-342, 2010. 6, 21
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp. 190-200, 2011. 6
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.1 1684, 2024a.
Kaibing Chen, Dong Shen, Hanwen Zhong, Huasong Zhong, Kui Xia, Di Xu, Wei Yuan, Yifci Hu, Bin Wen, Tianke Zhang, al. Evim: An efficient vision-language model for visual understanding. arXiv preprint Arxi::2407 /4177, 2024b. 3, 7, 10
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, al. Sharegpt4video: Improving video understanding and genera- tion with better captions. arXiv preprint wxXiv:2406.04332 2024c. 10
Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W Cohen. Murag: Multimodal retrieval-augmented generator for open question answering over images and text. arXiv preprint arXiv:2210.02928, 2022. 2
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEF/VV Conference on Computer Vision and Pattern Recognition, pp. 24185-24198 2024d. 2,3
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temport modeling and audio understanding in video-llms. arXiv preprint arXi:22466.07476 2024. 10
Chaoyou Fu, Yuhan Dai. Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme, mmn: The first-ever comprchessiv evaluation benchmark of multi-modal llms video analysis. arXiv preprint arX1v:2405.2 2024a. 2, 7, 22
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive, 2024b. 23
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision 127:398- 414, 2016. Utthttps / api semanti wachoollcccllll.o arr. org/ org/Corpus /Crrrruusss ID: 9081284 244. 6, 21
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709 2019. 6, 21

---

Published as a conference paper at ICLR 2025
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved immge instruction tuning. arXiv preprint 240.::2005.01483, 2024. 2, 6, 73
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 235-251. Springer, 2016. 71
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2455.02244 2024. 2, 6, 10
Hugo Laurençon. Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023. 2,
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleaae:: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprins arXiv:2407.07899 2024. 3, 6, 7, 10, 23
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ArXiv, abs/2301.12597, 2023a. URL https://///. api. semanticschen
org/ http/CorpussDD 256390509. 10
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 2023b. 10
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video understand- ing benchmark. ArXiv, 1.xx2311.170000 2023c. URL https: /api. www.nticscholo.......o.. org/ Ptg//orppsIDD... NDDD::25562211 2, 7, 7,
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji rong Wen. Evaluating object hallucination in large vision-language models. ArXiv, 100.2255510355.... 2023d. URL https: / /api...... semant commanticcchhoarr /org/ooorrugrri/eo :0::25877400 258740697. 7,21
Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. Fine-grained late- interaction multi-modal retrieval for retrieval augmented visual question answering. Advances in Neural Information Processing Systems, 36, 2024. 2
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv, hoss/2344008455, 2023a. URL htttps://app semanticscholed Ptrg/CorrusD::
258179774. 2, 10
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 26296 2200622666,... 2024a. 2, 6.8
Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, et al. Mibench: Evaluating multimodal large language models over multiple images. arXiv preprint arXiv:2407.15272 2024b. 16, 17
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint 200.022377.0688/ 2023b. 21
Ziyu Liu, Tao Chu, Yuhang Zang. Xilin Wei, Xiaoyi Dong. Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, et al. Mmdu: A multi-turn multi Malli--mage dialog understanding benchmark and instruction-tuning dataset for Ivlms. arXiv preprint arXiv:2406. //33, 2024c. 6
Pan Lu. Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao. Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chur Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:21 10. 13214, 2021. 2
12

---

Published as a conference paper at ICLR 2025
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: To- wards detailed video understanding via large vision and language models. ArXiv, abs/2306.05424, 2023. URL https: / api. aala...emaa semant
org/ wwwgg//oopssDi:ioeeee :2910083322991333... 6
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cyf conference on computer vision and pattern recognition, pp. 3195-3204, 2019. 6, 21
Yingzhe Peng. Chenduo Hao, Xu Yang, Jiawei Peng, Xinting Hu, and Xin Geng. Learnable in-context vector for visual question answering. arXiv preprint arXiv:2406.13) 13185, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019. 6, 21
Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. ArXiv, abs/181 1.0099,, 2018. URL https: //api. wemanticschoooooooeooeeeccccccee
Sttp:/CCopuSI::::: 53178855 2, 7.
Ruben Tito, Dimosthenis Karatzas, and Ernest Valveny. Hierarchical multimodal transformers for multipage docvqa. Pattern Recognition, 144:109834 2023. 2
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13990 13971, 2023a. URL https: /appipppp...///p semant icscholar orrg CorpusID 257225221992 1404. 2
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv, 0a002237700998 2023b. URL https:/ /ttos://app.. cemanticscholl / 25:
Jordy Van Landeghem. Rubèn Tito, Lukasz Borchmann, Michat Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Anckaert, Ernest Valveny, et al. Docu- ment understanding dataset and evaluation (dude). In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 19528-1999 19540, 2023. 2
Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv preprint arXiv:2406.01000 01014, 2024. 2
Weihan Wang. Qingsong Lv. Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models. ArXiv, 0Ms/2311.03077 2023. URL https://apl
wttooorg//oooooooo Corppo :2: 255034288. 7
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the JEEE/CVF international conference on computer vision, pp. 4581-4591, 2019. 6
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. arXiv preprint 245 2024. 2, 7, 22
13

---

Published as a conference paper at ICLR 2025
Junbin Xiao, Xindi Shang, Angela Yao, and Tat seng Chua. Next-qa: Next phase of answering to explaining temporal actions. 2021 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 9772-9781, 2021. URL https://api.
semant i escholarr www.hholrrrrrooooooooooo Corp//rruuusss ID: 234763093
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2288-5296,.2 2016. 6
Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
Xu Yang. Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. Explor- ing diverse in-context configurations for image captioning. In A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural In- formation Processing Systems, volume 36, pp. 40944 Curran Associates, Inc., 2023. URL https::/pproceetee file/ 80 000.030000oeeeeeee33771111616113
neurips
Paper Cowfee--Ceeeecccccceeemeeee pdw.seenneeee..........
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023a. 6, 16
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXivv2:044141778 2023b. 2, 10
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13040-13051, 2024. 2, 4,5
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308 02490, 2023. 21
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-trane mre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11975-11986, 2023.
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, al. Ufo: ui-focused agent for windows os interaction. arXiv preprint arXiv:2402.070 2024a. 2
Ruohong Zhang. Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward. arXiv preprint wrXiv:2404.0/2550 2024b. 6
Zicheng Zhang, Haoning Wu, Erli Zhang, Guangtao Zhai, and Weisi Lin. A benchmark for multi- modal foundation models on low-level vision: from single images to pairs. arXiv preprint arXiv:2402.07116. 2024c. 23
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning, arXiv preprint arXiv:2309.00 07915, 2023. 2
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-hench and chatbot arena, 2023. 2
14

---

Published as a conference paper at ICLR 2025
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task halti-task long video understanding. arXiv preprint (rXx::24000042644. 2024. 7, 22
15

---

Published as a conference paper at ICLR 2025
A TRAINING DETAILS
Stage 1: Pretraining
Stage 2: Multi-Image Training
Stage 3: Self-Supervised Fintuning
Dataset Name
Percentage
Dataset Name
Percentage
Dataset Name
Percentage
DataComp-1B
35.22%
ShareGPTVidee
34.63%
LLAVA-SFT
57.95%
LAION-en
26.07%
Selective Caption
19.29%
The Cauldron
12.50%
COYO-700M
14.47%
M4-Instruction
16.69%
Mantis
10.41%
COYO-700M-OCK
9.60%
VATEX
15.77%
M4-Instruction
9.26%
LAION-zh
7.73%
Text Reading
7.36%
ALLAVA
6.95%
Wukong
5.64%
Interleaved Caption
5.25%
ShareGPTVidde-QAA
2.02%
CC12M
0.81%
MMDU
1.01%
Video Instruct
0.84%
Others
0.46%
MSRTT/MSVD Caption
0.06%
Table 7: Dataset percentages used in Pretraining, Multi-Image Training, and Self- Self-Supervise Fin- tuning. Others include CC3M, OCR-CC, COCO and SBU.
Setting
Stage Pretraining
Stage Multi-Image Training
Stage Self-Supervised Fintuning
Learning Rate (Max, Min)
(le-3, Ie-5)
(2e-5, le-7)
(2e-5, le-7)
Global Batch Size
2048
1024
1024
Training
Training Steps
20K
3K
1KK
Warmup ralio
0.03
Lineur Projection Visual KV Projection Adaptive Gate
Trainable Modules
Lineur Projection Full Language Model
Projection Full Language Model
Resolution
384'
up to 384 x
up to 384 x
Model
Sequence Length
Precision
Mixes-precission Mixed, precision FP16BFI6
Accelerating
Optimization
Gradicat Chockpointing
No.
Yos.
Yos.
Model Parallll
TP-1
TP-4
TP-4
Table 8: The training settings across three stages: Pretraining, Multi-Image Training, and Self- Supervised Finetuning.
A.I TRAINING DATA PROCESSING
For high-resolution images, inspired by UReader (Ye et al., 2023a), we introduce similar adaptive method for image cropping. For given image, we select from cropping grids (2.2), (1.3), (1,4), (3,1), (4.1), (2.3), and (3,2) that most closely matches the shape of the input image. Additionally, we retain global version of the original image. During the Suppervsed-FFinttnme rich in text, perform cropping with probability of 100%. For datasets containing single image without text, we apply cropping with a probability of 20%. For datasets containing multiple images videos, do not perform cropping. During evaluation, cropping is enabled only for single-image tasks.
stage, for datasets
For videos, we sample frames per video by default. Meanwhile, we replace the video markers in the text with multiple <limagel> placeholders corresponding to the number of sampled frames.
B FINE-GRAINED INVESTIGATION OF CAPABILITIES IN MULTI-IMAGE SCENARIOS
To more comprehensively investigate the fine-grained abilities of mPLUG-OW13 in multi-image scenarios, we conduct experiments on MI-Bench (Liu et al., 2024b), a recently proposed multi- image benchmark. We exclude Fine-Grained Visual Recognition from evaluation because it consists of images from mini-ImageNNt that may have been seen by existing models.
Table 9 shown that mPLUG-Ow13 achieves state-of-the-ar performance aspects of General Com- parison, Subtle Difference, Temporal Reasoning, Logical Reasoning and Text-Rich Images across popular open-sourced MILMMs. It also outperform GPT-4V and GPT-4o General Comparison.
16

---

Published as a conference paper at ICLR 2025
Model
GC
SD
VR
TR
LR
TRI
VTK
TVK
Closed-source MLLMs
GPT-40
80.7
90.5
46.8
68.0
69.8
74.8
54.7
63.3
GPT-4V
72.8
79.2
45.8
61.8
66.3
71.0
52.0
56.0
Open-source MLLMs
mPLUG-Ow12
64.2
40.1
35.6
30.7
41.3
39.0
17.0
25.6
MMICI
53.7
46.4
41.1
47.0
59.6
27.6
22.1
35.9
Idefics2-1
83.1
49.7
32.6
44.8
56.4
43.9
25.6
39.0
Mantis
83.0
54.1
37.6
45.5
63.4
37.7
26.4
41.7
mPLUG-OW13
86.4
70.1
33.0
46.8
67.2
50.1
34.0
48.8
Table 9: Multi-image evaluation MI-Bench (Liu et 2024b). We use bold mark the highest score open-sourced multimodel large language models. The evaluation consists of the following tasks: General Comparison (GC), Subtle Difference (SD). Visual Referring (VR), Temporal Rea- soning (TR), Logical Reasoning (LR), Text-Rich Images (TRI), Vision-linked Textual Knowledge (VTK) and Text-linked Visual Knowledge (TVK).
The results demonstrates that our model possesses robust capabilities various multi-image input scenarios. The Hyper Attention structure of mPLUG-Ow13 better preserves the original visual fea- tures, enabling it to excel in single-image tasks as well. And this type of multimodal knowledge also assists it in more accurately completing multi-image tasks.
C VISUAL-LINK KNOWLEDGE ENHANCE
Based on MI-Bench, we further examine the performance in scenarios involving supplementary visual knowledge and present the results Table 10. Specifically, we test the models' performance in the Vision-linked Textual Knowledge task under conditions with and without additional visual knowledge. Since the provided visual knowledge does not always correspond to the images related to the questions, this setup more accurately simulates the performance of models in real-world Retrieval- Retrieval-Augmmented Generation systems. Results show that LLaVA -L achieves higher accuracy without external knowledge, while noisy interleaved text and image inputs severely degrade its performance. In contrast, mPLUG-OW13 is capable of identifying relevant visual knowledge from noisy inputs, thereby enhancing its accuracy.
Models
w/o Visual-link Knowledge w/ Visual-link Knowledge
ILaVA-Interleave mPLUG-Ow13
31.2
28.7 (-2.5)
32.1
34.0 (+1.9)
Table 10: Visual-link Knowledge Enhance experiment between LLtVA--nterleave and mPLUG- Owl3.
D DISTRACTOR RESISTANCE IN LONG VISUAL CONTEXTS
We notice that multimodal models, when modeling multiple images, are susceptible to interference from surrounding images, leading to visual illusions. Therefore, we develop challenge evaluation method to assess the distractor resistance of multimodal models in long visual contexts.
Specifically, we take samples from the MMBench dev set. For each test sample, we randomly select N 11 images from the original MMBench dev set distractor and construct the model input in the format of Image 1: <limage|>> Image 2: <limage|>... Image N: 
. In Image X. /question], where N = 1, 5. 10, 20, 50, 100, 200, 400 and X denotes the index of the image corresponding the question. We use the EiraularEval to measure the accuracy scores. For each question, we construct test samples with different orders of options and varying distractor images. The model needs to
17

---

Published as a conference paper at ICLR 2025
Hyper Attention Layers Indices
GQA
TextVQA
MvBench
VideoMME
NLVR2
Mantis-Eval
99, 27]
55.1
51.3
42.2
38.2
58.3
48.4
1. 1,5,9, 13, 25, 1,,,17,23 27
56.2
48.3
41.5
39.5
52.4
47.5
57.6
50.0
42.8
39.4
59.5
51.6
Table 11: Comparison between different layers for integrating hyper attention structures. We use bold to mark the highest score.
answer all test samples for given question correctly for to be counted as correct. Consequently, as the number of distractor images increases, the evaluation becomes significantly more challenging.
The results are shown in Figure 4. can be observed that the introduction of distractor images results in certain degree of performance loss for all the models. When the number of images reaches 20 and 50, the performance of LLaVA-Next-Interleae 12.52%, respectively. We observe that when the number of images reaches 50, LLaVA struggles to consistently answer the questions accurately when different distractor images are present, resulting in low accuracy rate. And when the number of images reaches 100, Nantis-Idefics2 fails solve most of the problems correctly. In contrast, mPLUG-OW13 only drops to a performance level of 43.09% when processing 50 images. As the number of images increases to 400, the performance of mPLUG-OWI3 decreases to 28.58%. Since our multi-image training data consists of only about 6-8 images, this also presents challenge for our model. Nonetheless, mPLUG-Owl3 can serve as baseline for future research.
dramatically drops to 43.18% and
IMPACT OF INTEGRATION DENSITY OF HYPER ATTENTION
We explore the integration density of the hyper attention. As shown in Table 11 the results indicate that even with just two layers of Hyper Attention, the model achieves impressive performance on single-image benchmarks, while also demonstrating generalization capabilities for videos and multiple images. However, when we apply a denser integration strategy by introducing eight layers of Hyper Attention, we find that it does not yield improved single-image performance at this scale of training data, and its zero-shot generalization is even worse. Therefore, we ultimately integrate only four layers into the entire model.
PERFORMANCE OF SMALL-SIZED MODELS
Small-scale language models can be more easily deployed on the edge, and mPLUG-Ow[3 can further enhance the efficiency of these smaller language models on multimodal scenario. We provide the performance of mPLUG-Ow13 based on 0.5B and 1.5B language models across single-image, video, and multi-image understanding tasks in Table 1, Table 2, Table 3 and Table 4. The results indicate that for single-image understanding, the IB and 2B models exhibit average performance drop of 19. 19.1% and 6.7% compared to the 8B model. Meanwhile, we notice that the 2B-sized model achieving performance comparable to LLaVA-NexI 8B. In the video domain, find that the smaller models do not suffer much performance loss (averaging 10.5% and 3.5%); even the mPLUG-OW13 2B outperforms the 8B scale model on long video understanding tasks. However, in multi-iman understanding tasks, we observe a more significant performance drop for the smaller models, with the IB and 2B models showing declines of 28.8% and 18%, respectively. This because understanding relationships among multiple images is more reliant on the capabilities of the language model.
G QUALITATIVE RESULTS
mPLUG-Owl can handle various number of images and videos as inputs. In this section, we further investigate the ability of MPL.U-OOW3 in real-world dialogue scenarios.
18

---

Published as a conference paper at ICLR 2025
Can the animal in the first image survive the second image? Tell me the answer and why
insect in the image, which cicado, would not be Cicadas are insects that adapted to warm, sunny environments. They require temperature range of 68-86 degrees Fohrenheit for their development and typically found in regions with such climates. The second image depicts snowy, cold environment, which not suitable for the cicada's survival. The cold temperatures would be detrimental to the cicada's physiological processes, and the presence of snow would prevent from accessing food sources and mates. Therefore, the cicada would not survive in the snowy, cold conditions shown in the second image.
survive in the second image.
Which country can be identified in the clothing styles in each image? Tel me one by one
clothing styles in image suggest different countries. first image features woman wearing cheongsam, which traditional Chinese indicating that the country likely China. The second image shows which commonly associated with Mexican culture. The third image depicts two individuals traditional Japanese kimonos, which are traditional Japanese garments, suggesting that the country is Japan.
in traditional Mexican attire, including sombrero and poncho,
Figure 5: Examples for Multi-Image Understanding. highlight the correct answers in green.
G.1 MULTI-IMAGE UNDERSTANDING
mPLUG-Ow13 demonstrate state-of-the-an performance on multi-image understanding benchmarks. In this section, we present Figure 5, can be observed that mPLUG-OW13 can activate knowledge learned based on content of the images and perform cross-image reasoning. The second example demonstrates that the model can accurately distinguish the content of multiple images and respond appropriately based on cultural knowledge.
dialogue examples in real-world. In the first example shown
Figure shows multi-turn dialogue example. mPLUG-Owe can find the differences between two images in various views. Besides, it describe the correlations between images.
As shown in Figure 7, mmLUG-LWE can maintain continuous interaction with a smartphone. During this process, the model effectively understands the multimodal interaction history and makes accurate decisions using chain-of-te -f-thought approach. Additionally, it retains the initial user instructions without any degradation throughout the interaction.
19

---

Published as a conference paper at ICLR 2025
R
What artistic styles are depicted the images and how are they different?
The first image is pixel art style drawing of man wearing cowboy hat, while the second image is colorful cartoon drawing of boy and green tree-like creature. The pixel ort style is characterized by its blocky, pixelated appearance, which is evident in the first image. In contrast, the second image features smooth, vibrant colors and more detailed, cartoonish representation of the characters. The artistic styles differ in their visual aesthetics, with the first image having a retro, digital feel and the second image being more modern and colorful.
appears both pictures? What the characteristics?
first picture, depicted both pictures, there a man wearing a cowboy colorful cartoon style.
pixel style, while in the second picture,
Figure 6: Examples for Multi- turn Multi-Image Dialogue. We highlight the correct answers green.
G.2 VIDEO UNDERSTANDING
We showcase the video understanding capabilities of mPLUG-OW13. First, compare it with LLa VLaVA--ext-nnteeleen in Short Video Question Answering. Long Video Fine-grained Question Answering, and Long Video Comprehensive Understanding. For LLaVA-Next -.Nex--Interleaee we input 8 frames, while for mPLUG-OWI3, we input 128 frames, which are the maximum numbers of images that can be accommodated by the models on V100-32G. The samples shown Figure 8.
In the short video tests, both LLaVA and mPLUG-Ow13 can provide correct answers. mPLUG-Ow13 tends describe the attributes of objects based the actual content seen. In long video lasting more than 40 minutes, when we ask about specific detail, LLaVA fails to handle the long sequence and loses fine-grained information, rendering unable to provide accurate information. On the other hand, mPLUG-OWI3 accurately captures key segment information within long video. Additional, we have both models summarize the content of longer video. mPLUG-OWl3's response very detailed, not only providing an overall summary but also introducing the process in order. LLaVA- Next-Interlean response, however, is more general and lacks detail. The comparative results indicate that mPLUG-Ow!3 not only efficiently encodes long visual sequences but also captures and effectively utilizes both global and local information.
We also test mPLUG-Owl3 in multiple rounds using long video that featuring many scenes. For clarity, we place the relevant segments beside the dialogue in the figure. During the test, we input only the complete video to the model. The dialogue is shown in Figure 9.
First, we ask a question with a temporal constraint, and mPLUG-Ow13 accurately understands the concept of at first" and correctly describes the detail of "sitting in room and discussing something on their laptops." " However, the response incorrectly counts the number of people. The segment has only people. We find that the model is confused by later scene involving people. We
20

---

Published as a conference paper at ICLR 2025
Query: Open Exenthnte Search for Language Leaming
Thought: currently
instructlore oper Exentbrtte search In
topped
Learnigg located the Ssentbrite opp and topsed its icon to span it After thi, the Evertbrite home sereen is dispioydd, where con proceed with my search for unguugge leariigg events.
Eventbrite for specift events My events relsted to "Longuage learning" spectically for "Spartor"" Operation. Tab (449, 45)
searching lask for cnline
Operation: Tob (577. 492)
Sunmary: tapped on the search bar to begin entering my smarch query.
Summary: happed on Eventbrite app ison app drower 1o open
Thought: have suco opaned the Eventorils and toppad on the search ber search for
Learning" events, Now that I'm in the search resutts need to filer these results to show only online events My next sleps are to hop on the "hithrx" lcon to cecees the fileer oprions and refine my snorch to show only online event.
Thought: 2 have antered the search herm Langusge Learning
Operation Step
Summary: searched for 'Language Learning"
Opershen: Type Baangaoge: Learning)
Summory: Eventorile "Language Learning"
loaking
filher the show online
Figure 7: Example of multi-image understanding interactions between mPLUG-OW13 and the smart- phone GUT in multi-round interactions. The tap operation is annotated by red sphere.
also notice that the visual content of this segment does not involve Australia as destination, but the model can infer this from some diagrams later in the video, which makes the response more detailed. Then, we ask about the camera brand in a frame that briefly appears, and mPLUG-Ow13 accurately notices the "Canon" logo in the image and provides the correct answer. Finally, ask the model to describe the travel in order of time. We use the same color to identify the content described by the model and the corresponding video segments. Since the video involves many scenes and events, this poses a great challenge to the model. It can be observed that mPLUG-OW13 accurately details the travel according the timeline of the video. However, we also notice that mPLUG-OWl3 exhibits some hallucinations, incorrectly interpreting the reefs captured in the video as a beach. Additionally, while the activities on the boat happen during the day, mPLUG-OW13, influenced by other nighttime scenes, makes an incorrect statement.
H EVALUATION BENCHMARKS
H.1 SINGLE-IMAGE BENCHMAKRS
We conduct experiments on diverse set of visual question answering benchmarks, including VQAv2 (Goyal et al., 2016), OK-VQA (Marino et al., 2019), GQA (Hudson & Manning, 2019), VizWizQA (Bigham et al., 2010), and TextVQA (Singh et al., 2019). The VQAv2 dataset is currently the largest visual question answering dataset available. OK-VQA involves questions that require external knowledge beyond multimodal inputs. GQA designed reasoning capabilities. VizWizQA is constructed from question-answer pairs sourced from visually impaired users. TextVQA focuses more on evaluating the model's 's ability to understand text natural scenes.
validate the model's
We also evaluate mPLUG-OW13 on various single-image general multimodal large language model benchmarks. MMBench (Liu et al., 2023b) provides a comprehensive evaluation of a model's multimodal capabilities in both Chinese and English contexts. MM-Vet (Yu et al., 2023) assesses the multimodal conversational abilities of model using GPT-4 evaluation. POPE (Li et al., 2023d) can evaluate the extent of multimodal hallucinations in a model. AI2D (Kembhavi et al., 2016) assesses model's 's ability to understand science diagrams inputs.
21

---

Published as a conference paper at ICLR 2025
Short Video Question Answering
00:00
00:02 00:08
00:13
mPLUG- Owl3
What object appeared when he shot the arrow?
LLAVA-Interleave
When the man shoots the arrow, red and white bullseye appears the wall behind
When the character shot the arrow, target bullseye appeared, and an arrow struck the center of the target. This is common sight in archery.
Long Video Fine-grained Question Answering
00:00
07:43
32:28-33:27
44:03
47:16
What instrument does the anime character play in front of the flames?
LLAVA-Interleave
mPLUG-OWL3
The anime character is playing an instrument Front of the Alaes,, it's not specified what instrument it is.
The anime character plays a violin.
Long Video Comprehensive Understanding
00:00
mPLUG-OWE Ow3l3
What you learned from this video
LLAVA-Interleave
In the video, we learn how to make delicious meat and rice dish with a secret ingredient that gives the dish its distinct taste. The man in the video shows the process of cutting and cooking the meat and then cooking the rice in pot. He then combines the meat and rice together, and the final product a delicious dish that is sure to satisfy any taste buds.
video demonstrates various cooking steps, including slicing row meat, roasting or grilling the meat, and preparing dish with roasted grilled ingredients. The specific techniques used these steps are not detailed the caption,
video gives an impression of hands-on, outdoor cooking process.
Figure 8: Comparison between mPLUG-OwL and LLaVA- across Short Video Question Answering, Long Video Fine-grained Question Answering, and Long Video Comprehensive Under- standing. We highlight the correct and relevant parts of the answers in green, while the parts that fail answer the question correctly are marked in red. Additionally, the segments of the video that are relevant to the questions are highlighted with green background.
H.2 VIDEO BENCHMARKS
diverse set of video understanding benchmarks, including Nex- 1QA (Xiao et al., 2021) and MVBench et 2023c), which are short video benchmarks with video durations all less than minute. Thus, sample 8 frames and 16 frames, separately. For bench- marks like VideoMME (Fu et al., 2024a), LongVideoBench (Wu et al., 2024), and MLVU (Zhou al., 2024), which involve longer video durations up hour. For, we
We conduct experiments
22

---

Published as a conference paper at ICLR 2025
00:00
16:40
What are they doing at first
At first, three the video sitting in and discussing something their laptops. They be talking about trip Australia that coming up.
The that the holding the opening belongs which brand?
The the opening is holding Canon camera.
Tell me what they did during the travel time order.
During the travel, they took plane, checked the airport, and took taxi to the hotel. Then they went for lunch, visited went snorkeling. They also stayed on
boat ride, for the night and had great time with the locals.
beach,
Figure 9: Examples of mPLUG-Ow/3's understanding of complex video content
H.3 MULTI-IMAGES BENCHMARKS
We conduct experiments on a diverse set of multi-image reasoning benchmarks, including NLVR2 (Suhr et al., 2018), Mantis-Eval (Jiang et al., 2024), Vath-erse-mv (Li et al., 2024), SciVerse-mv (Li al., 2024), BLINK (Fu et al., 2024b), and Q-Bench2 (Zhang et al., 2024c).
NLVR2 tests the model's ability to perform logical reasoning based on the content of multiple images. Mantis-Eval evaluates the model's reasoning skills complex scenarios involving mul- tiple images. Math and SciVerse-mv assess the model's multi-image mathematical and scientific capabilities. We use the version released by llava-next-nneellave for comparison with its reported results. BLINK and Q-Bench2 test the model's multi-image ability based low-level visual perception.
23


Reviews:
[{"review_id": "cExQoz2kxe", "reviewer": "Reviewer_3S6H", "summary": "This paper introduces a multimodal large language model, mPLUG-Owl3, which is capable of processing multi-image and video understanding through interleaved training. The proposed model adopts the Hyper Attention Transformer Block, which inserts an additional layer into each transformer layer to form a cross-attention-based architecture. The authors train a series of MLLMs of varying sizes based on three stages, and the experiments demonstrate superior performance of these models compared to the baselines.", "questions": "- For the MI-Rope, my understanding is that all patches within an image $I_n$ share the same position embedding as the placeholder $T_{img}$. Wouldn't this setting discard the relative positions of patches within the image?\n  - What is the number of frames and the corresponding frame sampling strategy when applying the proposed model to long video understanding (e.g., LongVideoBench)? Since the model’s maximum sequence length is 4096, wouldn't the large number of frames potentially exceed the maximum length of the position embedding?\n  - Regarding the distractor resistance in long visual contexts, what is the main factor that contributes to mPLUG-Owl3’s superior capability in this task? Could you provide a brief analysis or explanation?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Please see the summary above.", "weaknesses": "- Important details regarding the hyperparameter settings are missing when evaluating mPLUG-Owl3 in long video understanding scenarios (e.g., the frame sampling strategy, the number of input frames for mPLUG-Owl3 and other baselines).\n  - The content in Appendix H.2 is incomplete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8MgEk7dIdx", "reviewer": "Reviewer_RTBV", "summary": "PLUG-Owl3 address a core limitation in existing multi-modal language models: the challenge of effectively and efficiently processing long image sequences. While current models like LLAVA and IDEFICS handle single or limited multi-image scenarios, they struggle with latency and memory overhead as the sequence length increases. To tackle this, mPLUG-Owl3 introduces Hyper Attention Blocks, which incorporate visual and textual information in a shared semantic space through cross-attention parallel to the standard self-attention. This design significantly reduces memory usage and inference time, facilitating a smoother experience with large datasets of images or video frames.\n\nThe model is evaluated across various benchmarks (single/multi-image and video understanding) and demonstrates competitive results, particularly in multi-image and video understanding, where it reduces inference time by up to 87.8% and memory usage by 48.5% compared to traditional concatenate-based methods. Additionally, the authors propose a novel Distractor Resistance benchmark to assess the model’s robustness in long visual sequences containing distracting images, which is essential for real-world applications.", "questions": "1. Based on Table 11, I would like to know why you choose these three indices as the layers to introduce Hyper Attention Layers? Could we have a general insight of where to introduce Hyper Attention Layers?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "mPLUG-Owl3's Hyper Attention architecture shows a substantial improvement in computational efficiency and memory usage, making it a strong candidate for practical applications involving long video or multi-image data. The new benchmark for distractor resistance in long visual sequences fills an important gap in the multi-modal evaluation landscape, addressing real-world challenges where models encounter irrelevant data alongside relevant content. The model's consistent performance across various benchmarks highlights its versatility and robustness. Its capacity to handle up to 1,000 images with high speed and accuracy shows potential for scaling and deployment in extensive visual and textual environments. The adaptive gate and modality-specific components within the Hyper Attention blocks allow selective visual feature extraction, maintaining relevant information while reducing interference from unrelated data. This selective mechanism improves the model's performance in complex multi-modal tasks.", "weaknesses": "1. Lack strong baselines in the benchmark comparison including internVL2 [1] and Qwen2-VL [2]. For example, for TextVQA [3], Qwen2-VL 2B gets 79.7 while mPLUG-Owl3 8B gets 69. Including strong baselines into the paper may help readers see the effectiveness of the proposed model and architecture.\n2. Hyper Attention Transformer Block seems to be a improved Q-former [4] architecture or visual abstractor [5] architecture. However, Q-former architecture has been proven that will inevitably lose visual information based on previous works [6]. Maybe applying a projection to align visual and textual feature would be more effective.\n3. The design of mPLUG-Owl3 seems really improve the long-context multi image understanding ability based on figure 4, but again it may be helpful to compare the model with the most state-of-the-art models. Only then can readers see the real effectiveness of mPLUG-Owl3\n\n[1] https://internvl.github.io/blog/2024-07-02-InternVL-2.0/\n[2] Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution\n[3] Towards VQA Models That Can Read\n[4] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models\n[5] mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration\n[6] DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iz9K6JNmpE", "reviewer": "Reviewer_qTJ2", "summary": "mPLUG-Owl3 is a new multi-modal foundation model that efficiently processes long image sequences using hyper attention blocks. The model can handle up to 1,000 images at high speeds and excels in various tasks including retrieval-augmented generation and video understanding. Among 8B-level models, it achieves state-of-the-art results in 15 out of 21 benchmarks while reducing inference time by 87.8% and memory usage by 48.5% compared to traditional methods.", "questions": "How do the effects of your pretraining and fine-tuning data look without the hyper attention module? Can we attest the performance gain to the hyper attention module? Just want a fair standard of comparison between models you are comparing against since you have different pertaining and fine-tuning methods compared to previous models in addition to the hyper attention module.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides comprehensive coverage of existing benchmarks across various domains (including math reasoning, (NLVR), Mantis evaluations, and others), effectively comparing their hyper attention module against existing methods.\n\nThe authors present clarity in explaining the Hyper Attention module and its distinct differences from existing methods, particularly highlighting its efficiency improvements and architectural advantages.\n\nThe research addresses the important and challenging problem of long image sequence understanding in MLLMs, which is important as current models primarily focus on single-image tasks while real-world applications often require processing multiple images or video frames.", "weaknesses": "The paper compares mPLUG-Owl3 against diverse architectures (Mantis-SigLip, Cambrian, Idefics2) while acknowledging their different pretraining approaches, but doesn't isolate the benefits of Hyper Attention from other variables like training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NeP5Nv8BEn", "reviewer": "Reviewer_KHjw", "summary": "This paper presents a novel multi-modal large language model, mPLUG-Owl3, designed for effective and efficient multi-image understanding. The core design is a novel hyper attention block that integrates visual and textual information. Comprehensive evaluations across 21 benchmarks—including single/multi-image and short/long video understanding—demonstrate the strong performance of mPLUG-Owl3. Additionally, the authors introduce a Distractor Resistance evaluation to assess the model's ability to maintain focus in the presence of distractions.", "questions": "1.\tWhat is the training cost associated with mPLUG-Owl3?\n2.\tHow does mPLUG-Owl3 perform in terms of OCR, given that the hyper attention block could also be beneficial for high-resolution images?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe hyper attention block is an applaudable design that effectively fuses visual and textual information while preserving fine-grained visual input.\n2.\tThe detailed ablations in Section 4.4 convincingly demonstrate the effectiveness of key design choices in the proposed model.\n3.\tThe model shows strong performance across a wide range of image and video benchmarks.", "weaknesses": "1.\tAlthough the proposed model demonstrates strong performance across a broad range of benchmarks, the comparison with other models lacks fairness from a data perspective. It would be beneficial to discuss the influence of data mixing and to establish at least one setting that is directly comparable to existing models.\n2.\tWhat is the impact of joint training on single-image, multi-image, and video data?\n3.\tWhy does mPLUG-Owl3 freeze the vision encoder, considering that this design choice may adversely affect performance on BLINK, as discussed in Section 4.3?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)