MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
Authors: Haotian Zhang
Mingfei Gao
Zhe Gan
Philipp Dufter
Nina Wenzel
Forrest Huang
Dhruti Shah
Xianzhi Du
Bowen Zhang
Yanghao Li
Sam Dodge
Keen You
Zhen Yang
Aleksei Timofeev
Mingze Xu
Hong-You Chen
Jean-Philippe Fauconnier
Zhengfeng Lai
Haoxuan You
Zirui Wang
Afshin Dehghan
Peter Grasch
Yinfei Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal LLM
Primary area: foundation or frontier models, including LLMs
TL;DR: A family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning.

Abstract: We present MM1.5, a new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning. Building upon the MM1 architecture, MM1.5 adopts a data-centric approach to model training, systematically exploring the impact of diverse data mixtures across the entire model training lifecycle. This includes high-quality OCR data and synthetic captions for continual pre-training, as well as an optimized visual instruction-tuning data mixture for supervised fine-tuning. Our models range from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE) variants, and demonstrate that careful data curation and training strategies can yield strong performance even at small scales (1B and 3B). Additionally, we introduce two specialized variants: MM1.5-Video, designed for video understanding, and MM1.5-UI, tailored for mobile UI understanding. Through extensive empirical studies and ablations, we provide detailed insights into the training processes and decisions that inform our final designs, offering valuable guidance for future research in MLLM development.

Published as a conference paper at ICLR 2025
MM1.5: METHODS, ANALYSIS & INSIGHTS FROM MULTIMODAL LLM FINE-TUNING
Haotian Zhang Mingfei Gao", Zhe Gan Philipp Nina Wenzel', Forrest Huang", Dhruti Shah". Xianzhi Du", Bowen Zhang". Yanghao Li", Sam Dodge, Keen You, Zhen Yang. Aleksei Timofeev, Mingze Xu, Hong- You Chen, Phanlphilippe Fauconnier, Zhengfeng Lai, Haoxuan You, Zirui Wang, Afshin Dehghan, Peter Grasch", Yinfei Yang!
Apple
(hactian. zhang2, mgao22, zhe. gan, y yinfelyl 8apple.
Fiist authors; *Core authors; 'Project lead
ABSTRACT
We present MM1.5, a new family of multimodal large language models (MLLMs) designed enhance capabilities in text-rich image understanding. visual referring and grounding, and multi-iman image reasoning. Building upon the MMI architecture, MM1.5 adopts data-centric approach model training, systematically exploring the impact of diverse data mixtures across the entire model training lifecycle. This includes high-quality OCR data and synthetic captions for continual pre-training, well as an optimized visual instruction-tuning data mixture for supervised fine- tuning. Our models range from IB to 30B parameters, encompassing both dense and mixture-of-eeperts (MoE) variants, and demonstrate that careful data curation and training strategies can yield strong performance even at small scales (IB and 3B) Through extensive empirical studies and ablations, provide detailed insights into the training processes and decisions that inform our final designs, offering valuable guidance for future research in MILM development.
INTRODUCTION
Multimodal Large Language Models (MLLMs) have emerged as an increasingly active research topic in recent years. Closed-source models, such as GPT-4o (Islam & Moushi] 2024), GPT-4V (OpenAl 2024), Gemini- 1.5 (Team et al. 2023 Reid et al. 2024), and Claude-3.5 (Anthropic 2023), have demonstrated remarkable capabilities in advanced multimodal understanding. Mcanwhile, open- source models, such as the LLaVA series of work (Liu et al.) 22023b a 2024at Li et al. 2024c), PaternVL2 (Chen et al. 2024b), Cambrian-1 (Tong et al.) (024a) and Qw (Bai al. 2023
team 2024). are rapidly narrowing the performance gap. There has also been growing interest in developing models capable of understanding single-image, multi-image, and video data using single
set of model weights (Li et al. 2024c). Further discussion of recent works is in Appendix A.
Building upon MM1 (McKinzie al. 2024). introduce MM1.5, new family of MLLMs carefully designed to enhance a set of core capabilities. Specifically, we focus on the following aspects.
OCR. MM1.5 supports arbitrary image aspect ratios and resolutions of up to 4 Megapixels. By incorporating carefully selected OCR data to enhance text comprehension across different training stages, MM1.5 excels at understanding text-rich images.
Visual referring and grounding. MM1.5 offers robust, fine-grained image understanding, extend- ing beyond text prompts to interpret visual prompts such points and bounding boxes. Moreover, MM1.5 can generate grounded responses by grounding text output with image bounding boxes. This capability is notably under-explored in most open-source models (e.g., LLaVA-One Vision (Li et al. 2024c) and Phi-3- Vision (Abdin et al., (024b)). and even in strong proprietary models like GPT-4o, which rely set-of-mark (SoM) prompting (Yang al. 2023 to reference image regions.
Additionally, we introduce two specialized variants: MM1.5-Video designed for video understanding, and MMT,5-UL, tailored for mobile UT understanding, detailed in Appendix A.7 and A.8 A.8 correspondingly.

---

Published as a conference paper at ICLR 2025
Stage-1: Large- scale Pre-training
Stage-2: High-res Cont. Pre-training
Stage-3: High-res SFT
The impact image- text and text-only pre-training data
The impact OCR data and synthetic captions
The exact impact of each data category
The optimal data mixing ratio
Figure 1: Recipe for building MM1.5, Model training contains three stages: (i) large-scale pre- training with low-resolution images (378x378), (ii) continual pre-training with high-resolution (up to 4 Megapixels) OCR data and synthetic captions, and (fit) supervised fine-tuning (SFT). At each stage, we aim to identify the optimal data mix and assess the impact of each data type.
Multi-image reasoning. MM1.5 1. benefits from large-scale interleaved pre-training, resulting in strong in-context learning and multi-image reasoning capabilities right out of the box. We further improve its capabilities via supervised fine-tuning (SFT) on additional multi-image data.
Our primary focus is on the most efficient model scales, IB and 3B, together with their corresponding Mixture-of-Experts (MoB) variants, and demonstrates that even relatively small MLLMs can achieve competitive performance on various downstream tasks. Furthermore, we demonstrate that the MM1.5 recipe exhibits strong scaling behavior all the way to 30B parameters, achieving competitive performance across a wide range of benchmarks.
Building performant MILMs a highly empirical endeavor. While the overarching goal and the high-level training procedure are well-defined, the finer details of their execution remain unclear. In developing MM1.5, we choose to retain the same model architecture as MMI (McKinzie et al., 2024), enabling us to focus on refining and investigating the intricacies of our data-centric training recipes. Our attention is centered on the following key aspects:
Continual Pre-training. We introduce an additional high-resolution continual pre-training stage preceding the SFT stage, which we found crucial for boosting text-rich image understanding performance. We ablate the impact of two kinds of data: (i) text-rich OCR data, focusing detailed transcription of text within images; and (ii) high-quality synthetic image captions.
SFT. While considerable prior work discusses SFT data for MLLMs, there still limited exploration into how each category of SFT data can affect the final model's performance. In particular, the impact of data supporting each capability on other capabilities is understudied. We conduct. extensive ablations to identify trade-offs and synergies, ultimately constructing mixture from public datasets that contributes to well-balanced performance across wide set of capabilities.
Dynamic High-resolution. Furthermore, for high-resolution image encoding, popular any-resolution approach, dynamically dividing the image into sub-images (Li et al. 2023i
follow the
Zhang al. 2024b), and conduct thorough ablations refine key details in our design.
Unlike most open-source models focusing solely on SFT (Liu et al. 2023b a 2024a), MM1 demon- strated strong zero-shot and few-shot learning capabilities through large-scale pre-training. In developing MM1.5, we aim to retain these strengths and more effectively transfer them to the SFT stage. To achieve this, We further extend MMI's 's pre-training by exploring the impact of text-only data and optimizing the ratio of different pre-training data types. This approach improves performance on knowledge-inten intensive benchmarks and enhances overall multimodal understanding capabilities.
Our main contributions are summarized as follows: (i) We introduce MM1.5, family of MILMs that include both dense models (ranging from to 30B) and MoE variants. (ii) We conduct a thorough empirical study detailing the process and decisions leading to our final design choices. (Mi) MM1.5 excels handling wide range of multimodal tasks, from general-domain to text-rich image understanding, coarse- to fine- ime-grained understanding, and single- to multi-image reasoning.
2 RECIPE FOR BUILDING MM1.5
In this work, beyond pre-training and SFT stages as in MM1 (MeKinzie et al. 2024), introduce a continual pre-training stage with high-quality OCR data and synthetic captions. As outlined in Figure to obtain the best data recipe,
2

---

Published as a conference paper at ICLR 2025
We first present comprehensive ablations of our SFT data mixture (Section 2.2 SFT data into multiple groups based on the capabilities they aim to support. We carefully evaluate the impact of datasets from each category and adjust the ratio to balance different core capabilities.
We categorize the
To further enhance model performance, especially for text-rich image understanding, we further ablate the data choices for continual pre-training (Section 2.3). This includes 45M rich OCR data and 7M high-quality captions from public or generated by
image captioner.
We also provide detailed ablation regarding dynamic image splitting, also known as AnyRes ct al., 2024a) (Section 2.4p. for high-resolution image comprehension.
(Liu
Finally, to enhance performance on knowledge-heavy benchmarks like MMMU Yue et al. 2023a). we further study the impact of pre-training data (Appendix A.2
2.1 EMPIRICAL SETUP FOR ABLATIONS
Unless otherwise noted, we follow the default settings below in our ablation studies.
Model architecture and data preprocessing. We follow MM1 (McKinzie al. (022) and use the same architecture, focusing on the 3B dense model for all the ablation studies in this section. Specifically.
Static image splitting (Lin et al. 2023b) is enabled with 4 sub-image splits (plus an overview image), and each sub-image resized 672 resolution via position embedding interpolation. Note that we did not use dynamic image splitting during ablation for faster iteration of experiments.
As to the encoding of multi -ullti-image data, we enable image splitting only when the current training sample contains fewer than three images avoid excessively long sequence lengths.
Similar to capabilities introduced in Ferret You et al. 2023). MM1.5 directly supports referring and grounding. When requested, MM1.5 can produce bounding boxes in its textual output to ground its responses. Additionally, the model can interpret references to points and regions in the input image the form of referring coordinates and bounding boxes.
The CLIP image encoder and the LLM backbone are based on in-house models, with C- Abstractor (Cha et al. 2024) serving as the vision-language connector.
Model optimization. For both continual pre-training and SFT, we the batch size 256. We use the AdaFactor optimizer with a peak learning rate of le-5 and a cosine decay of O. For continual pre-training, we train a maximum of 30k steps. During SFT, all models are optimized for one epoch.
Continual pre-training Models are initialized with the MM1 pre-trained checkpoint. By default, we conduct continual pre-training on 45M high- resolution OCR data (including PDFA, IDL., Rendered- text (Laurençon et al.] (022a)) and DocStruct-4M (Hu et al. (0224a) at this stage. In each training batch, data is equally sampled from those four datasets. Similar the SFT stage, we use static image splitting, dividing each image into five sub-images, with each sub-image resized to 672x672 resolution. We find that this high-resolution setup is essential for continual pre-training.
SFT data categorization. Grouping datasets into categories can be helpful for data balancing and simplifying the analysis (Laurençon et al.) 2024a; Tong et al. 2024a). At a high level, we cluster datasets into single-image, mulli-image, and text-only categories based on the number of images presented in each example. For the single-imag group, we further classify each dataset into the following sub-categories: general, text-rich, refer&ground, science, math and code. See Table in Appendix A.3 for the details of each category used for the ablation study, and Figure for an overview of the group categories.
Evaluation benchmarks. We group our benchmarks into categories based on what capabilities a benchmark primarily measures. Our benchmark groups include general, text-rich, refer&ground, knowledge, and multi-image. See Table Appendix A.44 for more details. We propose Category Average Score, the average score of all benchmark numbers for each sub-category, to represent the average performance on that capability. We focus on the categories of general. text-rich, and knowledge, as these capabilities are widely considered essential for MLLMs. To evaluate model's impact on these capabilities, we refer to MMBase score, defined as the average scores on general, text-rich, and knowledge categories. Details of the evaluation metrics are provided in Appendix A.4
3

---

Published as a conference paper at ICLR 2025
80
General
61.58
63.99
68. 26
64.17
03 82
02.08
61. 28
Gentral+Text-rich
Average Score
60
40 86
42.1
42.4
40, G9
45.01
46. .9
46 37.37
49.03
42.2
General+Matth
General+Science
General+Code
40
M General+ Winneral-Reffr&kgrruud
18. 18.01
17.09
18:15
17.82
17.8
20
Text Rich
Knowledge
General
Refer&Ground
Figure Impact of different SFT data categories to different model capabilities (general, text-
rich, knowledge, and refer&ground)) Text-rich data significantly improves text-rich and knowledge benchmarks on average. Science data improves knowledge average score. Referring and grounding data enables this capability.
59
68
80
58
67
60
57
56
40
MMBuse Score
Gevur& Average Score
56
50
220
5 10
(e) Science Mixture.
(b) Math Mixtare.
(3) Code Mixture.
(d) R&G Misture.
Figure 3: Impact of a (x-axis) for different data categories to model's different capabilities. The selected ratio marked with red "x". a denotes the data ratio of target category (science, math, code, refer&ground (R&G)) when compared with the general category.
2.2
SFT ABLATIONS
To determine the optimal SFT recipe, we first study the impact of different data categories in Section 2.2.1 followed by investigating how to best mix all the data in Section 2.2.2
2.2.1
IMPACT OF DIFFERENT DATA CATEGORIES
In this subsection, focus evaluating the single-image data category. We begin by assessing the general data category and then progressively evaluate the impact adding other sub-categories individually. During training, mix data from different cub-categories and construct each training batch by randomly sampling data from the corresponding mixture. We compare models using each capability with the Category Average Score.
Our results are summarized Figure We observe that adding text-rich data can significantly improve the performance text-rich and knowledge benchmarks. The inclusion of math data follows a similar trend, though we observe a lesser degree of improvement in the text-rich average score. When science data added, we observe the expected improvement in the knowledge benchmarks, alongside minor improvement in text-rich performance. Adding the code category yields slight increase in the text-rich average score, while the performance on other benchmarks does not improve. Including the refer&ground data instills the model with referring and grounding capability, but we also observe slight regression all other capability categories.
2.2.2 DATA MIXTURE RATIO STUDY
We first study the mixing ratio within the single-image categories. Since directly mixing the general and text-rich data based on their data sizes shows strong results across a variety of benchmarks (see Figure 2 we this combination the starting point to study how to mix other categories this set. Then, we combine the entire single-image set with multi-image and text-only sets with sampling weights of Wsinglee Wmulti and When respectively, where Wsingle + Wmutt Wrext 1.
Mixture of single-image data. Directly mixing all datasets from different categories may not be ideal due imbalanced numbers of data samples across different sub-categories. For example, the
4

---

increases Published as a conference paper at ICLR 2025
68
56
54
52
MMBasseScrre
Multi-image Average Score
5- 10-2
0.15
0.2
0.1
0.2
0.3
0.4
0.5
Test-oaly
(b) Multi-image data muxture ratio
Figure 4: Impact of the mixing ratio for text-only and multi-image SFT data. The selected ratio is marked with red "x".
74.38
Average Score
60
80
62
23
60 00.97
59.12
53 15
52 $2.54
61.
05.41
61.00
64.45
43.59
55.
78
59
19
61.
ZZ
Base Mixture
Single Image Mixture
45
All Mixture
40
20
Text-rich
Knowledge
General
Refer& Ground
Multi- image
Average
Figure 5: Ablation study of mixing all the SFT data. Base Mixture denotes general, text-rich and knowledge (science, math and code). The "Average" column represents the performance averaged across the preceding five benchmark categories.
size the general data category is around 68x the size the science data category. In this study, we use the general data category as the reference, and upsample/downsampll data from a target category, such that in each training batch, the data ratio from the general and target category 1:a.
To measure the average impact of a, we propose MMBase score, average over general, text-rich, and knowledge average scores, for model comparison. As shown Figure 3 we vary the for different data categories. For science, math, and code categories, we find the best ratio of a to be 0.1, 0.5, and 0.2, respectively. As shown in Section 1.2.1 the refer&ground data is the main driver for improving referring and grounding benchmarks. Therefore, besides the MMBase score, also include the Refer& Ground average score as another metric for the a selection. As summarized in Refer&Ground
will drop slightly, while the Refer& Ground average significantly. With that, we select a 2.0 as good trade-off.
Figure 3d), the MMBe
Mixture of single-image, multi-image, and text-only data. Now, study the mixture ratios, Wsingle, Wmuli and Whext- Enumerating all combinations between the three ratios will incur significant computational cost. Therefore, instead separately ablate Wjet and Wmulti for text-only and multi- image data, respectively, to evaluate how sensitive our model is to these ratios. Finally, Wsingle is determined by Whet Wmulti
Similar to the single-image mixture study, we also start with the combination of general and text-rich data and enumerate different values for Wmulti and Wext For text-only data, we tested Whet from to 0.2. Figure Ileft) shows that varying different values for Whst has minor effects on the model's 's base capabilities in general. select Whest = 0.1 to allocate a higher weight for single-image data for potential performance improvements.
For multi-image data, we use the multi-image average score (evaluated on multi-image benchmarks in Table 6 as additional metric to assess model's capability of handling multi-image tasks. Results are summarized Figure rright.. right). We observe that increasing the sampling ratio of multi-image data would introduce performance drop of the base capabilities indicated by the decreased number of the MMBase score, while the multi-image average score increases. We select Wmulti = 0.1 since introduces surge in the multi multi-image average score.
Mixing multiple categories. Based on the studies above, we present three mixtures, the Base mixture, Single-image mixture, and All mixture, and analyze their trade-offs. The Base mixture includes the
5

---

Published as a conference paper at ICLR 2025
61
61
60. 21.266
Average Score
60
Averuge Score
50.77
60
69
00.41
60
50 68
59. 51
88.8
50
59
38 28
59
58
58
No Cont. PT
756
1344 x 1344
No Cont. PT
OCR LLaVARecap
ShareGPT4V
OCR+LLaVAReunp
ShareGPTP
378
(a) Impact of input resolution. OCR data used for all the continual pre-training experiments.
(b) Impact of data source. Continual pre-training is conducted in the high-resolution (1344x 1344) setting.
Figure 6: Ablation study of continual pre-training. Average Score indicates the MMBase score. Cont.
PT denotes continual pre-training.
general, text-rich, science (Oiszience 0.1), math (Ommah 0.5) and code (Orade 0.2) data groups. The Single-image mixture additionally adds refer&ground data (Arn 2.0) to the Base mixture. All mixture includes all data from single-image, multi-image, and text-only data, with Wsingle 0.8,
Omali = 0.1, and What = 0.1.
Our results are summarized in Figure The first three columns indicate that including refun& and multi-image data slightly reduces average performance text-rich, knowledge, and general benchmarks. The fourth column shows that adding refer&ground data significantly boosts referring and grounding performance, while the fifth column highlights that adding multi-image data greatly improves multi-image henchmarks. The final column reveals that our optimized mixture achieves the best overall performance, balancing all capabilities across benchmarks.
2.3 CONTINUAL PRE- TRAINING ABLATIONS
Unless otherwise specified, we use OCR data (45M in total), including PDFA, IDL, Rendered- text (Laurençon et al., 2024a) and DoeStruct-4M (Hu al., 2024a) high nigh-resolutiin setting (1344x 1344) for continual pre-training. During the SFT stage, all continual pre-trained models in this section are fine-tuned with data from the Base Mixture including general, text-rich, knowledge (science, math, and code) with their selected mixture ratios described in Section 2.2.2
Impact of image resolution. Intuitively, regher-resolution images are preferable when training with OCR data. We first ablate the impact of image resolution during this stage by setting up two baselines, continual pre- training with 378x378 and 756x756 resolutions, respectively. For the former, we disabled both image splitting and position embedding interpolation (our CLIP image encoder natively supports image resolution of 378 x x37)). For the latter, we enabled image splitting and turn-off' position embedding interpolation. The results are shown in Figure a).. Note that the final SFT stage always uses image resolution x344 1344 across these experiments, so the training only differs with respect the image resolution used in continual pre-training.
We can clearly see that using setting of 1344x 1344 image resolution for continual pre-training achieves the best overall performance. Decreasing resolution consistently leads lower final scores. In particular, continual pre-training with 378 x x37 resolution can underperform model without continual pre-training. We hypothesize this due insufficient visible detail lower resolutions, which may hinder the model's ability to effectively learn from the document- based OCR data in the continual pre-training mixture.
Impact of OCR data and synthetic captions. Besides OCR data, high-quality synthetic image captions (Chen et al., 2023c| Li et al., 2024a) are also widely considered useful for pre-training. To study its impact, we use our default setting except for the data used in continual pre-training. We study two synthetic caption datasets:
(Li et al. 2024a) and ShareGPT4V-PT (Chen et al. 2023c), and their combination with OCR data. When or LLa VLLaA--Recay 3M with our OCR data, we equally sample data from individual datasets in each training batch. Results are presented in Figure 6 b). We observe that all continual pre-trained models
combine ShareGPT4V-PI

---

Published as a conference paper at ICLR 2025
Table 1: Ablation on the image resolution and the number of image tokens used in dynamic image splitting. n denotes the total number of sub-images. Row 3: (Naiiny Amaxx = (4, 4); Row 4-7:
(Amins Fmmax) (9. 9). Image encoder resolution: (i) 378 378: no position embedding interpolation;
(+i) 672 x 672: with position embedding interpolation.
Row
Mode
12
(per sub img / lotal)
#amage tokens
Image Enc. Resolution
Effective Resolation
Text-rich
Knowledge
General
Refer & Ground
Average
144/144
672x672
0.45MP
49.4
$3.6
62.6
71.3
Static
5
144/720
672x672
1.MPP
$7.7
53.8
64.4
74.8
62.7
5
144/720
672x672
18MP
58.6
53.7
64.1
74.0
62.5
10
81/810
378x378
L.MP
57.6
53.3
62.9
74.0
62.0
Dynamic
10
81/810
672x672
4.IMP
58.3
53.8
64.3
74.9
62.8
10
144/1440
378x3788
1.3MP
58.5
54.0
63.2
74.5
10
144/1440
672x672
4.IMP
59.8
54.0
64.5
75.2
63,3
perform better than the baseline without continual pre-training. However, we did not find conclusive evidence that these high-quality synthetic captions improved performance over the arguably simpler OCR data. While prior studies (Li et al. 2024c) show synthetic captions boost performance, our results indicate further investigation into their exact impact is needed.
Therefore, we further investigate the impact of synthetic captions generated through self-training for even larger scales (up to 7M) and more controllable styles, using a pre-trained MMI model fine-tuned on human-annotated captions, similar to (Fang et al. (2024). This new dataset showed some promise in certain settings, see Appendix A.3 for details. We defer further study of this topic to future work.
2.4 DYNAMIC IMAGE SPLITTING ABLATIONS
Dynamic image splitting. Processing high-resolute images is essential for text-rich image under- standing. In static image splitting (Lin al. 2023b), images are split into a rigid 2x2 grid, which is often inefficient. Low-resolution images are splitted without any need, and images with non-square aspect ratios can result in sub-images being padding only. Therefore, we adopt dynamic image splitting approach, which is common in the literature (Li et al., 2024a Dong et al. 2024b Hu et al. 2024a| Lin al., 2023b Xu et al. 2024c Zhang et al. 2024d). for MM1.5.
Given a minimum and maximum number of sub-images, Amin and 7max consider the set of all candidate grids G {(1,, n..)) E N Amin S nng nu S [2maa J. Further, consider the resolution of the vision encoder and an input image resolution (h, w.). If there is grid that can cover the image, we choose the grid that minimizes the amount of padding after longer side resizing to the grid, iee. g" arg min (n.. )=EG 71, n= 12_ - Wg. subject to n,. 2 hg h and new 2 Wg 2 w, where hg W/ denote the image height and width after longer side resizing the candidate grid. If no such grid exists, we choose the one that minimizes the resolution loss due scaling the image down and fully covers the longer side resized image. Assume we allow up to 4 sub-images. With a static image splitting approach, all images use the grid (2, 2). The dynamic splitting approach instead allows for the following grids: (), (2, 1), 3), (3, 1), (1, 4), (4, (2, 2)).
always feed the original image with longer side resized to the encoder resolution r to the model. This ensures that the model has a global understanding of the image. If the grid is (1, 1). we omit the overview image. We consider two variants: (i) before: the overview image is put before the sub-images; (it) after: the overview image put after the sub-images. These variants yield different results because an autoregressive mask is used in the LLM decoder, and as such, the choice determines whether the decoder can attend to the overview image when processing the sub-images (i) or attend to the sub-images when processing the overview image (ii).
Global-Local Format. In addition the sub-images,
Sub-image position indicator. Given that an input image dynamically split into multiple sub- images, we explore whether it is helpful to indicate the position of each sub-image in the original high- resolution image to ensure the model can understand the original 2D image structure. Specifically, consider two methods.
Index. A tuple of (k, (,), is used to represent sub-image position information, where is the zero-indexed image number in the example (assuming there can be multiple images in training sample), and j are the one-index row and column id, e.g., (0, 0, 0) is the overview image of image 0, and (0, 2, 1) is the sub-image in the second row and first column, for image 0.
7

---

Published as a conference paper at ICLR 2025
Table 2: Ablation on the image grid configuration (7einn Rmmax) used in dynamic image splitting.
Row
(Fenin Plaus) Inference
DocVQA
InfoVQA
Text-rich
Knowledge
General
Refer Ground
Average
Train
Model Comparison
[4,4)
(4.4)
73.2
53.3
2
(4,9)
(4.9)
75.7
53.8
60.0
54.0
63.9
74.6
63.1
3
16)
(4)16)
76.3
55.2
60.7
53.4
64.0
73.8
63.0
(1.9)
(1.9)
76.2
54.1
60.4
53.7
62.5
71.5
62.0
(4.4)
(4.9)
73.4
52.9
59.7
53.5
74.0
(4.4)
(1.16)
72.3
53.5
59.6
53.8
63.5
74.0
62.7
(4.4)
(1,9)
73.5
52.7
59.8
50.7
62.6
24.5
49.4
7B Model Comparison
8
(4.4)
(1,4)
77.0
54.3
64.5
61.1
66.8
77.7
67.5
9
(4.9)
(4,9)
81.7
62.1
67.4
60.1
66.6
78.0
68.0
10
(4, 16)
(4,
83.3
64.1
58.7
77.2
Table 3: Ablation on the sub-image position indicator and the position of the overview image. We (Amin mmmxx) (4, 4) for experiments.
Row #
Sub-img indicator
Overview image pos.
Refer Ground
DocVQA
InfoVQA
Text-rich
Knowledge
General
Average
1
before
73.2
48.3
58.6
53.5
64.1
74.0
62.5
2
seps
hefore
74.3
49.7
58.8
53.0
63.8
74.5
62.5
3
index
hefore
73.4
48.6
58.6
52.7
63.4
74.8
62.4
4 none
after
73.3
49.7
59.2
54.3
64.1
73.8
62.8
Seps. Instead of using indexes, we use three text tokens. Specifically, ":" the overview image indicator, '," is the column separator, and '<n>' is the row separator. The latter two tokens are inserted between the of image tokens corresponding each sub-image so that the original 2D image structure can be recovered from the flattened image token sequence.
Inference for higher resolution. The tuple (Tmin: Tlmax) is used to decide the dynamic image splitting configuration for model training. During inference, it is possible to support even higher- resolution image processing simply by increasing these parameters. For example, we explore training at (Nmin: Rmmax) = (4, 9) to save model training compute, while during inference, we use (n'min ) (4, 16) to process images at even higher effective resolutions.
default experiment setting, including general, text-rich, knowledge (science, math and code), and refer&ground data. For fast iteration of experiments, all the models are initialized with the MM1 pre- pre-trained checkpoint without continual pre-training. Following Figure we report the average performance on text-rich, knowledge, general, and geoun& benchmarks. Our findings summarized follows.
Findings. We use the final Single-image Mixture
Dynamic image splitting outperforms static splitting in text-rich tasks. Both image resolution and the number of sub-images, along with image token counts, important (Table I.
Regarding image grid configuration (Table dynamic image splitting with larger T'max signifi- cantly improves performance on tasks with unusual aspect ratios, such as Doc and Info VQA, especially when trained natively for it. Grounding performance is highly sensitive to grid size changes, and larger LLM backbones yield greater performance gains, with the 7B model showing larger improvements than the model.
terms of position indicators (Table 3). they are not strictly necessary for text-rich tasks, though they can be beneficial for specific tasks like DocVQA and VQA. Additionally, placing the overview image after the sub-images improves performance, as it allows the decoder attention mask to attend to all sub-images more effectively.
practice, dynamic image splitting does not significantly increase the number of sub-images to process. In sample of 100k examples, static splitting generates 500k splitting with (Tmin Tlmax) (4,9) produces barely more, only 539k images in total.
while dynamic
8

---

Published as a conference paper at ICLR 2025
3 FINAL MODEL AND TRAINING RECIPE
We collect the results from the previous ablations to determine the final recipe for MML.5 training:
Architecture We use the same model architecture as MMI (McKinzie et al. 2024).
Data and training pipeline. As summarized in Figure ! MML.5 is trained in three stages:
Pre-training. The pre-training data comprises three parts: (i) 2B image-text pairs, (ii) 600M interleaved image-text documents with 1B images in total, and (iii) text-only data with 2T tokens. Except for the updated text-only data, the data remains unchanged from MM1 (McKinzie et al. 2024). However, the data ratio has been adjusted from 45:45:10 to 50: 10:40, significantly downweighting the interleaved data (from 45% to 10%) while increasing the proportion of text-only data (from 10% to 40%) as discussed in Section A.2
Continual Pre-training. We use 45M OCR data enhance text-rich image understanding. Notably, we do not include additional synthetic image captions based on empirical results.
-FFT. We use the data illustrated in Figure 8 and adopt the mixing ratios studied in Section 2.2.2 Our final mixture consists of 80% singleimaae data, 10% multi-image data, and 10% text-only SFT data. The single- -mage data can be further categorized into 37.2% text-rich data, 22.5% % refer& data (VQA data enriched with bounding boxes and/or point coordinates), 1 11.3% general data, 5.6% math data, 2.3% code data, and 1.1% science data, totaling 80% of all data.
Dynamic high-resolution. We set the image grid configuration (Mmin, mmax) (4, 9), using an index for the sub-image position indicator and placing the overview image after the sub-images. Dynamic image splitting is only enabled when the current training sample has fewer than three images. The supported resolution reaches up to 4 Megapixels (approximately 2016x201 for a square image, or 6048 x 672 for a long image).
We use the same image encoder and the LLM backbone from MM1 (McKinzie et al. 2024), and keep them unfrozen during all the model training stages. For pre-training, we follow the exact same learning rate schedule as in MM1 (McKinzic et al. 2024) and 200k training steps with sequence length 4096. For continual pre-training, we use a peak learning rate of 1e-5 with the cosine decay and 30k training steps for all the models (from 1B to 30B). For SFT, we use a peak learning rate of 2e-5 and 23k training steps for all the models F
Mixture-of -fpexss (MoE). We introduce two MoE models, a IB-MoE and 3B-MoE, with 64 experts replacing dense layers every two layers. We use the same hyperparameters as those applied to the dense models for both scales and top-2 gating with 0.01 load balance loss to encourage better expert load balance and a 0.001 router z-loss for training stability (Lepikhin et al. 2021).
3.1 RESULTS
We evaluate our MM1.5 models across multimodal benchmarks using an internal fork of Im-eval- harness (Gao et al.) 2023), covering task categories ranging from general multimodal understanding. knowledge, text-rich, referring and grounding, multi-image reasoning, to in-context learning. Detailed
results for each capability at various model sizes are further summarized in Table 7 g 9 10 and respectively. Below, we highlight a few key observations from Table
MM1.5 represents major upgrade over MM1. It delivers improvements across all model sizes and nearly all benchmarks, often by a substantial margin. For instance, MM1.5-30B boosts the Math Vista from 39.4 to 55,6, Doc VQA from 75.8 to 91.4, and InfoVQA from 47.3 to 67.3. It also offers much enhanced multi-image reasoning capability and introduces new capabilities not present in MM1, such as visual referring & grounding.
Both Dense and MoE model scaling are effective. First, scaling the dense model from 1B to 30B consistently improves performance, with benchmarks like AI2D increasing from 59.3 77.2 (see 77) Second, both the IB and 3B MoE models outperform their dense counterparts. Notably, the MM1.5-3B-Mo model can even surpass the MM1.5-7B model in knowledge, general, visual referring and grounding, and multi-image benchmarks, though it falls slightly behind on text-rich benchmarks. This suggests that MoE models show strong potential integrating diverse capabilities compared to dense models.
AAll models are trained using the AXI.cam framework https://gg hww.h/ubb.omm/ppapp/ aarn

---

Published as a conference paper at ICLR 2025
Table Comparison with SOTA models on the selected benchmarks. Comparison of more bench- marks and broader baselines can be found in Table 7
8
10 and 11 n the Appendix.
Tich
Knowledge
General
Refer&Cround
Muln image
In contest
Mivill
QA (val)
DoeVQA ((es))
InioVQA (1es))
MMMU (val))
MathY (estmini)
SEED
RealWoridQA
RecCoco aeg
LVIS avg
NLVR2 (val)
VLA avg
18 Model Compurtion
L L.VAOnaVisio--..5
70.0
41.8
31,4
34.8
65.5
55.6
63.4
SPHINX Tlay
57.8
53.0
26.3
26.4
77.2
MML-18
68.4
18.5
33.2
65.6
50.9
34.3
MMI.S-IB
50.5
51.0
35.9
71.4
83.2
Compariion
VL.222
73.4
86.9
58.9
36.3
70.9
77.7
51.1
67.4
18.5
MiniCPM 2.0-8
74.1
71.9
37.5
38.2
38.7
67.1
55.8
48.00
Phi-2-Vision-88
70.1
83.3
49.0
400.4
44.5
71.8
59.4
38.1
54.2
53.6
19.5
MMI-3B
71.9
75.2
64.7
33.9
32.0
68.8
55.8
53.4
$1.7
37.0
MML.533
76.5
97.7
58.5
37.1
44.4
72.4
56.9
35.6
67.9
83.8
56.3
MMI 5-1--MoR
76.8
85.0
53.6
42.9
46.9
25.3
60.7
86.2
66.3 9
86.0
59.6
78 Model Compurinon
MML-78
45.5
69.9
MMI 5-78
Model Compurizon
47.5
72.1
MMI.5-30R
79.2
91.4
67.3
47.4
75.0
89.0
90.1
73.2
90.6
77.6
MM1.5-1B is the state-of-the-art model at the 1B scale. While few models are available at this scale, MM1.5-1B clearly outperforms comparable models such as SPHINX-Time Tiny (Gao et al. (024). For reference, MM1.5-IB also significantly surpasses LLaVAOne B (Li et al.) (024c) (e.g.. ScienceQA: 67.2 vs. 82.1, VQA: 70.0 vs. 81.0, see Table [). but it should be stressed that this is of course an smaller model and as such cannot be directly compared.
MM1.5-3B outperforms MiniCPM-V 2.0 and is competitive with Intern and Vision. Compared to MiniCPM-V 2.0, MM1.5-3B achieves superior results Math Vista (38.7 vs. 44.4) and DocVQA (71.9 vs. 87.7) while also supporting visual referring and grounding, which MiniCPM-V general VQA and, although Phi-3-Vision (4.2B) has edge on knowledge-basee tasks like A12D as in Table 7 MM1.5-3B excels on text-rich benchmarks (e.g.. Voc 83.3 vs. 87.7) and outperforms Phi-3- in referring and grounding (see Table 9 as well as in-context learning tasks (see Table TO).
lacks. It surpasses Intern VL.2-BB
MM1.5 excels in visual referring and grounding. While most SOTA models foeus improving performance across general, knowledge, and text-rich benchmarks, few have integrated fine- fine-grained image grounding and referring ability into their design. Even GPT-4o relies on set-of-mark prompting to demonstrate visual grounding capabilities. As shown in Table MM1.5-31 1.5-BB outperforms Ferret-7B and is on par with Ferret-13B. both of which are fine-ture tine-tuned specifically for referring and grounding tasks. Notably, our model inherently possesses these capabilities while still excelling in other areas.
MM1.5 excels in multi-image reasoning and in-context learning. As shown Table [T MM1.5-1B model outperforms LisvAOneeVsiion-O0 Vision-0.5B at the 1B scale. Similarly, at the 3B scale, MM1.5-3B significantly surpasses Phi-3- Vision. Additionally, we evaluate MM1.5's zero-shot transfer capability for video understanding using MVBench (Li et al. 2024h), benchmark designed for video tasks. Moreover, we evaluate MM1.5's ability multimodal in-context learning on VL-ICL benchmark (Zong et al. 2024). As shown in Table TO our models outperform others in in-context learning. In Section A.7 we will further introduce MM1.5 Video, a model variant specifically designed for video understanding.
4 CONCLUSION AND LIMITATION
In this work, we build on the insights of MM1 McKinzie et al., 2024 and introduce MM1 55, family of highly performant generalist MLLMs. While MM1 focused on key pre-training choices, we improve post-pre-training techniques like continual pre-training, dynamic high-resolution image processing, and curation of our supervised fine-tuning datasets. Our extensive ablations show MM1.5 's strong performance in text-rich image understanding, visual grounding, and multi-image reasoning. Like most MLLMs, MM1.5 may produce harmful and counterfactual responses. Future efforts will aim to unify these capabilities for a stronger generalist model, with the hope of benefiting the community beyond specific architectures.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
wendlerc/renderedeex..
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Ben- haim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Qin Cai, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Yen-Chun Chen, Yi- Ling Chen, Parul Chopra, Xiyang Dai, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Victor Fragoso, Dan Iter, Mei Gao, Min Gao, Jianfeng Gao, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider. Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang. Lars Liden, Ce Liu, Mengehen Liu, Weishung Liu, Eric Lin, Zeqi Lin, Chong Luo, Piyush Madan, Matt Mazzola, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sam- budha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang. Hiteshi Sharma, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Xin Wang. Lijuan Wang. Chunyu Wang. Yu Wang. Rachel Ward, Guanhua Wang, Philipp Witte, Haiping Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Sonali Yadav, Fan Yang, Jianwei Yang. Ziyi Yang, Yifan Yang. Donghan Yu, Lu Yuan, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou, Phi-3 technical report: A highly capable language model locally on your phone, 2024a.
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.142119 2024b.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. NeurIPS, 2022.
Anthropic. Claude 3 model card, 2023. bbeeba9b01ccabbcbaee Accessed: 2024-08-13.
URL https: www cán anthrcpiccccoo com/ Card Claude 3. pdf
3b80b
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint 13X1::2308.013990 2023.
Anas Awadalla, Le Xue, Oscar Lo, Manli Shu, Hannah Lee, Etash Kumar Guha, Matt Jordan, Sheng Shen, Mohamed Awadalla, Silvio Savarese, et al. Mint- Scaling open-source multimodal data by 10x: A multimodal dataset with one trillion tokens. arXiv preprint arXiv:2406. 11271, 2024.
Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Carbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. Screenai: A vision-language model for ui and infographics understanding. In IJCAI, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large lision-language model with versatile abilities. arXiv preprint arXiv:2308. 2023.
Jonas Belouadi, Anne Lauscher, and Steffen Eger. Automatikz: Text-guided synthesis of scientific vector graphics with tikz. arXiv preprint arXiv:2310.00366, 2023.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin,, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726 2024.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawa- har, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV, 2019.
11

---

Published as a conference paper at ICLR 2025
Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. arXiv preprint arXiv:2405. 2024.
Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhancee projector for multimodal llm. In CVPR, 2024.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09475 2023a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306. 15/95, 2023b.
Kezhen Chen, Rahul Thapa, Rahul Chalamala, Ben Athiwaratkun, Shuaiwen Leon Song, and James Zou. Dragonfly: rolti-resolution zoom supercharges large visual-language model. arXiv preprint arXiv:2406.0099) 2024a.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023c.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint MiXi::999.00166 2019.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.0033 00325, 2015.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.
2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguitin tasks. In CVPR. 2024c.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Hao Huang, Bryan Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. arXiv preprint (XXx:221990002222 2021.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. VideoLLaMA 2: Advancing natial-temporal modeling and audio understanding in video-Ilms. arXiv preprint 2rXiv:2406.074700 2024.
Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo. Yan Gao, Shi Han, Jian -uang Lou, and Dongmei Zhang. Hitab: A hierarchical table dataset for question answering and natural language generation. arXiv preprint arXiv:2108.067 067 2021.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. mntemlm-xcomposer22. large vision-language model handling resolutions from 336 pixels 4k hd. arXiv preprint 055::22044005122. 2024a.
A pioneering
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. peemll-xcomposer22 large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv preprint arXiv:2404.06512, 2024b.
A pioneering
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang. Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Vlmevalkit: An open-source toolkit for evaluating large multi madiitty models, 2024. URL httts:/:: :/arxiiv////x....///a www.vooorrraeee wtt//bs///aa
11 11691
12

---

Published as a conference paper at ICLR 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv preprint arXhv:240.217783, 2024.
Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang, Pavlo Molchanov, Jang Hyun Cho, Marco Pavone, Song Han, and Hongxu Yin. vila?: Vila augmented vila. arXiv preprint arXiv:2407.17453 2024.
Maxwell Forbes, Christine Kaeser-Chen, Piyush Sharma, and Serge Belongie. Neural naturalist: Generating fine-grained image comparisons. arXiv preprint arXiv:1909, 04101, 2019.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprchensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. /3394, 2024a.
Chaoyou Fu, Yuhan Dai. Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal Ilms video analysis. arXiv preprint arXiv:2405.2 2024b.
Stephanie Fu*, Netanel Tamir Shobhita Sundaram Lucy Chai, Richard Zhang. Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv preprint arXiv:2306.09344, 2023.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng. Haoyu Wang, Xudong Lin, Dan Roth, Noah Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404. 2390, 2024c.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h. h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 12 2023. URL https:t:: /twp:///eeooo.oooooo o orq http://dd..rggeeeeerr///////2
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.05933. 2024.
Chunjiang Ge. Sijie Cheng, Ziming Wang, Jiale Yuan, Yuan Gao, Jun Song, Shiji Song, Gao Huang, and Bo Zheng. Convllava: Hierarchical backbones as visual encoder for large multimodal models. arXiv preprint 24Xiv:2405.1572 2024.
Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang. Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, et al. Apple intelligence foundation language models. arXiv preprint a4X 2075 2024.
Agrim Gupta, Piotr Dollár, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR, 2019.
Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv preprint 1rXiv:2402.11530. 2024.
Musashi Hinck, Matthew L Olson, David Cobbley, Shao-Yes Yen Tseng, and Vasudev Lal. Llava-gemma: Accelerating multimodal foundation models with compact language model. arXiv preprint 2rx1vv2404.011311 2024.
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding, arXiv preprint arXiv:2408. 16500, 2024a.
Wenyi Hong, Weihan Wang. Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang. Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In CVPR, 2024b.
Yu-Chung Hsiao, Fedir Zubach, Maria Wang, and Jindong Chen. Screenqa: Large-scale question- answer pairs over mobile app screenshots. arXiv preprint 2rXiv:2209.08/11 2022.
13

---

Published as a conference paper at ICLR 2025
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXhv:2403./2895 2024a.
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei uplug-docowl arXiv preprint arXiv:2433
Huang, et al. mplug- docowl 1.5: Unified structure learning for ocr-free document understanding. 2024b.
Mingxin Huang, Yuliang Liu, Dingkang Liang, Lianwen Jin, and Xiang Bai. Mini-monkey: Alleviate the sawtooth effect by multi-scale adaptive cropping. arXiv preprint WrXiv:2408.02034 2024.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. NeurlPS, 2023.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. CVPR, 2019.
Raisa Islam and Owana Marzia Moushi. Gpt-4o: The cutting-edge edge advancement in multimodal Ilm. Authorea Preprints, 2024.
Harsh Jhamtani and Taylor Korg-Kirkpatrick. Learning to describe differences between pairs of similar images. arXiv preprint arXiv: 10584, 2018.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi- iaatt instruction tuning. arXiv preprint arXiv:2405.01483, 2024.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In CVPR, 2024a.
Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, et al. Efficient multimodal large language models: A survey. arXiv preprint 20X::2240S.1073, 2024b.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. CVPR, 2017.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualiza- tions via question answering. In CVPR, 2018.
Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning. arXiv preprint arXiv:2312.12241 2023.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In ECCV, 201
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, 2016b.
Geewook Kim, Teakgyu Hong. Moonbin Yim, Jeong Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang. Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In ECCV, 2022.
Wonkyun Kim, Changin Choi, Wonseok Lee, and Wonjong Rhee. An image grid can be worth a video: Zero-shot video question answering using vlm. arXiv preprint arXiv:2403. /8406, 2024.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
14

---

Published as a conference paper at ICLR 2025
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In CVPR, 2024.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better under- standing vision-language models: insights and future directions. arXiv preprint arXiv:2408. 12637, 2024a.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. NeurIPS, 2024b.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint 2.Xiv:2005.0224. 2024a.
Hugo Laurençon, Léo Tronchon, and Victor Sanh. Unlocking the conversion of web screenshots into html code with the websight dataset. arXiv preprint arXiv:2403.09029 2024b.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Hard: Scaling giant models with condi- tional computation and automatic sharding, In ICLR, 2021.
Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023a.
Bo Li, Hao Zhang. Kaichen Zhang, Dong Guo, Yuanhan Zhang, Renrui Zhang, Feng Li, Ziwei Liu, and Chunyuan Li. Leava-next: What else influences visual instruc- tion tuning beyond data?, May 2024a. URL bttps:://laaaeee vl. github.io/blog/
2024-05-255 25- 25vllava next ablations/
Bo Li. Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimodal models, March 2024b. URL https: : //t:://ggttttt com www.o/////vviii.oooooooooooooooe
Imms-eval
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprins arXiv:2408.0m 033366 2024c.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Iims with generative comprehension. arXiv preprint arXiv:2307.16125, 2023b.
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang. Linjie Li, Lijuan Wang. Jianfeng Gao, et al. Multimodal foundation models: From specialists to general-pune purpose assistants. Foundations and Trends@ in Computer Graphics and Vision, 16(1-2):1-214 2024d.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next saaavaanex-inttrleaaee Tackling amati-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07880 2024e.
Feng Li. Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma. and Chunyuan Li. Leava-next: Tackling multi-image, video, and 3d in large multimodal models, June 2024f. URL https://llava-vl..or 2024-06-16- ava- next- Lnneerleave
oo/lllllo og/
Gang Li and Yang Li. Spotlight: Mobile UI understanding using vision-language models with focus. In ICLR, 2023.
Jiachen Li, Xinyao Wang, Sijie Zhu, Chia-Wor Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, and Longyin Wen. Cumo: Scaling multimodal I1m with co-upcycled mixture-on cexture-of-exae of-experts. arXiv preprint 4005 405.5990 2024g.
15

---

Published as a conference paper at ICLR 2025
Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. IntentQA: Context-aware video intent reasoning. In ICCV, 2023c.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: 2: Bootstrapping language- immaee pre-training with frozen image encoders and large language models. In ICML, 2023d.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprinl arXiv:2305.06355 2023e.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video understanding 2023f.
benchmark. arXiv preprini arXiv:2311.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024h.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. arXiv preprint 1xXi:224030002:: 2024i.
Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shenglong Ye, Zhenjiang Jin, Guanzhou Chen, Yinan He, Zhangwei Gao, Erfei Cui, et Omnicorpus: unified multimodal corpus of 10 billion-level images interleaved with text. arXiv preprint arXiv:2406.08418, 2024j.
Wentong Li, Yugian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, and Lei Zhang. Token- packer: Efficient visual projector for multimodal Ilm. arXiv preprint 2xXiv:247.003322 2024k.
Yang Li, Gang Li, Luheng Hc, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile interface elements. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), EMNLP, 2020.
Yanwei Li, Chengyao Wang, and Jiaya Jia. LLaMA-VID: An image is worth 2 tokens in large language models. arXiv preprint 1xivv2211..704. 2023g.
Yanwei Li, Yuechen Zhang, Chengyao Wang. Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 18814, 20241.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023h.
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint 1.0Xi::2311.06600 2023i.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium" Openorca: An open dataset of gpt augmented flan reasoning traces. https::/https:// huggingfacee hwtph/aa.........eeeeeeeeee http://pennO///pppeeeeee Orwa/OpenOrcaaaaa 2023.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:231 10112, 2023a.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-Ilava: Mixture of experts for large laiounlanguage models. arXiv preprint arX1v22401.15447, 2024a.
Lin, Hongxu Yin, Wei Ping. Pavlo Molchanov, Mohammad Shocybi. and Song Han. Vila: On pre-training for visual language models. In CVPR, 2024b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
16

---

Published as a conference paper at ICLR 2025
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.0757.. 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Llava-1.5: Improved baselines with visual instruction tuning. arXiv preprini arXiv:2310.03744 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304. 08485, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024a.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang. Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv preprint X3ix:230500789) 2024b.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepscek-vl: Towards real-world vision-language understanding. arXiv preprint arXhv:2403.05525. 2024.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In ACL, 2021a.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chur Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In NeurlPS Track Dalasets and Benchmarks, 2021b.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurlPS, 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023a.
Pan Lu, Liang Qiu, Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In ICLR, 2023b.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. Wizardeoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08855 2023.
Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reducing hallucination in video language models via equal distance to visual tokens. In CVPR, 2024.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-ChatPP1 Towards detailed video understanding via large vision and language models. In ACL, 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: diagnostic benchmark for very long-form video language understanding. NewrlPS, 2024.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. CVPR, 2019.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint 0rXiv:2203.10244 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar, Docvqa: dataset for vqa on document images. In WACV, 2021.
17

---

Published as a conference paper at ICLR 2025
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographievqa. In WACV, 2022.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal IIm pre-training. arXiv preprint arXiv:2433.0961 2024.
Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zuxuan Wu, Jianfeng Gao, and Yu-Gang Jiang. Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for Imms. arXiv preprint 4rXiv:2406.04334 2024.
C. Mike, H Matt, M Ankit, X Jianwei, W Jun, Sam, G Ali, W Patrick, Z Matei, and X Reynold. Free dolly: Introducing the world's first truly open instruction-tunet 1lm, 2023.
Anand Mishra, Shashank Shekhar, Aject Kumar Singh, and Anirban Chakraborty, Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019.
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402. 14830, 2024.
Varun K Nagaraja, Vlad Morariu, and Larry Davis. Modeling context between objects for referring expression understanding. In ECCV, 2016.
Jason Obeid and Enamul Hoque. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. arXiv preprint 2rX0::2010.09142, 2020.
OpenAL. Gpt-4 vision.
OpenAT, 2024.
https://opena: ww..////eeaaccii
gpt 4 vision
Panupong Pasupat and Perey Liang. Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508 00300, 2015.
Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. ICCV, 2015.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad Khan. Glamm: Pixel grounding large multimodal model. In CVPR, 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In CVPR, 2024.
Eldon Schoop. Xin Zhou, Gang Li, Zhourong Chen, Bjoern Hartmann, and Yang Li. Predicting and explaining mobile ui tappability with vision modeling and saliency analysis. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2022.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In ECCV, 2022.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019.
18

---

Published as a conference paper at ICLR 2025
Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin. Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal Ilms with mixture of encoders. arXiv preprint arXiv:2408. 15998, 2024.
Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: How far we from automating front-end engineering? arXiv preprint arXiv:2403.03163 2024.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioningwith reading comprehension. In ECCV, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019.
Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, and Gaoang Wang. Movicchatt: Question-aware sparse memory for long video question answering. arXiv preprint 1717: 2024.
Tomasz Stanistawek, Filip Graliński, Anna Wröblewska, Dawid Lipinskii Agnieszka Kaliska, Paulina Rosalska, Bartosz Topolski, and Przemystaw Biecek. Kleister: key information extraction datasets involving long documents with complex layouts. In International Conference on Document Analysis and Recognition, 2021.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv: 1811 1aee:811110000000 1,00491, 2018.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312. 13286. 2023a.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint ar 23t::2307.00222... 2023b.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In CVPR, 2024.
Stacey Svetlichnaya. Deepform: Understand structured documents at scale, 2020.
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehension on document images. In AAAI, 2021.
Benny Tang, Angie Boggust, and Arvind Satyanarayan. VisText: A Benchmark for Semantically Rich Chart Captioning. In ACL, 2023.
Jingqun Tang, Chunhui Lin, Zhen Zhao, Shu Wei, Binghong Wu, Qi Liu, Hao Feng, Yang Li, Siqi Wang, Lei Liao, et al. Textsquare: Scaling up text-centric visual instruction tuning. arXiv preprint arXiv:2404. 12803, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11800 2023.
Qwen team.
Qwen2-vl,
August 2024.
URL
https: /wwegm.e..mmmmmm..mm/...mmmm..mm wwt.gh..hhhhoooo http://ggeiinngttubbbo/oooggg
qwen2 -v1/
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya lyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: 1: A fully open, vision-centrie exploration of multimodal Ilms. arXiv preprint arXiv:2406. 16860, 2024a.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal lmss. In CVPR, 2024b.
19

---

Published as a conference paper at ICLR 2025
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. NeurIPS, 2021.
Bryan Wang. Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Au- tomatic mobile ui summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, 2021.
Fei Wang, Xingyu Fu, James Huang. Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding, arXiv preprint arXiv:2406.0944 09411, 2024.
Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:231 /. 03079, 2023a.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305. //75, 2023b.
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. arXiv preprint arXiv:2405.09711 2024a.
Haoning Wu, Hanwei Zhu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Annan Wang, Wenxiu Sun. Qiong Yan, et al. Towards open-ended visual quality comparison. arXiv preprint arXiv:2402.16611 2024b.
x.ai. Grok-1.5 vision preview. URL https::/x..a//blo///rrott
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. NEXT-QA: Next phase of question- answering to explaining temporal actions. In CVPR, 2021a.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In CVPR, 2021b.
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-fie free llava extension from images to videos for video dense captioning. arXiv preprint 169,:,4404,,1994, 2024a.
Mingze Xu, Mingfei Gao, Zhe Gan, Hong- You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. SlowFast-LLaVA, A strong training-free free baseline for video large language models. arXiv preprint arXiv:2407. 15841, 2024b.
Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an Imm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403. 7703, 2024c.
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al. xgen-mm (blip33): A family of open large multimodal models. arXiv preprint 11X1::2408.08872 2024.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arXiv:2310.1144) 2023.
Huanjin Yao, Wenhao Wu, Taojiannan Yang. YuXin Song, Mengxi Zhang, Haocheng Feng, Yifan Sun, Zhiheng Li. Wanli Ouyang, and Jingdong Wang. Dense connector for milms. arXiv preprint arXiv:2405.13880 2024a.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianci Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.00 01000, 2024b.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:23100 200.:2310000704.. 2023.
20

---

Published as a conference paper at ICLR 2025
Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Sweamgin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal llms. arXiv preprint arXivv2404.05779, 2024.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2014.
Licheng Yu, Patrick Poirson, Shan Yang. Alexander C Berg, and Tamara L. Berg. Modeling context in referring expressions. In ECCV, 2016.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02444 2023.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. ActivityNet- QA: A dataset for understanding complex web videos via question answering. In AAAI, 2019.
Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In CVPR, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: massive multi disti-discciline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:231 1.5502 2023a.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05663 2023b.
Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with multimodal large language models. arXiv preprint arXiv:2305.18277 2023.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. CVPR, 2019.
Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang. Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint (0Xi::2444400972 2024a.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, al. Interlinxccompose-2255. A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407. 033200, 2024b.
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimizationn of video large multimodal models from language model reward. arXiv preprint arXiv:2404.01225,, 2024c.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint 277iv:2307.03601, 2023.
Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond Ilava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406. 08487, 2024d.
Zicheng Zhang, Haoning Wu, Erli Zhang, Guangtao Zhai, and Weisi Lin. A benchmark for multi- modal foundation models on low-level vision: from single images to pairs. arXiv preprint 1rXiv:2402.07116. 2024e.
Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding in multi-modal 1lms. arXiv preprint arXiv:2307.0888) 2023.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter:: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402. 14658, 2024.
21

---

Published as a conference paper at ICLR 2025
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint
2017.
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:240..1428, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. TAT-QA: A question answering benchmark hybrid of tabular and textual content in finance. In ACL, 2021.
Yongshuo Zong, Ondrej Bohdal, and Timothy Hospedales. VI-icl bench: The devil in the details of benchmarking multimodal in-context learning. arXiv preprint arXiv:2403. 13164, 2024.
22

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 RELATED WORK
Multimodal Large Language Models (MLLMs) (OpenAl 2024 Islam & Moushi 2024 Team et al.
2023; Li et al. 2024d; Huang et al. 2023) have recently emerged as significant area of research
focus. The development of MLLMs can be traced back to Frozen (Tsimpoukelli et al. 202T and
Flamingo (Alayrac et al., 2022 Awadalla et al. 2023). with more recent advancements such as
LLaVA (Liu et al., 2023b) and MiniGPT-4 (Zhu et al. 2023) introducing the concept of visual instruction tuning. The past year has witnessed a boom of open-source MLLMs, some of which
claim to rival GPT-4o on certain benchmarks. Notable examples include Emu2 (Sun et al. 2023b
2024), VILA (Lin et al. (024b), Idefics2/3 (Laurençon et al. [202aa] Laurençon et al. 2024a).
Cambrian-1
(Tong et al. 2024a), InternI.M-X WWW.MMM-moooo--2..... (Dong et al. 2022a Zhang et al. 2024b
VnternVI.2 (Chen et al. 2024c b). MiniCPM-V (Yao et al.) (024b), VL. (Wang et al.
2023a
Hong et al.
2024a), BL. (Li et al. 2023d] Xue et al.) (024)) LL.VA- Vision (Li al.
2024e).
Llama3.1- (Dubey al. 2024), and the latest Qwen2- vi (Bai et al.. 2023).
Research in MLLMs has expanded across several fronts: (i) scaling up the pre-training data (Lin
et al. 2024b; McKKnnzie et al. [024] Xue et al. 2024] Awadalla et al. [2024] [Li et al. (0024] and
supervised instruction-tuning data (Hu et al. [202bb] Tang et al. [2024] Laurençon et al. [022aa] Tong
ct al. (224a):: (it) enhancing high-resolution image comprehension (Lin et al.) 2023b Li ct al. 20231
Liu et al. 2024a Dong et al. 2024a Gao et al. 2024 Ge et al. g Chen et
2004a Zhang et al.
2024d
Xu et al. 2024c| Li et al. (0241]; (fii) exploring various vision encoders (Tong al. 2024b
Shi et al.
2024) and vision-language connectors (Cha et al. 2024 Yao et al.,
2024a Li et al.
2024k
Cai et
2024) ) (iv) using mixture-offexperts (Lin et al. 2024a Li al.
2024g); (v) extending
LLa VA- architectures to region-level (Wang el al.] [023b; Zhao et al.
et al. 2023
Peng et al. 2023] Chen et al. 2023b] Zhang et al. [023] You et al. 2023 Zhang et al. (024a) and
pixel-level Lai et al. 2024 Rasheed et al. 2024] Yuan et al. [204]] Ren et al. (2024) understanding
multi-image reasoning (Jiang et al. 2024 Li et al. 2024e). UI comprehension ( (ou et al. 2024
Hong et al. 2024b), and video understanding (Lin et al. 2023a Xu et al. 2024a among others.
Among the extensive body of literature on MILMs, MM1.5 distinguishes itself as a significant upgrade over its predecessor, MMI (McKinzie et al. 2024). The MML.5 model family integrates a di- verse set of core capabilities, including text-rich image understanding, visual referring and grounding. and auulti-image reasoning. In contrast, recent ceneral-purpost MLLMs such C (Tong jet al. (2024) and LLaVA-One Vision (Li et al. 2024e) have shown less satisfactory performance in handling referring and grounding tasks, and GPT-40 has rely on set-of-mart (SoM) prompt- ing ( (Yang et al.
2023 to understand image regions.
While several recent works have open-sourced detailed SFT data mixtures for public use (Laurençon eet al.] 2024a] Tong et al., (024a)), the precise impact of each data category and the best recipe to combine them remain under-explored. This particularly true for models requiring diverse capabilities. MM1.5 stands out by providing a comprehensive empirical study that presents mature recipes for building performant MLLMs. The extension of MM1.5 to mobile UI understanding further enhances the uniqueness of this work.
Another emerging trend in the field is the development of lightweight MLLMs for potential edge
deployment (Jin et al. 2024b; Huang et al. 2024, Beyer et al. 2024 Lu et al. 2024 Hinck et al. 2024, Li et al.] 20241 Zhou et al.] 2024] He et al., 2024). In MM1.5, .5, models with 1B and 3B parameters are offered, which outperform similar-sized models, such as Phi-3-Vishe (Abdin et al. 2024b) and MiniCPM-V (Yao et al. 2024b).
A.2 PRE-TRAINING ABLATIONS
Beyond the SFT and continual pre-training, we emphasize the importance of large-scale, task-specific data used during pre-training in establishing robust foundations for models to effectively handle diverse tasks. For knowledge-heavy benchmarks like MMMU (Yue et al. 2023a), we found that model performance is highly sensitive to its text comprehension capabilities. The LLM's ability to understand and process textual content is pivotal in addressing the complex reasoning and knowledge- representation challenges posed by these benchmarks, as also observed in Cambrian-1 (Tong et al.) 2024a).
23

---

Published as a conference paper at ICLR 2025
settings in MMI (45:45:10) HQ-Texx (45:45:10) HQ-TeHTT -Txt (50:10:40)
78.12
79.5
76.41
80
Average Score
70
62.64
49
65.55
68.
06.22
63.9 00.00
65
29
62.
63.
50 00..5
56.41
57.20
58 28
2.24
60
60
50
Text-rich
Knowledge
General
Refer& Ground
Multi-
Average
image
Figure 7: Performance comparison of all categories across different text-only data and pre-training data ratio. The figure highlights the performance improvement when replacing with HQ-Text data and the additional gains achieved by adjusting the ratio to 50: 10:40. Note that the default setting for continual pre-training (OCR) and All Mixture for SFT are used for all models.
We incorporated a higher-quality and more diverse set of text-only datasets, referred to as HQQ iix-Tex,, introduced by (Gunter al. 2024), during the pre-training phase. These datasets specifically curated to enhance the model's 's language capabilities by providing deeper and more varied textual contexts, with focus on general knowledge, mathematics, and coding. This update aims strengthen the model's ability in language-basse baaed reasoning.
As shown in Figure 7 by simply replacing with the new data, the average score on knowledge improves by 0.85 points.
In conjunction with the text-only datasets and the latest SFT recipes discussed in Section 2.2] we further refined our pre-training data composition. The original data ratio proposed in MM1 (McKinzie al.) 2024) was 45:45:10 10 for image-caption, interleaved image-text, and text-only data, respectively. Further experiments revealed that decreasing the amount of interleaved pre-training data and, respec- tively, increasing the weight of text-only data a ratio 50: 10:40 resulted improved performance across most tasks after SFT. We note that in contrast pre-training ablations in MM1, for MM1.5, we conduct evaluations on downstream benchmarks post SFT to select our final pre- pre-training mixture. We hypothesize that relying primarily improvements on such evaluations may not effectively transfer to downstream performance. Our newly optimized data mix for MM1.5 not only enhances multimodal capabilities but also strengthens language understanding, leading to superior overall performance across benchmarks.
few-shot pre-traine pre-training metrics may not be ideal, as the
With the updated mixture, performance text-rich average increased by 0.85, knowledge average by 0.99, and refer&ground tasks by around 1.4, as shown in Figure 7 Although there was slight decrease of 0.05 multi-image datasets due the lower weighting interleaved data, we consider this trade-off reasonable for maintaining strong performance across all tasks.
A.3 DETAILS OF SFT DATA FOR ABLATION FINAL SET MIXTURE
As presented in Table 5
subset of final SFT data when conducting the ablation study.
The MM1.5 final mixture is presented in Figure
A.4 MM1.5 BENCHMARK DETAILS
Benchmarks used for MM1.5 evaluation summarized in Table where for each Category Average Score, we directly calculate the average of each metric number within that capability category as follows, with detailed evaluation metries listed in Table
24

---

Published as a conference paper at ICLR 2025
conversation
(5677)
LLAVA V1.5 YQAZ OKYQA Marino al
LLAA v1. GQA Hudsoa
Manning
AUDKYO
Capton Caaamcnatt Liat
2015
(82.8k)
(66.2k)
Reforing Grounding
1.L4VA Complex Remsoning Lin
2023b
StareGPPT-Y Chen al 2023
(66.
Image
Tex:Caps Sidoouu
2020 (22k)
Knowlrdge
SFT Meture
Only
Synthdog
GcreeQAAllSa
Ei Kim et
2022
(80.5)
(500k)
ArivQA Li al. RUZ3 (T00k)
WikiSQU
(75)
WikEE
(38.2k)
Ganeral
TabdW
41450
[=247k]
TextVAA
2019
(39.00)
ST-VQA 1322 Srap 2019
(17.2k)
Renderdd
10st
VisTeu
Pinn)A dr 2012
DeepForm Syeneii
2120
Q/ZZu
HiTal (Thah or 122115511
DVQA Kane ar
2071
(200))
Chart Comnooiiissyyy 23
(66.7k)
DocV Morriieewwww 21 1D
2021
(12%
5k0
Into 211.0:1011111110000
2022
(52.3k)
Wikilabbl 135 (roups Loung 2015
TabFaat
VisualMRE Treak
Kleider Laris ww..hhawwkwkkk..........k. ctall
2021
(27/7k)
OCRVQA Mistira atal 2010 (ilkk)
TextV
Singlreru TeatVQ&ASngglreee
2012
[30,00]
Knowledge (MathhSeicnce
CLEVR et
2017
(70k)
lecaQA Lueeral 2021h) (27.3k)
2haang et al 2019 (623)
Inter GPS
et
(202aa
(133k)
Geomn Kazemi el
2021 (9.3k)
A2D/Rembbuy Kembluy al Nu (3.2k)
ScienceQ +
ATP
(5k)
WebSigh Laurengun et
20226 (10k)
DallkZ
Design2
Referving Grounding
Visual Genome
2023
Krishna
You
2023
(267
(76.6))
GRIT-FFick s0 You at (20038)
2023
Plommer 2t 2015
Keceeno You et 20223
Lin
2014
GRIT-Sparial Negstive Mining
You ot
Yu al 2016 Nagaraja of X2000
(210))
2023
Stao et (2019) (213.6k)
Multi-image
Birds-to-Woodk Jiang
2024
Forbes
Cainstract
al.
2024
Wi
2019 (2.6k)
31)
DreamSin Dinge al
2011 Tut et
2023 (15 (15.9k
IooRQA Jiang et 2024 Lae 2021b] (61 5k
Kip ViA et 2024
NLVR2 Taing ofall
2024 Sine at
0185666443
Sput-ihe-dnijj-aa Fang et 2024 Thamlami & Berg
NEXT-QA Trang ei at
2024
Xino
Kirspanek (8k)
20218
(3.9k
Star Jiaag al.
2024
Wi ct
20240 (3k)
Ias-usst (0.4k)
Coco instruet intesteaved (2.1k)
Text
OpenOrea Lin 2023
(944)
Mathlastru: Yue 2023h (2622)
OrcaMatk Mitta
2024
(200))
WianlCoder
OpenCockhiiepeettrr
2024
(66k)
Dally Mike et al
2023
(11))
Figure 8: A high-quality data mixture used for MM1.5 supervised fine-tuning, including (i) single-image data for enhanced math/science reasoning. text-rich image understanding. and visual referring and grounding, (ti) multi-imag image data, and (iii) text-only data. (f) denotes in-house datasets with curation details in Appendix A.9
Table 5: Overview of the SFT data used in ablation study.
Data categoty
Sub-category
Dalasets
QA
L.LVV Complex Reasoning, Conversation, ShareGPT-4v, Cocu Capsion, LLaVA v1.5 VQA22 OKVQA, LLaVA GQA,
General
542K
A-OKVQA
OCRVQA, Synthdog.FF Fn, TexiCaps, TexVQQ. DVQA, CharQA,, Do:VQA, InfoVQA, VisualMRC. WikiTQ, DeepFoom, ChhrsterChharity TabFact
Rich
Single-image
Gisua-isual Genome, GRIT-Ragion reasoning. GRIT-FFickrOO., GRIT-Refsoco, GRIT-Spatial Negative Mining
Refer&Grrondd
1.8M
Science
ScienceQA
Math
GeomVerse, CLEVER, IconQA, RAVEN, Inter-GPS
150K
WehSight, DaTikZ, Design2Code
DreamSim, NLVR2, Star. Birds-to-Wordd Words, IconQA. Spot-the-diff, ICL-instruct, Coinstract, MultiVQA. NEXT-QA, Coco Instruct Interleaved
Multi-image
324K
Dolly, OpenOrea. Mathllstruct, WizardCoder, OrcaMath, OpenCooklneepprtter
Text-only
1.7MM
General Average Score:
average of the corresponding metric scores from MME-Normalizg Seed-IMG, POPE, LLaVAW W. MM-Ve. Vet and RealWorIdQA.
Text-rich Average Score: average score of the corresponding metric scores from WTQ, TabFact, OCRBenct a TexIVQA, DocVQA and InfoVQA.
'MME-Normalize (MME-Perccption + MMF-Cognition/
100%.
"The accuracy of OCRbench the total score normalized by 1000 x 100%.
Average of human part accuracy and augmented part accuracy.
25

---

Published as a conference paper at ICLR 2025
Table 6: Details of benchmarks and their metrics used in MM1.5 ablation study. Benchmarks marked with (f) are excluded from the category average.
Canggoy
Benchutark
Metni
MME Fu et
2021a
Nomallzzd Accuracy
SEED
Bead-MGG
General
POPE
of random, popular advensarial score
MM-Var
GiT-assisted score
Red/Worine Kaa
Accuracy
Paeyril & Libny
2015
Accuracy
Tabrus
A19
Accuracy
OCRBen-n
Accuracy
ChunQA Marve
Accuracy
TextV (Sishh
VQJ Open Hamning Accuracy
DocV Kcallew et
2124
ANLS Score
InfoVQA Xrainiee 0
ANLS Score
Flickr3K Youns cl
2014
Recall (loUs-0.5 any protacel)
Cupia
2023
Accuracy Recalld (lols00
Refetio
Refeacor
Keventiizll
Recall
Releucog
KazemzMich cla
201
Recall (loU>0.5)
Ferat-Bekk Bench
You et aI:
GPT-assisted score
A12D
Kemblyni et
7016b
Accuracy
Knuwledge (MathoScieme/CCeee))
Scicho&
KU543
GPT-assisted score
Accuracy-
MMMU
Accuracy
@tencha Zrang et al.
30240
Accuracy
Mamin piin ce
4070
Accuracy
Multi-lmag
NT NuFe
BLINKITTCC
X013
Accuracy
Accuracy
MVbend
Accuracy
Muabench Wang el
M124
Accuracy
Refer & ground Average Score: average of the scores of Flickr30k, RefCOCO avg. and LVIS avg.. where RefCOCO avg. is the average of RefCOCO A, RefCOCO B, RefCOCO+ A, RefCOCO+ B and RefCOCOg, and LVIS avg. is the average of point and box metries.
Knowledge Average Score: average score of the corresponding metric scores from A12D, Sci- enceQA, Math and MMMU.
Multi-image average score: average of Qbench, Mantis, NLVR2, BLINK and MVBench metric scores.
MMBaseScore: average score of the General Average Score, Text-rich Average Score and Knowl- edge Average Score. This aggregated metric is used in Section 2 to measure the impact of the general, text-rich, and knowledge capabilities of a model.
A.5 IMPACT OF SYNTHETIC CAPTIONS BY SELF- TRAINING
Besides using public captioning data mentioned Section 2.3
we also follow (Fang et al.) 2024)
to study the effect of self- seff-training using synthetic captions. This is particularly important because captioning data generated by black-box commercial models can sometimes be difficult to scale and data from open models, such from the LLAVA-NeXT family may be constrained by these model's inherent limitations. We develop a self-augmented image-caption data engine building on our previous work, MMI (McKinzie et al. 2024) with a goal to provide high quality captions in computationally efficient manner. Specifically, we fine-tune a pre-trained 3B MMI-style model on a mix of synthetic and approximately 8k human-annotated paragraph-length image captions (approximately 70 tokens on average). We then apply this captioner at scale to 290 million web-crawled images with resolutions ranging from 512 to 1024px. Following the approach in (Liu et al. 2024 we perform concept filtering based on the generated captions, resulting in a dataset of 7 million high-quality captions, which includes numerous text-rich examples and alt-text-derived knowledge.
Building on this, we investigate the impact of the volume of synthetic caption data. The synthetic captions, ranging from 1.4 million to million, show consistent improvements across various metrics, illustrated in Figure Specifically, ratio of 0 on the x-axis indicates that no image-caption data is added to the original OCR dataset, while ratios ranging from 0.2 to 1.0 on the x-axis represent the proportion of the total 7M dataset included in the training. For example, ratio of 0.2 corresponds to approximately 1.4M image-caption pairs added with the original OCR data while 1.0 denotes all 7M added into training.
model performance
26

---

Published as a conference paper at ICLR 2025
Text-rich Average
Knowledge Average
MMBase Average
6..00
60.8
53.5
60.4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.0
3.0
Ratio of Caption-Data Continaal Predtdiing
Rello of Captien-Dataa Cuntinual Phedtriining
Rekio of Capi-oo-Daata in Continual Prestraining
Figure 9: Impact of synthetic captions for continual pre-training the 3B model, building on top the final MMI.5 strategy introduced in the main text, í.e., including the OCR continual pre-training stage. We report the impact adding incrementally more synthetic captions, up to 7M in total.
In contrast the ShareGPT4V-PT and LLaVA- MaaV--ecap--31 captions as explored in Section 2.3 we find that adding our in-house synthetic captions to the OCR data mixture can lead to consistent improvement for continual pre-training We observe improvements in knovledge-related benchmarks and the aggregated MMBase scores that further scale with increased data volume. This is especially notable since our in-house captioner uses only a 3B model while LLaVA-Recap, for example, uses 34B model for captioning.
Our results suggest that while a comparatively simple OCR mixture represents a strong baseline for continual pre-training data, high-quality captions can still lead to further improvements. However, the quality, distribution, perhaps even style and length the generated captions seem crucial to realize gains. While our in-house captions empirically outperformed publicly available data in specific setting, further research is necessary as to what, specifically, these improvements are attributable to and whether further improvements be achieved. This investigation goes beyond the scope of this paper, and we aim to study synthetic captioning further in future work.
A.6 COMPARISON WITH SOTA MODELS
We compared our model with selected benchmarks Section 3 more baselines extensive benchmarks in Table 7 (Knowledge & General), (Text-rich), Table 9 (Refer s Ground), Table 10 (In-context Learning), and Table [T (Multiimaage)
Here, we present comparisons with
'Experiments in this section based the final recipe from Section with slightly different settings compared those Section A.2
27

---

Published as a conference paper at ICLR 2025
Table Comparison with SOTA models on knowledge and general benchmarks. (1) The score the summation of perception and cognition scores. Gemini- 1.5-Pro, GPT-4V and GPT-4o numbers from Open VLM Leaderboard
Knowledge
Benchmarks
Model
A12D (test)
SQA (les))
MMMC (val)
MathV (lestmini)
MME (PIC)
SEED!
POPE
LLaVA
MM-Ner
RealWoridQA
18 Model Comporison
LLaVAOne 5
2024c
57.1
67.2
31.4
34.8
1238 02400
65.5
55.6
SPHINX
DeepSock-V1
MPA
32.2
31.1
87.6
T aF M14
60.1
86.1
6088
25.8
Gemin Nant Toom cC
2025
37.9
26.3
27.3
VL2-28 Ciai
74.1
94.1
1864.3
70.9
85.2
60.0
57.4
Mckiziis ar
MM1.5-18
59.3
82.1
35.8
37.2
1365 7/245.7
70.2
88.1
71.6
53.1
BB-MME
67.1
87.6
41.2
42.9
1511 9/361
71.4
88.6
75.5
39.8
57.8
JB Model Comparison
MisiCPM-V 30-30-1000 You
2024b
55.8
67.9
TinyLLaVA Zhyu 2h
69.1
1461.91-
86.4
75.8
32.0
Gemin Nant Tcom
2025
$1.0
32.6
30.6
Bunny fl a 7074
78.3
41.4
1581 5/361
72.5
87.2
60.5
202
MM1-3B McKinizi or 2009
62.4
33.9
32.0
5/279
68.8
87.4
72.1
35.8
MM1 5-355
65.7
85.8
37.1
44.4
1478 A/319
72.4
88.
73.0
41.0
56.9
MML.5-3 3B3B-BB MoE
69.9
89.8
42.9
46.9
1591 A1365.
73.3
87.2
76.1
43.7
60.7
7B Model Comparison
NB
2024a
1519.0335..0
MM1-7B Mckimiie et
66.0
72.6
37.0
35.9
1529833328.
69.9
86.6
81.5
42.1
55.7
MM1.5-2
72.2
89.6
41.8
47.6
1514.91366.4
73.4
88.6
74,2
42.2
62.5
308 Model Comparivon
[LaVA-NeXT-34B)
MMI-300 McKinzii stale MM1.5 5:306
2021
73.3
81.0
11.7
39.4
1637 6433
72.1
87.6
89.3
59.1
77.2
91.9
47.4
55.6
1646 21405
75.0
88.6
80.4
$2.0
69.0
Gemani- 1.Pha etid M12
79.1
85.7
60.6
57.7
2110.6'
88.2
95.3
64.1
GPT-4V OrenAl
21.6
93.1
56.5
GPT-4u Kamm & Moreli
2024
$4.6
90.7
69.2
61.3
2310.3'
77.1
85.6
102.0
69.1
75.4
28

---

Published as a conference paper at ICLR 2025
Table 8: Comparison with SOTA models on text-rich benchmarks. Numbers marked with (f) are obtained from Li et al. (2024c).
Text-rich Benchmarks
Model
WTQ (lest)
TabFact (test)
OCRBench (test)
ChartQA (test)
TexVVQ (val)
DecVQA (test)
Info (test)
1R Model Comporison
LLaVAOneVisioon-O 5B Li
2024c
61.4
70.0
41.8
SPHINX- -iny Gao
51.1
34.1
57.8
53.0
26.3
DeepSeck-Vi Lü al
M74
40.9
TinyLLaVA Zixu al 2024
51.7
Gemini Nano Team
2023
53.6
62.5
72.2
51.1
InterVLL2-2B Chen et 2024b
35.8
56.7
78.1
76.2
73.4
86.9
58.9
MMI-IB(McKinziee
49.8
56.6
68.2
68.4
38.5
MMI.5-1B
34.1
66.1
60.5
67.2
72.5
81.0
50.5
MML.5-I--Mol
38.9
71.4
62.6
73.7
76.1
84.8
55.9
Model Comporison
MiniCPM-V 2.0-3 Yao al. 2024b
24.2
58.2
60.5
74.1
71.9
37.6
Tinyl TinyLLaVV La Zhou et al 2114
59.1
Gemini Nano Team
2023
74.3
54.5
BLIP-3-IB Bucc
Xuc al 2024
71.0
Phi-3-Vision- Abdin al 2024b
47.4
67.8
63.7
81.4
70.1
83.3
49.0
MMI-3B McK-ize
52,9
57.0
71.9
75.2
44.7
MM1.5-3B
41.8
72.9
65.7
74.2
76.5
87.7
58.5
MM..5-3B-MoE
73.1
63.8
73.6
53.6
Model Compurison
L.aVA-NeXT-78 I Liuce 2024a
64.9
Idefics2-8B Laurençon et 20243
73.0
74.0
DocOwl-1 Chal
MMI-7B Mininzie et
28.8
55,5
62.6
72.6
72.80
76.8
45.5
MML.5-7B
75.9
63.5
78.6
76.5
88.1
59.5
30B Model Comparison
LLaVA-NeXT-311 Liu 2024a
69.5
Cambrian 34B Tong al 20.4
60.0
75.6
76.7
75.5
MM1-30B McKinzie
MM1.5 30B
54.1
84.0
65.8
83.6
79.2
91.4
67.3
Gemini-1 5-Pro Reid et al. 2024
75.4
87.2
78.7
93.1
81.0
GPT-4V ApenAl 2024
64.5
78.5
88.41
GPT-4o Islam Moushi
2024
73.6
85.7'
92.8
Table 9: Comparison with SOTA models on referring and grounding benchmarks.
Refer and Ground Benchmarks
Model
RefCOCO (lestA/B)
ReiCOCO+ (lstA/BB
ReiCOCOg
Flickr30k (lest)
LVIS Ref (box/point)
Ferret Bench (avg.)
(test)
Model Comparison
SPHINX-Tiay Gao al MMI-IB McKnzze af
2024
86.9/77.9
78. 78.5/63.7
78.9
1-M1--11B MM-
89.3/81.9
83,7/69.3
82.8
83.0
69.7/54.7
67.4
86.0/73.0
7154/
.7 Model Comparison
MiniCPM-v2-38 Yao et al 2024b
48.2/47.7
22.1
Phi- Abdin
46.3
28.8
53.8/54.5
32.2
Intern VL2 2F Chen ot 202-16
88.2 75.9
82.8 63.3
78.3
51.6
$1.0/111
35.0
MM1-3B MML.5-35
52.9/53.9
92.0/86.1
87.7/73.9
86.4
85.9
76.3/59.5
69.5
79.3/54.5
7B Model Comporison
Qwen-VI..77 Rai al.
2023
92.3/84.5
88.6/76.8
86.3
MiniGPT-v2 7B Chen
2023g
91.3/84.3
85,5/73.3
84.3
I...VA-One Visian- Abdin el al, 2024h
80.0661.6
76.9/56.2
70.0
50.1
51.2/5 1.4
38.4
Ferrei 7B You et 20155
91.4/82.5
$7.4/73.1
84.8
82.2
79.4/67.9
64.5
Ferret-V277//haann et
2024n
94.7/88.7
92.8/79.3
89.3
85.8
86.6/7..6
75.6
MM11-B Mckmzze
0/0
0/D
53.1/53.3
48.5
MMI.5-78
92.5/86.7
88.7/77.8
87.1
85.3
79.4/53.4
72.6
Larger (>> 13B) Model Comparison
Ferret-138 You et 2025
92.4/84.4
88.1/75.2
86.3
84,8
80.5/68.4
66.3
Ferret-V2. Zann
2024g)
95.0/88.9
92.8/81.4
87.7/55.1
74.9
MM1-30B Me Kiozic et
2024
0/O
0/O
0
53.4/52.7
50.9
MMI.5-301
29

---

Published as a conference paper at ICLR 2025
Table 10: Comparison with SOTA models on VL-ICL benchmark Zong et al. (2024) for multimodal
in-context learning. 4-shot accuracy reported for each subtask.
VL-ICL. Benchmark
Model
CLEVR
Matching MiniTinageNet
Open MinilnageNet
Operator inductioo
Operator induction interleaved
TexOCR
Avg.
1B Model Comparison
MMI-IB
MMMiniBMMKinziie McKinzio et 2024
25,0
49.3
73.0
16.7
33.5
MML.5-18 MM-.5-IB-MoF
39.0
52.0
84.0
60.0
36.7
34.0
51.0
3B Model Compurison
Phi-3 Sili----iiinnndi 4B Ahdiin al
22044
17.0
50.0
1.0
26.7
14.0
19.5
AMMI-BBMMKiin McKinzze 2024 MM1.5-38
27.5
50.0
79.0
18.3
13.3
34.0
37.0
33.5
59.0
88.0
48.3
66.7
42.5
56.3
MM1.5. 5.3...33-MM0 MoE
32.0
58.0
92.0
63.3
65.0
47.5
39.6
Model Comparison
OpenFlamingo 9B Awadalla
2023
18.8
50.0
51.2
2.8
2.8
0.0
20.9
Idetics-98) Lourena Cral 20248
27.7
50.0
53.8
7.8
22.8
28.0
Otter-9B (eer-99/Lree Eraral 20230
8.2
50.4
28.5
12.2
7.2
0.8
17.9
Qwen- Bioa-VV-Chaat-00 Ba Bai el
2023
26.8
56.4
58.0
18.9
8.9
22.3
31.9
NeXT-77 hirer 2024m
17.8
50.0
3.3
12.7
MM1-7B McKinzie el 2074
33.0
69.5
97.5
40.0
32.0
MM1.5-7B
25.5
52.8
98.5
68.3
60.0
31,0
$6.0
Targer Model Comparison
Idetics-80B- 1an--88BBLaaagg Laugencon 2024b
31.5
50.0
52.5
Ema2-Chat-33 Sin er an 20.40
14.8
50.0
28.2
21.7
10.0
36.5
26.9
MM1-300 300 McKinge
25.0
63.0
98.5
51.7
38.3
36.0
52.1
46.5
65.0
GPT-4V OpenAl 2024
42.0
81.0
56.0
92.0
74.0
50.0
65.8
Table 11: Comparison with SOTA models on multi-image benchmarks. The result with mark (f) in the row of GPT-4V is from GPT-4o. MVBench [i al. (2024h) is treated as imag
benchmark to test the zero-shot transfer capability of MM1.5 1.5 to video understanding tasks.
Multi-image Benchmarks
Model
QBench2 (val)
Muntis (test)
NLVR2 (val)
BLINK (val)
Muirbench (test)
MVBench
18 Model Comparison
LLaVA-NecT--nterlave.00 Lict
2024f
52.0
45.6
67.8
45.6
39.2
LLaVAOne (.55nn-0.5BB 2024C
48.8
39.6
63.4
45.5
52.1
25.5
MM1-IB
AMIlIB/McKinziee McKinzie et dt, 2024
43.4
50.9
43.8
40.3
30.7
MM1.5-IB
66.4
50.7
79.0
45.8
46.3
34.7
MM1.5-IB-MoE
70.9
51.2
83.2
48.3
43.7
40.9
3B Model Comparison
BLII-334BB Xue al. 2024
75.1
56.7
49.7
Phi-3- Visioonn Abdin 2024b
56.8
47.9
53.6
46.7
44.2
38.0
MM1-3B McKinzie et at. 2024
41.4
51.7
44.8
41.5
28.0
MM1.5-33
83.8
46.8
44.3
MM1.5-3B-MoE
73.8
54.4
86.0
50.3
49.8
45.6
7B Model Comparison
LLaVA-v1.5-70 Liu
2023a
49.3
31.3
53.9
36.0
37.1
23.5
MeX---NeXT-nntrreea
ef al. 2024r
74.2
62.7
88.8
53.1
52.6
38.9
Idefics2-8B Laurencon el A014a
57.0
48.9
86.9
29.7
45.2
26.1
Mantis-Ideries2:8B.
57.1
89.7
51.4
44.5
MM1-7B McKinzic et
43.6
51.6
59.9
45.3
40.0
30.4
MM1.5-75
73.2
57.6
86,9
48.3
48.2
49.1
Larger (> (d14)) Model Comparison
LLaVA-NeXT -en.----eT--ntreeeneee 14B Li al.
20240
76.7
66.4
91.1
54.9
52.1
Emu2-C Cha--hhht 37B Sun et 2023a
50.1
37.8
58.2
39.7
36.2
33.6
MM1-30B McKinzie
43.5
36.7
MM1.5-30B
79.3
64.6
90.6
54.0
50.2
58.2
GPT-4V OpenAI 2024
62.7
88.8
43.5
$1.1
68.0
30

---

Published as a conference paper at ICLR 2025
A.7 AMM1.-VVIDER
The multi-image reasoning capability shown in MM1.5 naturally leads us develop MM1.5- for video understanding. It takes a video and an instruction as input and generates the response. For the inputs, we uniformly sample N frames from the video at arbitrary length and feed them into the model as multi-image inputs without special frame assembly. Due to the token limits, we disable the dynamic image splitting for each frame, and the vision encoder generates the feature maps couee-b--frane independently. Specifically, we sample 24 frames for each video, and each frame is represented by 144 tokens.
We introduce two variants for MM1 1,5- Video. First, we build MM1.5 --5ee as training-free model, which is achieved by directly adopting the pre-trained MM1.5 image models to video tasks without being fine-tuned on any video data. This saves lot of computation resources and demonstrates MM1.5's capability of transferring knowledge to new domains.
Second, we introduce the supervised fine-tuning (SFT) model where we fine-tune MM1.5 image models on video instruction- tunnin datasets improve its temporal modeling capability for video tasks. We use mixture of public video datasets from ShareGPT Video (Zhang al. (024c) (556K), VideoChat2 (Li et al. 2023e) (225K), and Netivity (Yu et al. 2019) (31.5K). These datasets contain a variety of videos types, spanning different tasks (e.g., open-ended and multiple choice questions), viewpoints (e.g., first- and third perdon views), and lengths (e.g.. videos from a few seconds to tens of minutes).
A.7.1 BENCHMARKS AND METRICS
We compare our video training-free and SFT models with state-of-the-art methods multiple video question-answering (VideoQA) tasks and benchmarks.
Open-Ended Benchmarks evaluate the performance of model answer questions in free-form style. For this task, we include ActivityNer-QA (Yu et al. 2019) and VCGBench (Maaz et al. 2024). Following prior work (Xu al. 2024b), use GPT-3 and score for the prediction. Considering that the labeled answers of these two datasets are typically short (e.g., one word or phrase), we also evaluate on the LLa requires the model to generate more detailed answers. This is useful for assessing performance on tasks involving detailed video understanding. We follow their original setting to report the score from GPT-3.5 3-5--urbo--
turbo-022 to assess the accuracy
(Zhang al. (024c). which
and consider score value 2 3 as correct for accuracy calculation.
Multiple Choice Benchmarks require the model to pick the correct answer from multiple choices. For this evaluation, we include VideoMME (Fu et al. 2024b), EgoSchema (Mangalam et al. (024). NEXTQA (Xiao et al., 2021a). and IntentQA (Li et al.) (023c)). VideoMME is a comprehensive evaluation dataset containing video from few seconds to hour in length. EgoSchema consists of egocentric vidcos and involves complex long-form temporal understanding and reasoning. NExTQA and IntentQA collected from the same video source, but IntentQA focuses predicting intents in daily social activities. For all these datasets, the accuracy of selecting the correct answer from the options is used the evaluation metric.
A.7.2 RESULTS
Training-fres results are shown in Table 12 and 13 MMI.5-Video demonstrates greater capability on already outperforms state-of-the oh-the-rrt training ffree
Multiple Choice VideoQA, where MM1.5 MMI15-Videoo Video-3B 7B models on all benchmarks. We also find that MM1.5- Video can follow the instruction to precisely output the predicted option; however, most existing methods (Li et al.) 2024h) use structured answer prompts (e.g, " Best iption: (") to guide their models to generate answers in a desirable format. On the other hand, MMI.5- achieves only on-par performance compared to SlowFast-LLaVA on the open-ended benchmarks. We hypothesize that this is because our multi-image SFT datasets contain primarily multiple choice tasks, making such a task formulation most similar the training data.
First, we observe that fine-tuning MM1.5- 1-1.5-Vidco on video datasets can improve its performance on all tasks. Second, on both open-ended and multiple choice benchmarks, small model, MM1.5- significantly outperforms L.La VAOne VAOneVision-
SFT results are also shown in Table 12 and 13
31

---

Published as conference paper at ICLR 2025
Table 12: Comparison with SOTA models on Open-Ender and Multiple Choice benchmarks.
Opea-Ended Benchmarks
Multiple Choice Benchmarks
Model
Data
ActivityNa-QA
VCGBench (lest)
VidcoMME subs)
EgoSchema (subset)
NEXTQA (val)
IntentQA (val)
(est)
Training Free Model Comparison
DeepStack-L LB- Meng al
2024
49.3
38.4
61.0
IG-VLM-7B (LLavevvo
SlowFustLLLaV, 2D
202216
x
55.5
3.04
40.7
47.2
64.2
60.1
MM1.5-Video-B (Trainnng
2,86
45.6
45.4
Milee-Viido---- (Trining free)
50.9
3.04
48.4
48.4
72.8
72.7
MM1.5- Vidoo-88 (Training-froe)
$2.5
3,05
52.4
49.6
76.1
76.7
SPT Model Comparisow
VideoChntGP1:76 Monz dl
2024
35,2 2
2.42
2.84
Vsta-LLaMA-78
48 3
60.7
MovieChatt--78 Song
48.
54.8
VidesChat2-7B LTer
Vide-LLaMA2-755/
Cien cl clal
2024
50.2
3.13
47.9
51.7
PLLaVA Xu
56.3
LLaVA-NeXT-mmereen Uhis
20241
3.07
59.5
ILaVA-NeXT-lnterrl Lirc
55.3
3,42
78.2
LLaVAOneVision-
26.8
LLaVAOmeVisione 21 202200
56.6
3.51
58.2
60.1
79.4
MMI.5-Video-B (SFT)
56.1
3.14
45.7
51.0
71.8
74.2
MM1.5-Vida-388 (SFT)
57.9
3.17
49.5
52.4
74.7
81.2
MML.5-Video778 (SFT)
60.9
3.22
$3.5
57.2
76.9
86.6
Table 13: Comparison with SOTA models LLa benchmarks. (+) indicates the published version released at https:: / /ttggguuggggg// ingfaacce face. co co.ShareGPTV Lew.iiddee/LLLaAaoAA.oa SFT
Model
In-domain Benchmarks
Out-of-domain Benchmarks
ActivityNet-QA
VIDAL-QA
WebVid-OA
MSVD-QA
MSRVTT-QA
TGIF-QA
SSV2-QA
Vidko-haiGGPT-79 Maaz et al. 2024
34.2
29.4
38.9
34.1
25.7
31.4
19.4
LLaMA-VID e dl 20123
36.5
30.6
37.0
34.1
25.0
27.2
22.2
Chat-Uniiii-7Biiig
39.4
31.4
40.1
35.6
33.2
20.6
Vido-LLaVA-788 cE
LLAVA-HOUNI-SFFI 20230
41.4
34.3
42.5
39.5
30.8
33,0
24.3
6288
56.3
66.8
62.2
$2.6
61.1
35.4
MMI.S-Video-1B (Training-fic)
49.0
55.8
49.8
47.6
MM1. V-Vicoo-3 (Training-free)
$1.5
45.4
58.5
51.1
46.0
49.2
28.2
MMI.5-Videe77B (Training free)
5288
48.7
58.5
51.9
48.1
49.8
30.4
MM1.5-Video-1 (SFT)
65.7
MM1.5. 3B (SFT)
67.8
63.4
71.1
65.2
57.2
64.9
35.2
MMI.5-Videe-7H (SPT)
68.5
68.5
21.5
67.2
65.5
0.5B (e.g., 24.2% EgoSchema and 14.6% NExTQA) and achieves the state-of-the-art results. Third, our 7B model achieves state-of-the-art performance on ActivityNet-QA (e.g., outperforming LaaVAOneVision-- by 4.3%) and very strong results (mostly runner-up) on other benchmarks by using only public video datasets. We are impressed by the superior results of LLaVAOne Vision-7B. especially on long-form video benchmarks such as VideoMME and EgoSchema. We hypothesize this can be due to that (i) it is trained on their re-annotated video datasets with better labeling quality, (ii) takes more video frames as inputs (i.e., 32 vs. 24), (fid) uses multiple training stages on joint image and video datasets. We will explore these directions to improve model in future work. Lastly, MML.5- Video achieves state-of-the-art performance on the LLaVA-Hound benchmarks, which demonstrates our capability for detailed video understanding.
A.8
1M1.5-U1
One of the most promising applications of MLLMs that has recently gained popularity is using them to understand and act on user interfaces (Uls) on behalf or alongside users (Hong et al. 2024b) (aachler et al.) 2024 You et al. 2024 Li & Li 2023), which could significantly boost users' productivity and efficiency when interacting with digital devices. This application typically involves providing a model input of: (i) an image of the graphical user interface (GUT) of device (i.e., phone computer) screen; and (1z) instructions on either knowledge grounded on certain areas or the entirety of the screen (e.g., Is this element <x1,x2. 1,y2> clickable?), asking refer to certain areas of the screen that fit the questions' criterion (e.g., Where is the text 'login' on the screen?). Beyond referring and grounding abilities, excelling on UI tasks also requires text-rich image understanding ability to
32

---

Published as a conference paper at ICLR 2025
Referring
Grounding
Widget classification Icon recognition OCR
Find widget Find loon Find text
Sottings box
Display Brightness
Classify the Ui type of
APREARANCE
Where is the widget of Dark Mode?
the widge: <boxt>.
9-41
(143
Checkbox
Dark Mode <bexO>
boxo
Identify the icon type of the widget <boxl>
Where is icon Settings?
Light
Dark
settings
Settings <cborit>
Automra wiic
What lext is displayed in element <boor
Where is the Lext of "BRIGHTNESS"?
box2
BRIG-ITNESS
BRIGHTNESS
<bow2>
Thu Tone
Function Inference
Conversation
andiieet oo viniins make overo appoir cono stont oitenon
can do UI?
What "Night Shift??
Night Shift
Off
This screen all ows you to customize the displey and brightred ess settings of your device. You betweon light and dark modes, with automatic mode also available. You can also adjust brig ntness level manually. There are options to turn off features like right shift and nuto- lock, and you can the auto- look time to 30 seconds. Lastly, there aption to change text size.
"Night Shift" feature adjusts the screen's color temperatuure
box3
choase
Auta-Locck
30 Seconds
hues,
Raise Wakke
How
on? Answer with bounding boxes.
You can turn on "Night Shift" tapping on the "Off"" > button located a: the coordinates <box->>
Text
Figure 10: Illustration of the UI understanding capability shown in MM1.5-UT. Our single model is able to perform a variety of referring and grounding tasks and establish new state-of-the-are More- over, it can summarize the functions of the UI screen and engage with users through conversations.
understand text-dense Uls, and background knowledge about typical user interactions on devices, which makes MM1.5 a perfect candidate to be developed into highly capable UI understanding model.
Towards this goal, we developed MML.5-UT, an MM1.5 model variant further fine-tuned specifically on UI data that achieves competitive performance on UI understanding tasks and establishes new state-of- the---r-tt--tttt- performance in various benchmarks. Figure [OJillustrates single MM1.5-UI model's wide range of UI understanding capabilities on an iPhone screenshot. The model can find certain text ("BRIGHTNESS") on the left side (box2), correctly identify the settings icon at the top left (box 1), classify a UI element on the right as a checkbox (box0), and maintain a multi-turn conversation about the "Night Shift" function (box3) in the UI.
A.8.1 BENCHMARKS AND METRICS
We train and evaluate MMI.5-UI on variety of public and elementary UI understanding tasks used in Ferret-UI You et al. 2024). These tasks are established benchmarks in literature that cover multiple aspects of UI understanding, and allow us to fairly compare MM1.5-UI against prior work:
Public Benchmarks include screen2words (Wang et al. 202T]: : a screen-Level captioning task; widget captions (Li et al.) 2020);4 widget-level captioning task; and taperception (Schoop et al. 2022): predicting the tapability of a certain widget on the UI.
Ferret-UI elementary tasks are split into two categories: Grounding (Grd-#) are questions querying for certain area on the screen, such as finding an icon; and Referring (Ref-*) are questions given a certain area the screen, such recognizing text within screen area (i.e., OCR). Each of these tasks also has an iOS (+-i) and Android (*-A) version, forming four categories of tasks (e.g.. Grounding task Android is Grd-A).
33

---

Published as a conference paper at ICLR 2025
Table 14: Comparison with SOTA models on UI benchmarks. S2W: screen2words, WiC: widget
captioning, TaP: taperception. (f) denotes per-task fine-tuning.
ep. means epoch model training.
Public Benchmarks
Ferret-Ul Elementary Tasks
Model
S2W
WiC
TaP
Ref-i
Ref-A
Grd-i
Grd-A
Spotight Li Li 2023)
106.7
141.8
88.4
PaliGemma-33l 3B Bever et al. 2024
119.6
148.4
Ferret-UI-13B You et al. 2024
113.4
142.0
78.4
80.5
82.4
79.4
83.5
MMI.5-UI-IB
103.0
144.4
79.3
90.0
88.6
86.5
88.2
MM1.5-UI-3B
103.3
145.0
80.4
90.8
89.2
87.3
88.8
MM1.5-U1-7B
100.6
149.7
80.3
91.2
89.2
87.2
88.6
MM1.5-U1-30B
106.0
145.9
80.6
91.8
89.7
88.2
89.1
Ablation on MM1.5 SFT on UI tasks
MMI.5-UI-3B (1 ep.)
103.9 103.8
145.2
77.4
88.6
87.7
86.0
87.9
MMI.5-U1-3B (1 ep., MML.5 SFT)
139.5
75.3
88.2
87.4
85.5
87.1
More details of the benchmarks can be found in Appendix A.10 and the original Ferret- UI paper You ct al. 2024).
A.8.2 RESULTS
MM1.5-UI models trained by further fine-tuning the final MM1.5 models the Ferret-UI data mixture (You et al. 2024). which includes training data corresponding to the above elementary UI tasks and additional GPT-4-generated conversations about functionalities and descriptions about the UIs' functionality and layouts. There are 801K samples in total. All models are trained with the same batch size and learning rate as the original MM1.5 model.
Comparison with Prior Art. Results are summarized in Table 14 Our MM1.5-UI models outperform prior best models nearly all benchmarks except Screen2words. particular, even our IB model is able to outperform the Ferret- UI model in its proposed elementary tasks by wide margin despite being ten times smaller. The performance difference is most significant on iOS tasks at 9.1 points on average. This demonstrates that the abilities learned by MM1.5 are relevant and useful for UI tasks.
When comparing the performance across individual benchmarks, MM1.5 1.55UU demonstrates clear hierarchy of difficulties among tasks that focus on different types of UI elements, similar to Ferret- UI (You et al. 2024). Tasks focused on text are the most challenging, followed by those involving icons, while wid widget-based tasks are the easiest. This trend holds for both referring and grounding tasks. However, MM1 U-U shows a notable performance improvement in icon- based tasks, significantly narrowing the gap between icon and widget tasks. Ferret- UI highlighted the importance of resolution for tasks involving smaller elements like icons. The higher resolution and dynamic image splitting used in MM1.5-UT further confirm that resolution is particularly beneficial for enhancing performance in icon-related tasks.
Impact of MM1.5 SFT on UI tasks. To highlight the effectiveness of the MMI.5 SFT mixture on downstream UI tasks (i.e., in MM1.5-UT, we compare the performance of the full MMI.5-UI model with baseline UI model fine-tuned with UI data on the pre-training checkpoint that MM1.5 was trained on. Both models are trained for one epoch using the Ferret-UI dataset, and their results are presented in Table 14, The final MM1.5-UI model, which underwent SFT for general domain, text-rich, and refer&ground tasks, achieves superior UI performance within the same number of training steps. This demonstrates the strong transfer capability of MM1 ..5 for UI applications and prior SOTA models.
contributes to its performance improvement
Impact of model scaling, We observe overall performance improvements as models scale, though gains in all metrics remain modest, suggesting that larger models may be constrained by factors such as data diversity, image resolution, or overfitting. For instance, in the most challenging OCR tasks, 47.8% of incorrect responses contain the ground truth strict substring of the generated response, or vice versa. This suggests the model accurately recognized the text but failed to trim or include the correct amount. Additionally, the performance of the 7B and 30B models appears to have plateaued,
34

---

Published as a conference paper at ICLR 2025
indicating that larger, more diverse datasets and joint SFT of UI and core capabilities could further improve the performance.
A.9 MM1.5 IN-HOUSE MULTI-IMAGE SFT DATA
We generated two sets of in-house SFT data for the multi-image capabilities of our model: coco- instruct-interleaved and iel-instruct.
Coco-instruct-intelleaaee was generated using LLa Va-sttl approach using our text-only LLM. We input COCO image captions for two images, along with a prompt requiring the LLM to generate a conversation between a user and an assistant, with the two images inserted in context. This dataset helps our model learn conversational multi-image capabilities, An example from this dataset is shown below. This dataset contains -2k examples, with images per example.
User: <image 1> What do you see in the following picture?
Assistant: see an open beige toilet with a monitoring device attached the seat. The device appears to be displaying various metrics, such as temperature and pressure, and it likely used for medical purposes.
User: <image 2> What about this picture?
Assistant: see a row of parking meters sitting next to each other near a street. The meters are typical of those used to enforce parking regulations in many cities.
User: What is the difference between the two pictures?
Assistant: The first picture depicts a toilet with monitoring device, while the second picture shows a row of parking meters. These are two very different objects in different settings. The first picture is likely used for medical purposes, while the second picture is used to regulate parking in cities.
Icl-instruct was generated manually to help the model retain in-context learning abilities from pre- training along with more nuanced in-context learning tasks that require multi-step reasoning from the model. This dataset contains different varieties of in-context learning tasks, some of which are similar to those found in the VL-ICL benchmark (Zong et al., (024)). We found that including this dataset, which contains -500 examples, greatly boosts the models' in-context learning performance.
A. MM1.5-UI BENCIMARK DETAILS
The public benchmark tasks and metrics for evaluating MM1.5-UI are:
Screen2words is a captioning task where each complete screen is paired with 5 ground high-level summaries under ten words. The generated summaries' quality is measured by CIDEr score between the ground-trutt and generated summaries.
Widget Captioning a captioning task where certain sereen area that corresponds to a widget (e.g., button, list item) is paired with 3 ground-truth captions. The generated summaries' quality is measured by CIDEr score similar to Screen2words.
Taperception is binary classification task where certain screen that correspond widget (e.g., button, list item) is paired with a ground-truth binary label of whether the screen area is 'tappable' (i.e., clickable by users). The generated labels' quality is measured by F1 score.
The Ferret-Ul UI Elementary task benchmarks used to evaluate MM1.5-UI, organized by capability categories, are:
Ferret-UT Grounding (Grd-i/A) is of three grounding-basse based UI tasks introduced in Ferret- UI (You et al. 2004).. These tasks query for certain areas of screens that meet certain criteria. They include finding widget given text description, finding an icon given the class of the icon, and finding text location on screen. The expected response from the model is bounding box, and the quality of the bounding box is measured by Recall with IoU50.5.
Ferret-UI Referring (Ref-i/A) is a set of three referring-based UI tasks introduced in Ferret- UI (You et al.) 2024). These tasks query about knowledge or characteristics that correspond to certain of the screens. They include classifying the type of widgets in the given areas, recognizing the type of icons in the given areas, and recognizing texts in the given areas. The quality of the responses is measured by exact match accuracies.
35

---

Published as a conference paper at ICLR 2025
padding
resolution to downscaling
Figure 11: Illustration of image grid selection used in dynamic image splitting for high-resolution image encoding. (Left) grid can cover the full image without scaling down, we choose grid that minimizes padding. (Right) Otherwise, we choose the grid that minimizes the resolution loss due to scaling down.
Each of these of operating systems (iOS/Android), of which Android tasks are denoted as -A (e.g., Grd-A), and iPhone tasks as -i, which results 12 tasks in total.
Ferret-UI tasks further have two variants with screenshots from two types
A.11 DYNAMIC VS STATIC IMAGE SPLITTING
Detailed ablation study of MM1.5 with different image splitting strategies is shown in Table 15 16 17 and 18 All models are using the final setting except for the image splitting (dynamic vs. static).
Table 15: Comparison of our models when using dynamic vs. static image splitting. We follow our final settings for all models. (S) and (D) indicate static and dynamic splitting, respectively.
Knowledge Benchmarks
Model
A12D (test)
SQA (test)
MMMU (val)
MathV (testmini)
MME (P/C)
SEED
LLaVA
MM-Vet
RealWorldQA
18 Model Comparison
MMLL5-1B())
59.5
83.9
36.1
37.4
139334/244.9
69.99
87.9
67.9
34.2
51.8
MML.5-IB(D)
59.3
82.1
35.8
37.2
1365, 7/2457
70.2
88.1
71.6
37.4
53.3
MML.5-1B-Mote
66.4
86.8
41.2
42.2
1481 4/29.6
71.3
89.5
76.5
41.8
60.1
MMM.5--BB-MEEDD
67.1
87.6
41.2
42.9
1511.9/66.1
71.4
88.6
75.5
39.8
57.8
3/ Model Comparison
MML.5-3B(S)
66.1
87.2
36.8
43.1
8/297.
71.9
88.3
72.1
38.3
58.8
5-3D5-3((D
65.7
85.8
37.1
44.4
4/39.9
72.4
88.1
73.0
41.0
56.9
MMI.5.3B MoB(S)
41.9
43.1
1527. 8/342.5
72.4
88.4
78.5
41.4
69.9
89.8
42.9
46.9
159144/365.7
73.3
87.2
76.1
43.7
60.7
7B Model Comparison
XML.5-7BCS)
49.1
73.5
5MLD5-7B()
72.2
89.6
41.8
47.6
1514.9/346.4
73.4
88.6
74.2
42.2
62.5
308
Comparison
MML.5 30B((S)
75.4
92.8
46.8
56.0
1605 2/402
74.1
89.0
79.5
49.4
68.0
MM115-30BBD)
77.2
91.9
47.4
55.6
1645.22405.7
75.0
88.6
80.4
52.0
69.0
A.12 METHODOLOGY FOR RUNNING COMPETITOR MODELS
This section covers the methodology used report results for Phi-3-Vision (Abdin al. 2024b).
LLaVA-One Vision (Li et al. 2024c), Intern VL2 (Chen et al. 2024b) and MiniCPM-V2 (Yao et al. 2024b). When available, we reported the results published by the original authors, either in their technical reports or public leaderboards" When not available, we implemented inference runners using publicly released checkpoints. Commonly, we followed (Li al. 2024b) 's implementations that we adapted on our own internal fork of In-eval-harness (Gao et al. 2023 McKinzie et al. (024). To verify the validity of our inference implementations, we ensured we could reproduce previously published results within standard deviation. Below, share details for each model implementation:
co/ http://hggg.g.gooooo.o//c//////s
36

---

Published as a conference paper at ICLR 2025
Table 16: Comparison our models when using dynamic vs. static image splitting. We follow our final settings for all models. (S) and (D) indicate static and dynamic splitting, respectively.
Texi-rich Benchmarks
Model
WTQ (lest)
TabFact (tesi)
OCRBench
ChurtQA (test)
TexVVQ (val)
DocVQA (test)
Info (test)
(lest)
IB Model Comparison
MML.5-IB(S)
31.0
65.4
60.4
67.5
72.8
79.7
40.8
MMLL5-IB(D)
34.1
66.1
60.5
67.2
72.5
81.0
30.5
MMI.5-IB-MEEIG
34.1
69.6
58.0
72.7
75.8
82.5
46.0
MMI.5-IB-MoEON
38.9
71.4
62.6
73.7
76.1
84.8
55.9
Model Comparison
AMM15-3B(S)
36.3
71.0
61.1
74.3
75.2
45.8
MM1.5-3B(D)
41.8
72.9
65.7
74.2
76.5
87.7
58.5
32.5
70.1
60.2
73.0
75.9
81.0
44.2
MMM1.5-3BMMOEOD
39.1
73.1
63.8
73.6
76.8
85.0
53.6
7B Model Comparison
MMIL5-78(S)
38.4
73.7
$9.7
77.9
76.1
84,5
47.3
MM1.5-7B()
46.0
75.9
63.5
78.6
76.5
88.1
59.5
308 Model Comparison
MM115-30B(S)
46.0
81.0
64.5
82.4
78.7
88.5
53.2
MMI. (33.5))3
84.0
65.8
79.2
91.4
67.3
Table Comparison our models when using dynamic vs. static image splitting. We follow our final settings for all models. (S) and (D) indicate static and dynamic splitting, respectively.
Refer and Ground Benchmarks
Model
Flickr30k
RefCOCO avg.
LVIS avg.
Ferret Benchh avg.
(test)
1B Model Comparison
MM1.5-1B66
82.0
82.7
62.4
69.7
MM1.5-1B(D)
81.4
83.0
62.2
67.4
MM1 1B-MMBB)
79.3
80.9
63.9
73.4
MMI.5-1B-MoE(D)
84.8
85.4
64.6
69.6
3B Model Comparison
85.1
85.3
68.1
71.2
MML(5-3B(D)
85.6
85.9
69.5
MM1.5-3B-M -3--MSS)
82.8
82.6
67.5
70.8
MML.5-BB-MoE(D
86.2
85.8
66.9
72.2
7B Model Comparison
MMM)5-7B(S)
87.2 86.0
68.8
71.2
MML.5-7(()
86.6
85.3
66.4
72.6
30B Model Comparison
MMI (S)(SS)
90.1
87.7
75.6
MM1.5-30B(D)
90.1
87.5
73.1
77.1
Phi-3- Vision. We used the public Phi-3- Vision checkpoin and ran it on families of benchmarks. For general, text-rich, knowledge and refer&grount benchmarks, when the position of the image determined by the task, prepend the image the text input following the examples given in the Phi-3-Vision cook book For grounding benchmarks, we introduced the following prompt: "Question: {question| <b>Answer this question by listing the requested entities and their bounding boxes. The bounding boxes are formatted as follows: <x1.y1.x2.22, each value is between 0- (upper_bound) (onud)):nn2Anwwe.. For both referring and grounding, we experimented with variety of upper bound bounding boxes. Through our experiments, we noticed that an upper box of yielded to
htwp://wwmhooooooooooooooooooooo www.e/eeeeeeeeeeeeieni.nnc. hawpa//mene......oo.ooooeeoeoenn Infeee/ecceeeeedd.... comparsonn-ommuiiill-mnggge
37

---

Published as a conference paper at ICLR 2025
Table 18: Comparison our models when using dynamic vs. static image splitting. We follow our final settings for all models. (S) and (D) indicate static and dynamic splitting, respectively.
Multi-image Benchmarks
Model
QBench2 (val)
Mantis (test)
NLVR2 (val)
BLINK (val)
Muirbench (test)
MVBench
1B Model Comparison
MMI.5- I
65.8
48.4
78.6
46.1
41.9
34.0
MMI MMI.S-IB(D (-D()
66.4
50.7
79.0
45.8
46.3
34.7
MMI.5-IB-MOO
70.2
52.1
83.0
47.4
44.8
42.5
AMI.5-IB-MoENH
70.9
51.2
83.2
48.3
43.7
40.9
3B Model Comparison
MMI.5-3B(S
72.0
53.5
83.9
47.8
42.5
44.5
MMI.5-2 5333BD
73.2
54.8
83.8
47.7
46.8
44.3
MMI.5-3B-MoEN
70.4
54.4
85.3
47.2
47.1
44.2
MMI MMI.5-3B-MoE(D)
73.8
54.4
86.0
50.3
49.8
45.6
7B Model Comparison
MMI (MI.S-7B(S)
73.0
56.7
87.2
49.7
47.6
53.8
MMI.5-7B(D)
73.2
57.6
86.9
48.3
48.2
49.1
30B Model Comparison
MML.5-30BB
77.0
64.5
90.2
49.9
48.4
60.1
79.3
64.6
90.6
54.0
50.2
58.2
better results, in line with the answers produced by Phi-3-Vision. For Flickr30k, we slightly simplified the benchmark and asked the model to ground one entity per prompt, grounding multiple entities jointly did not lead to satisfactory results.
ALaVA-Onee Vision. We used the public checkpoint VLaVA-Oneiisioo 7B 10 and we followed closely the LLaVA documentation 1I| When not baked directly into the benchmarks, we used the original LlaVA prompts specified (Liu al., 2023a Li et al. 2024c) for all families of benchmarks. In particular, for grounding benchmarks, we used the prompt introduced in Table 18 of (Li et al. 2024c): "Provide the bounding box coordinate the region this sentence describes". On Flickr30k, we followed the single-entity approach outlined above.
The Intern VI.2 authors provided already a comprehensive set of benchmarks on general, text-rich, knowledge and refer&ground families which we reported first. For the few remaining benchmarks, we used the 2B public checkpoint released. Intern code base relies on both VLMEvalKit (Duan et al. 2024) and custom internal evaluation 13 We carefully reviewed the logic implemented" especially regarding the decoding parameters and prompts used. For grounding. used the prompt shared by the authors: "Please provide the bounding box coordinates of the region this sentence describes: (questiin </ref>".. On Flickr30k, we followed the single-entity approach outlined above.
InterVV..2.
MiniCPM-V2. We used the publicly released MiniCPM-V2 2.8B checkpoin Similarly to In- VL2, MiniCPM-V2 code base relies both VLMEvalKi and custom internal implementation which we reviewed carefully to reproduce decoding parameters and prompts. For offer&ground benchmarks, noticed that regardless of the prompt used, MiniCPM-V3 could not produce satisfac-
attorias..pynb
38

---

Published as a conference paper at ICLR 2025
tory results on RefCOCO and Flickr30 We decided not to include those results. For multi-image and in-context learning benchmarks, we found that MiniCPM- V2 does not accept multiple images imput in its Hugging Face implementation. We acknowledge MiniCPM-V2? V22'ss authors used custom image concatenating approach for B but we could not reproduce easily the same approach on other benchmarks. We decided only include the BLINK score reported by the authors
Language Response
Decoder Only LLM
Coprrimate tokems
Image Encoder
Language Instruction
Image patches
Dynamic Splitting
Dynamic Splitting
Single Image
Multi Image
Figure 12: The overview of model architecture. MM1.5 excels at (i) text-rich image understanding with dynamic image splitting, (ii) visual referring and grounding with coordinate tokens, and (tii) multi-image reasoning.
"essposess commonnly produced incomplete bounding boxes, preventing any parsing for fair evaluation.
7dff3bb74 waw.:/64444444f33ff3f33o33
eval mm http:///aaaiiiiiiiae//////////aa
39

---

Published as a conference paper at ICLR 2025
A.13
QUALITATIVE EXAMPLES
A.13.1 MM1.5 CAPABILITIES
Refer Ground
Whe ow I saain with these lugnddenz cIT D4,15 2000 CORN 235. A85
DiN
The rugradiete 1111 24,111 2316,135 doL 340 aido
maie
compranere
colo.
lad stufl sa olde
you shouid skfinndo auand daasko (14,111106 your cále. Thyy sbl and cale calazsing
I ando odoo, daroe the bod inguoaianta
louz.
haking soch
Gas kane are laii coedimis!
UPA
Sunvl The foor cits 131 agga co0s rags 200-3922213 2012180> Ingredieate woald reed colos
haklng
Multi Multi-Image
Math Reasoning
Hebbe
Try to mane about the speticns step by step.
To dndi the abie speed at tae bettors. er cun use the osservatice mange petactoks. ataar, sāke porintis. cacre
Melritty matn out
Image subject
poople at
eassgg the Jilloizz oqustion:
langee 2: The uasa vuljest the Sagaia Busilas laue Goluaottie dauds
Bandeas.
where E exrrg the mose of okkms g ocedenation anocty este', anaate
Imagge d: The masn vttject is rap of wottee vitu boart chaped taa dedge.
valoctey
Tiseriio
thene ercc
Ve are thve the setaal binght 100 mi and the thral bright: (0 vince groond. play
lanaee camaei and Linda,
the
lono
Inage Liturical andittcttnn,, stovcadeg gradinr
11/2 eigh
Dosiz both
lasqee eay inviau doop conslictica ac:
=gh
Chaft Dettter post tast incorpucatio mals anject and
Now, plua is the glvea vabaa dee and
"Böque eoege a the connect [Cooort Vecrej or nomert
Mustppy bith ddde
Actoctine] Eejoy oothe with woers
(Cltbe shop)*
tuke the upaao sides to the dead webeity XX Frtas
So, the adork queed the button apprrseasstty 28
Mull aniil I&M Precualiing
Text Rich
thom Beer
Resl oat far abstes oi Lab paptt.
Iepertacee valior wrthhwetes
cackes Though curbil compubbrade abaztine
the incas modie tae vidie laggage ouseemon, and vortose gre inttlle dica cheoo. oe Kerntyy sovers meziil
deiig inseee. snarple, dravierrste
apge vale malltand mettriinng udae
shot capptiiiiiiing
tavges
pelideed nattined pestaaiin mostls Punter, show that the taage eacode: tog vata iroge metation aod the haag tolo cocat has eiotoaitss impert, viile cotre-isggage conne dedas ouscirate aqilztos irpettiion saiing prosenedd radip, viziaato
seatimate nuupane
chat SCII contranttz suecko atiave excpetiion prrfomaaces sepertied vusine de ratus of extitibaaa uncittaecil benehmariio. Thando leaoeeile pn-tmisiin,, 363i sppos appe slon, pnopertios wads mskansed aucoenoo kinning. ana ansioingge peweing cnziiina ierolse daiz-d--llrall jecaspling.
Figure 13: Examples of MML.5 capabilities. The examples we demonstrated are generated by the MM1.5-3B model..
40

---

Published as a conference paper at ICLR 2025
A.13.2 TEXT-RICH IMAGE UNDERSTANDING
The New York Times.
Couli you summorize every article on this newepsper page?
ARMISTICE SIGNED, END OF THE WAR! BERLIN SEIZED BY REVOLUTIONISSS
NEW CHANCELLOR BEGS FOR ORDER: OUSTED KAISER FLEES TO HOLLAND
nesspaper page November 11, 1918. The main headline announces the signing of the Armisaiion marking the and of World Wa- revoution Germang, Berlin being sclaed by revolutioniss and the abdiestion of Kaiser Wilhelm II. The page also incluces call for German Emperor
dated
BO FUB
Chanedlor,, Friedrich Ehert. Additionally, reports on the otating of Kaiser Wilbelan and his flight to the Netherladds, The page featirres
well the resignation of
German Genoral report on the armistive being signed in France, zaarking the end of the
THE
Could you summarise the articles this newspaper page?
THE GUARDIAN
The newspaper page features several articles. The main headline reports the tragic event of 60 people dying in blets article discuses President Clinson consoling the families. There's
TRUTH On Saturday
eubessies. Anoller
80 orm
ADCORT 16
60 die in blasts at US embassies
piece about Ojukwa focecloeing on presidential ambition. Additionally, the page inclutes special advertisement for Sumsonite Carlton lugaye and higi-quality crystals.
Virims Lerilliss
SPECIAL
jukwu forecloses presidential
ambition
41

---

Published as a conference paper at ICLR 2025
Allied Health
What percent growth is expuected the primary technician john the U.S.?
32%
How much tup assistantt in the U.S.?
medical
$60,190
What the expected growtls plysical therapiet in the U.S.?
39%
Sabes
give table games nombers, only zames relessen befow 2010
Done
table pames relessed be before
numberee
Sares
Maaform
befel
Wil Spora
12530 506 WS
Wi
Hore 2009
Myy
Hoak pfo
May 2011
Ouloor
Part?
#TARE DVE Pulceas
Polwaay
Plac Star
42.200 506 Faas Man
MuSoleatom
May 1990
Uatryiaturs
Gream
Foosy
42

---

Published as a conference paper at ICLR 2025
13. VISUAL REFERRING AND GROUNDING
Watermelon
Show me the guidelines for selecting the Ripe and Swent mekon
wasermeion This indicates that of juice.
and Scels horvy
Larger "Weshing" : watermeion lurger weblhing, as likely sweeteer. to be enjoyed.
ripe watermelon will have a
ecterior, which mosess madd
Fiekd Spot: Look for a wat ermeron with a foldd spot, which is yellow or ceunge spot oo the bottom where
resting on the ground. This iadicattes that was allowed to zipen
the vine.
Orange Field Spot: The ideall watermolon will have
field which
the perfoct
will shing, avoid guidelines, you can a ripe watermelon every time.
ripe
Sollowingg
guidelines. coordiaates of the things that mention.
tipe milou? show mne
The ripe and sweet medon 4114, 4114,408.221 106 would be the one with dark and duil orangy field spot, larger websiing
SUPERMARKET DEPARTNENTS
place sboukl
lemons?
Fruits and Vegetables
cornect place?
you most The correct place to buy Vegetaables department as chown in the sxperraties imsage.
Olay, heve the place, Can help find the lecaons? Also show the coordinates of things that you mention.
Sure!
in imagee. nestled between the tomatoes and the peppers.
43

---

Published as a conference paper at ICLR 2025
A. 13.4 MULTI-IMAGE REASONING
DO NOT
ENTER
In which of these situations can drive on alead??
second image, you can drive because traffic light green.
What the bile exporieneing image 1? Which one best, image 2 or image 3, and why?
The bliz in image exporieneing rust and wear, as indieated by the rusedd metal parts and the hent bandlebars, Image with its focus the bicycle wheel, fits best it provides close-up view of the bike's wheel, which the part most directly related bike's condition.
have these ingredients in my fridge. What can make using all of these? Explain.
You can make banama bread using all of these ingredients. The ripe banauas from image mashed combined with flour from image the eggs image create moist and flavorful banana bread. The chocolate chips from image can be mixed into the batter for added texture and sweeluess.
44

---

Published as a conference paper at ICLR 2025
A.13.5 UI UNDERSTANDING
List all UI elements en the screen.
Projects
widgets present <16 803,8 "more" Ioon 2273, 38,303,79- Text dieplaring "No Projects" <106,250,200 2877 Text daspayyng Start New Project" <85,352 205,8800 "Magie Mavie, Chosse media from Photos and let iMovie, ereate msviee you. templates to mally ceate video or movic trailer." Button =00,487,3100 310,5000, "Movie, Make movie from seratch using media, from -0,505,3106677
screen
displaying "Projects"
20,418.81045822
"Storyhourd,
No Projects
Phosos
Magic Movie
Sturyboard
14:20
type -1257.118.2991100
Tips
microphoae
Search
Setup checklist
Exppens it?
sourch
Get Started
Is there another way to search?
Nerigatee your Phone
Practica kay gesitures
Yes, you search Seld 110,118,903, the IcnNn.
Next Steps
Per ice you Phooe
+ Fe prepaned
Discover More
Whhr www.miieatttiiii
45

---

Published as a conference paper at ICLR 2025
brief description the
Home
page disploging varions
Current Bestsellers
DISCE TGFT 10%
Where find the cureent best booksellers?
Liane Moriarty
VINCE FLYNN
SAPTURE
You can find the current best, booksellees by clicking the 'Curret
Bestsellless 13,0012,100>.
Where should
my account?
The Collection
You can sign in to your account by tapping on the "persoo" leon 2255,70,294, 111,
TEACHER
Mix LIÈS
FREIDE NcFADDER
What will happen I tap 118.03.118.6600-7
Tapping on the 'Librayy' tob 660 003, 11.11866000 6,000 will take you to your personul library where view all the books have auded..
46


Reviews:
[{"review_id": "2EdDTJQxUI", "reviewer": "Reviewer_Q6Sj", "summary": "This paper proposes a new set of multimodal LLM, MM1.5 as a result of a detailed data-centric pretraining experimentation. The experimented pretraining scheme includes 3 stages: (i) large-scale pretraining with text-only and image-text data, (ii) high-resolution continuation with OCR data and synthethic captions, (iii) supervised finetuning (instruction tuning). This work performs both model-centric and data-centric ablation analysis on the first two pretraining stages.", "questions": "- Have you performed ablation studies on skipping the second stage completely?\n- As far I have seen, Refer&Ground data requires 2x general purpose data. How could one make this more data-efficient?\n- I am having trouble to interpret Table 1. So, as long as a person trains their model using multiple images, they could achieve significant performance gain on text-rich and refer&ground tasks. The dynamic image splitting makes less difference according to this table.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Clarity: The paper is well written and easy to follow.\nSignificance: Multimodal model development is currently a trending topic, and this work employs a data-centric analysis that offers valuable insights for future researchers. Apart from the most commonly-used models, the examined model is able to comprehend about the referred objects and process multiple images.", "weaknesses": "A drawback of this study is that it serves as a comprehensive ablation analysis without offering any novel methodologies or tasks/benchmarks. I don't have any suggestion to improve this aspect at this point, but it should not be any problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8zbQIok4F1", "reviewer": "Reviewer_aLqd", "summary": "This paper introduces the MM1.5 model family and provides a comprehensive recipe for training multimodal large language models. It systematically explores the impact of data mixture ratio, training strategy, and model architecture, offering valuable findings for future research in multimodal large language models.", "questions": "My questions are listed in weaknesses and I look forward to the authors addressing these issues.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper conducts thorough experiments to explore the design of data, training, and architecture.\n2. This paper provides several empirical guidance for training MLLMs. The data mixture ratio during the pre-training stage (50:10:40 for image-text, interleaved image-text, and text-only data) could be particularly important, as it highlights the significance of text understanding in complex multimodal scenarios.\n3. This paper is clearly and concisely written, making each concept easy to understand.", "weaknesses": "This paper currently resembles a technical report more than academic research. It employs several well-known techniques (e.g. MoE, Dynamic Res) and mainly investigates the data mixture ratio at each training stage. While the empirical findings are valuable, additional theoretical insights would be beneficial. I have two concerns below: \n-  Why does the optimal ratio of text-only data change between the pre-training and SFT stages? Does the role of text-only data differ across these stages?\n- Given a collection of multimodal data, is there a theoretical guideline for determining the optimal mixture ratio between multimodal and text-only data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ml8ExmmU2m", "reviewer": "Reviewer_QN18", "summary": "This paper presents a set of MLLMs, MM1.5, scaling from 1B to 30B and also explore MoE variants (1B and 3B).\nMM1.5 is built upon MM1, upgraded with stronger capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning, through extensive empirical studies and ablations on data mixture in three stages of training and high-resolution image processing.\nMM1.5 shows strong performance and significant improvements over MM1 on various benchmarks.\nMM1.5 provides detailed insights into data mixture and other engineering choices in training a large-scale MLLM capable of general understanding, text-rich image understanding, visual reffering and grounding, and multi-image reasoning.", "questions": "1. In Figure 2, it seems that Refer&Ground data will downgrade the performance of all other categories. Although it will greatly improve the performance of Refer&Ground and Figure 3 shows that MMBase score will only slightly decrease, the other categories of MMBase score will decrease in fact. What is the reason for this phenomenon? Is it fair to use such an average score, MMBase, between different categories to evaluate the model's performance? Similar questions can be raised for Figure 5.\n2. With full respect this paper for its exploration and contribution to engineering decisions (very important for MLLM), but in the table of experimental results (Tables 4, 7-11), the authors seem to be selectively presenting some models that are weaker than their own. Take Table 7 as an example, authors selectively show the results of LLaVAOneVision-0.5B, InternVL2-2B and Cambrian-34B. But the selected three models have their corresponding model family, which can provide a more comprehensive comparison in different model scales. In addition, authors quote Qwen2-VL, but do not compare with it in these tables. The reviewer fully understands that most of the powerful models from the MLLM era are not comparable (training data, training details, evaluation details, etc.), but don't support such selective comparisons, which are not conducive to the reader's understanding of how the field is progressing and comparing.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper conducts extensive empirical studies and ablations on continual pre-training, dynamic high-resolution image processing, and curation of our supervised fine-tuning datasets, which can provide insights and experience for future research on large-scale MLLMs.\n2. This paper presents a set of MLLMs, not only scaling from 1B to 30B but also exploring MoE variants (1B and 3B), which can provide important scaling insights for future research on large-scale MLLMs.\n3. This paper shows strong performance and significant improvements over MM1 on various benchmarks.\n4. This paper also explores two SFT variants of MM1.5, i.e., MM1.5-Video and MM1.5-UI, which shows strong performance on video and UI benchmarks.", "weaknesses": "1. Although this paper presents detailed empirical ablations on several important engineering decisions in data mixture and high-resolution image processing, it does not provide strong new insights or findings compared to previous works. The ablations on SFT data mixture, continual pre-training, and dynamic high-resolution image processing are not novel and have been explored in previous works.\n2. Specifically, data-specific hyperparameters from detailed ablations for data mixture (pre-training, continual pre-training or SFT) may not be quite practical for other researchers to follow, as they are highly dependent on the specific dataset, model size, model architecture, and training setup.\n3. Lack of in-depth analysis on the underlying reasons for the phenomenon observed in the extensive ablations. Simple assumptions or hypotheses about ablation results are not sufficient. The ablation decisions are pushed by empirical results step by step, but the underlying reasons for the phenomenon observed in the ablations are not well analyzed.\n4. This paper does not claim any form of open-sourcing, no matter model weights, code or data, which may limit the reproducibility and practicality of this paper, especially considering the first two weaknesses mentioned above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tsClxD1090", "reviewer": "Reviewer_CjsW", "summary": "The paper introduces a new family of multimodal large language models (MLLMs), ranging from 1 to 30B parameters, including MoE models. They build on top of MM1 and focus on data curation for **continual pretraining** and **supervised fine-tuning** (SFT), providing several ablations for the choice of data mixture(s) and summarize their findings in a recipe.\n\nTo run their ablations, they use the following setup:\n\n- Static image splitting, splitting large images into four sub-images, plus an overview image, for both SFT and continual pretraining.\n- They use either single image or multi image data, and enable splitting only when processing less than three images for a single sample.\n- They use a custom (non-disclosed) CLIP text encoder and LLM as their backbones.\n\nFor evaluation, they consider general, text-rich, refer & ground, knowledge, and multi-image tasks and report two metrics:\n\n- Category average score: average score for all benchmarks in a given category.\n- MMBase score: average score on general, text-rich, and knowledge categories.\n\n## Supervised fine-tuning\n\nSFT is conducted using a large amount of datasets, categorized as follows:\n\n- Single image\n\t- General\n\t- Text rich\n\t- Refer & Ground\n\t- Science\n\t- Math\n\t- Code\n- Multi-image\n- Text only\n\nThey first focus on the impact of single data categories, finding that text-rich data is crucial for text-rich tasks and that good refer & ground capabilities only emerge if the model is trained on appropriate data.\n\nThey then focus on optimising for the sampling ratio between the general category and other categories (for single image data), and find that the right mixture of science and math data can significantly improve performance. They also find that doubling the sampling probability of refer & ground data provides a good tradeoff between general performance and refer & ground performance.\n\nOnce the single-image data ratios are fixed, they integrate text-only and multi-image data. They find that text-only data has a minor impact and fix its ratio to 0.1. They pick a similar ratio for multi-image data, as it improves the multi-image score significantly while also slightly improving the MMBase score. A larger quantity of multi-image samples decreases the MMBase score. Finally, they fix the single-image ratio to 0.8.\n\nThey combine these results to produce three data mixtures:\n\n- Base: general, text-rich, science, math, code.\n- Single image: base + refer & ground.\n- All: single image + multi-image and text only.\n\nAs expected from the previous findings, the single-image mixture unlocks the refer & ground capabilities, while the multi-image one largely improves the multi-image capabilities. The **all** mixture is the best on average (although the base mixture performs best on tasks that are not refer & ground and multi-image).\n\n## Continual pretraining\n\nThe paper explores continual pretraining using ~50M OCR high-resolution (1344 x 1344) sampled across four datasets. Once continual pretraining is complete, they perform SFT using the Base mixture. Their findings are:\n\nOCR data must be high-res. Downsampling images to the encoder input dimension (378 x 378) can cause the model to lose up to 2% in average score and even perform worse than when no continual pretraining is performed! OCR data improves performance, but only if it has a high enough resolution.\n\n- They also experiment with data with synthetic captions and find that they improve performance compared to a baseline without continual pretraining. Still, they perform worse than only using OCR data, even when combined.\n- In the appendix, they show that self-training with synthetic captions can improve performance compared to an OCR-only baseline.\n\n## Dynamic Image Splitting\n\nThese experiments use the single image mixture and start from an MM1 checkpoint without continual pretraining.\n\nThe authors claim that static image splitting has inherent issues, as low-res images are also split, and splitting non-square images can result in sub-images that consist only of padding. Thus, they experiment with dynamic image splitting, picking a grid that minimizes the amount of padding. In addition to the sub-images, they include the resized original image for additional context and optionally positional indicators for sub-images.\n\nBased on their results, they claim that:\n\nDynamic image splitting outperforms static splitting for text-rich tasks. Image resolution, the number of sub-images, and the number of image tokens are important. Moreover, it greatly improves performance on tasks whose images have unusual aspect ratios.\n\n- These benefits come at a relatively small cost (< +10%) in terms of encoded images\n- Positional indicators are useful for specific tasks but not necessary in general\n- Placing the overview image after the sub-images improves performance\n\n## Final Results\n\nGiven the findings illustrated in previous sections, the authors propose a three-step recipe (MM1 pretraining, continual pretraining, and SFT) to train 1, 3, 7, and 30B models, plus 1 and 3B MoE models.\n\nTheir models improve over similarly-sized models, sometimes by large margins, and the performance scales well as models grow in parameters. MoE models often outperform their dense counterparts. This is particularly evident for 1B models, where the MoE variant often improves by over 5% compared to its dense counterpart. Their models also provide significant improvements in refer & ground and in-context learning tasks.", "questions": "- If I understand correctly, in section 2.2.2, using a data ratio of 1:2, e.g., for general and science categories results in a sample from the general category (68x the size of science) being 34x more likely to be picked compared to a science sample when building a batch, correct?\n- Is dynamic image splitting used in both continual pretraining and supervised fine-tuning?\n- Why were the 7B models not compared with similarly sized open-source models such as LLava 1.5 7B?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and provides novel, valuable, and actionable findings for the MLLM community. While the backbones to replicate the experiments are not publicly available, the authors release as many details as possible regarding their datasets, data mixtures, hyper-parameters, and data processing strategies (e.g., dynamic image splitting, positional indicators, and overview image position).\n\nMoreover, the models trained using the data curation strategies illustrated in the paper significantly outperform a model with the same architecture but a simpler data curation method (MM1) and several current state of the art models, especially in refer & ground, multi image and in-context learning tasks, across various sizes.", "weaknesses": "While there are clear benefits in using dynamic image splitting for processing input images, I am not fully convinced it leads to better performance overall, as when comparing similar setups in terms of images and tokens in Table 1 static image splitting works best. I am wondering if it would outperform dynamic image splitting when $n = 10$ and 1440 total tokens are processed.\n\nI am also unsure why the final recipe used indices to indicate the positions for sub-images, even though Table 3 shows that separators work better overall. Could you please clarify this choice?\n\nPresentation-wise, I only have a couple of minor remarks:\n\n1. In section 2.2.1 (in the second paragraph), the authors claim that \"adding the code category results in a slight increase in the text-rich score\" (i.e., ~1.54%), and that \"adding text-rich data can significantly improve the performance on [...] and knowledge benchmarks\" (i.e., ~1.78%). I would maybe slightly rephrase either of those sentences.\n2. Bolding the best results in tables would make it easier to identify the best configuration(s).\n3. Maybe introducing the continual pretraining ablations first and following up with supervised fine-tuning would improve readability. The paper would then follow the recipe logically.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)