Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resources
Authors: Weizhi Wang
Yu Tian
Linjie Yang
Heng Wang
Xifeng Yan
Venue: COLM 2025 (2025)
Keywords: multimodal large language model
efficient pre-training
high quality image-text filtering
TL;DR: Open-Qwen2VL is a fully open-source multimodal LLM which is trained on only 5B high-quality multimodal tokens but outperforms Qwen2-VL, trained on 1.4T tokens, on various benchmarks.

Abstract: The reproduction of state-of-the-art multimodal LLM pre-training faces barriers at every stage of the pipeline, including high-quality data filtering, multimodal data mixture strategies, sequence packing techniques, and training frameworks. We introduce Open-Qwen2VL, a fully open-source 2B-parameter Multimodal Large Language Model pre-trained efficiently on 29M image-text pairs using only 220 A100-40G GPU hours. Our approach employs low-to-high dynamic image resolution and multimodal sequence packing to significantly enhance pre-training efficiency. The training dataset was carefully curated using both MLLM-based filtering techniques (e.g., MLM-Filter) and conventional CLIP-based filtering methods, substantially improving data quality and training efficiency.

The Open-Qwen2VL pre-training is conducted on academic level 8xA100-40G GPUs at UCSB on 5B packed multimodal tokens, which is 0.36\% of 1.4T multimodal pre-training tokens of Qwen2-VL. The final instruction-tuned Open-Qwen2VL outperforms partially-open state-of-the-art MLLM Qwen2-VL-2B on various multimodal benchmarks of MMBench, SEEDBench, MMstar, and MathVista, indicating the remarkable training efficiency of Open-Qwen2VL.

We open-source all aspects of our work, including compute-efficient and data-efficient training details, data filtering methods, sequence packing scripts, pre-training data in WebDataset format, FSDP-based training codebase, and both base and instruction-tuned model checkpoints. We redefine "fully open" for multimodal LLMs as the complete release of: 1) the training codebase, 2) detailed data filtering techniques, and 3) all pre-training and supervised fine-tuning data used to develop the model.

Published as a conference paper at COLM 2025
Open-Qwen2VL: Compute-Efficient Pre-Training of Fully- Open Multimodal LLMs on Academic Resources
Weizhi Wang', Yu Tian', Linjie Yang", Heng Wang', Xifeng Yan UC Santa Barbara, 2Sed Vision Team, ByteDance, NVidia Research weizhiwang@ucsb.cc eddu
Abstract
The reproduction of state-of-the of-t-e-rtttt-tt multimodal LLM pre-training faces barriers at every stage of the pipeline, including high-quality data filtering, multimodal data mixture strategies, sequence packing techniques, and training frameworks. We introduce Open-Qwen2VL, a fully open-source 2B-parameter Multimodal Large Language Model pre-trained efficiently on 29M image-text pairs using only 220 A100-40G GPU hours. Our approach employs low-to-higg dynamic image resolution and multimodal sequence packing to significantly enhance pre-training efficiency. The training dataset was carefully curated using both MLLL-bassed filtering techniques (e.g., MLM-Filler) and conventional Ca.e- filtering methods, substantially improving data quality and training efficiency.
The Open-Qreen2 VL pre-training is conducted on academic level 8xA100- 40G GPUs UCSB on 5B packed multimodal tokens, which is 0.36% of 1.4T multimodal pre-training tokens of Qwen2-VL. The final instruction- tuned Vwen-Q outperforms partially-open state-of-the MILM Qwen2-VL-2B on various multimodal benchmarks of MMBench, SEED- Bench, MMstar, and Math Vista, indicating the remarkable training effi- ciency of Open-Qree22L..
We open-source all aspects of our work, including compute-efficient and data-efficient training details, data filtering methods, sequence packing scripts, pre-training data in WebDataset format, FSDP-based training code- base, and both base and instruction-tuned model checkpoints. We redefine "fully open" for multimodal LLMs the complete release of: 1) the training codebase, 2) detailed data filtering techniques, and 3) all pre-training and supervised fine-tuning data used to develop the model.
Website
https: httpe://bbcooo...tthuio/////eppp
Code
Models Data
1 Introduction
The Multimodal Large Language Models (MLLMs) (Anthropic, 2025; OpenAI, 2023; Chen et al., 2024b; Wang et al., 2024a; Wu et al., 2024) present strong emergent capabilities on multimodal understanding and visual reasoning, eliciting the Artificial Intelligence Ap- plications to comprehend and analyze images, charts, and PDF documents. Different from conventional Vision-Language Models (VLMs) (Radford et al., 2021; Jia et al., 2021), trained on image-text caption data from scratch with small model size, the MLLMs are typically constructed on a well-trained text-only LLM and then continually pre-trained on diverse large-scale multimodal data. However, recent state-of-the-arr MLLMs are neither "fully-open" to the community for reproduction nor compute-friendy) to academic insti- tutions with limited GPUs. In Table 1, we compare the openness of recent SOTA MLLMs of VILA (Lin et al., 2024), MM1 (Zhang et al., 2024), Ideflics (Laurençon et al., 2024), BLIP- 3 (Xue et al., 2024), Hama-3.2-Vissee (Dubey et al., 2024), Phi-3.5-Vision (Abdin et al., 2024),

---

Published as a conference paper at COLM 2025
Filtering
Sequence
Pre-Training
Pre-Training Codebase
Model
Instruct Model
Models
Techniques
Packing Scripts
Data
Checkpoint
Data
Checkpoint
VILA
None
None
Open
Open
Open
Open
Open
MMI
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Open
Open
Open
Open
Open
Open
Open
BLIP-3
Closed
Closed
Open
Closed
Open
Closed
Open
Llama-3.2-Viiie
Closed
Closed
Closed
Closed
Open
Closed
Open
Phi-3.5-Vision
Closed
Closed
Closed
Closed
Closed
Closed
Open
QwenZVL
Closed
Closed
Closed
Closed
Open
Closed
Open
Open QwenVV
Open
Open
Open
Open
Open
Open
Open
Table 1: Comparisons of openness between several state-of-the-art MLLMs. MLLMM
and Qwen2VL. (Wang et al., 2024a). Even if most of the SOTA MILMs release their base or instruction-tuned model checkpoints, their killer secrets of data filtering techniques, se- quence packing scripts, pre-training data, training codebase, etc are completely close-source, in which they even hide such technical details in their technical reports.
In this work, we introduce Open-Qwen22L, a 2B-parameter MLLM which outperforms close-source Qwen2-VL-2B on various multimodal benchmarks and achieves outstanding compute efficiency.
is pre-trained on approximately 5B well-curated high- quality caption data tokens, which is 0.36% of 1.4T multimodal tokens of Qwen2-VI. (Wang et al., 2024a) pre-training. Such remarkable data-efficiency enables us to perform the pre- training on academic-level computing resources of 8* A100-40G GPUs. In addition, we conduct compressive visual projector (Yao al., 2024) to scale-down 729 image patches to 144 visual tokens and perform multimodal sequence packing to further enhance the pre-training efficiency.
We perform comprehensive ablation studies on pre-training data mixture strategies and data filtering models. The best pre-training data consists of CC3M-CC12M-SRBD (Ordonez et al., 2011; Sharma et al., 2018a) caption dataset curated by CLIP and DataComp-Medium caption dataset curated by both the DFN-CLIP and MLM-Filter. Adopting efficient MLLM as the data filtering model significantly enhances the model capabilities on various benchmarks. Additionally, we scale up the visual supervised fine-tuning (SFT) data to 10M level (Guo et al., 2024) further enhance the model capabilities.
We open-source everything to the community to help easy and convenient reproductions to our model Qwen- including the data filtering details, sequence packing scripts, pre-training data in webdataset format, training codebase based on FSDP, and both base model and instruction-tuned model checkpoints. Meanwhile, our open-source codebase is the first comprehensive solution that supports all stages of Multimodal Large Language Model training, including large-scale caption data preparation, quality score generation, data filtering, multimodal sequence packing, pre-training, supervised fine-tuning, and evaluations on multinodal benchmarks. We redefine "fully open" for multimodal LLMs as the complete release of: 1) the training codebase, 2) detailed data filtering techniques, and 3) all pre-training and supervised fine-tuning data used to develop the model. We wish to demonstrate that the research pre-training is not only a game for giant tech companies and encourage the academic community to work on pre-training data and pipeline research even with very limited computing resources.
Compute-Efficient Multimodal Pre-Training
2.1 Dataset Choices and High-Quality Data Filtering
The current advanced multimodal LLMs are continually pre-trained a large-scale image- text caption dataset. In addition to image-text caption dataset, some of latest MLLMs like VILA(Lin et al., 2024), MM1 et al., 2024), DeepSeek-VL2 (Wu et al., 2024) also mix the image-text interleaved data with caption data for multimodal pre-training. Mixing image-text caption data and interleaved data will enhance the multimodal in-context
2

---

Published as a conference paper at COLM 2025
Filtering Model
#Image-Text
ID
Dataset
Resources
Pairs
1
CCS
CLIP
8.5M
https://githur:
2
DataComp
DEN
15M
hugginghoeeet///aaalllllgggtttt
3
LAION
CLIP
15M
4
DataComp
MLMFilter DEN
19.9M
Table 2: Image-Text Caption Datasets for Open-Qwen2VL pre-triinng
learning and multi-image reasoning capabilities of MLLMs. However, MM1 (McKinzie et al., 2024) demonstrates introducing image-text interleaved documents into pre-training data will reduce the zero-shot single-image reasoning and understanding capabilities of base MLLMs. Thus, to control the scale of pre-training data and ensure the pre-training efficiency, Open-Quenn2V. focuses the pre-training paradigm image-text caption data only.
To motivate the easy reproduction to work from the community, choose 4 most popular image-text caption datasets shown in Table 2, which are widely used in open-source vision-language model pre-training. BLIP-1 (Li et al., 2022) releases the high-quality caption data curated from combination of CC3M-CC12M-SBU (CCS) using CLiP-based (Radford et al., 2021; Hessel et al., 2021) filtering. LAION-400M (Schuhmann al., 2021) implements strict 0.3 threshold based on CLIP image-text cosine similarity to curate its high-quality dataset of 400M image-text caption pairs. We download the CCS and LAION caption datasets based on the release image-urls using img2dataset (Beaumont, 2021) tool. We only download 15M LAION data to perform controlled-siz data mixture ablation studies in Section 2.5.
Secondly, we choose DataComp-Medium: data choice. Based the leaderboard of DataComp medium filtering track performance, Data-Filtering-Networt (DFN) (Fang et al., 2023) is the top-1 independent data filter on the leaderboard We successfully download 99.8M out of 128M original released 128M Meddeum data. Then we adopt the official resharder script² to select the DFN- curated high-quality subset based on the released top-15% data uids from DFN It is worth noting that the DFN only releases the top-15% curated data rather than the DFN model checkpoint. Thus, it is impossible to change the retained data fraction based on the quality scores generated by DFN-model. For DataComp-DFN high-quality dataset, finally we get 15M image-text caption data.
(Gadre al., 2023) as another pre-training
The MLLM-based data filtering method emerges since the introduction of MLM-Filter (Wang et al., 2024c), in which these methods adopt efficient MLLM as the high-quality data filter instead of CLIP model. MLM-Filter provides four distinct image-text data quality metric for high-quality data filtering, including image-text matching (ITM), object detail fulfill- ment (ODF), caption text quality (CTQ), and semantic understanding (SU). Based on the conclusions from ATIQE (Huang et al., 2024), the Semantic Understanding (SU) quality metric yields the best performance for MI.LMs trained on the high-quality data curated from such metric. Thus, we generate the SU quality scores for dataComp-Medium data using 5-1. 5b-gpt data filtering model and set filtering score threshold as 85 out of 100. With such threshold, We get 8M MLM-Filter curated data and union them with 1FN-15M. After deduplication, we get 19.9M high-quality data.
1 httt::://www..dttacooo http://ww.bbaaammm....a/.cc//aaa html
>https://hugginggaace http:://wggggg....aa../////a//aa Redium_dfn__2m_indds "https://hugging httpe//nnnnan.eeeeeieeieiiiinnnn
3

---

Published as a conference paper at COLM 2025
2.2 Model Architecture with Low-to-High Image Resolution
We adopt simple architecture with Qwen2.5-1..B-Instrree LLM Backbone (Team, 2024), Adaptive Average-Pooling Visual Projector (Yao et al., 2024), and SigLIP-SO-400M Vision Encoder (Zhai et al., 2023). Specifically, the Adaptive Average-Pooling Visual Projector contains an Adaptive Average-Pooling layer followed by a two-layer ML.. With the Adap- tive Average-Pooling layer, we scale the 729 output visual patches from SigLIP to any resolution. We adopt 144 visual tokens for representing an image in the pre-training stage and scale up the resolution to vanilla 729 visual tokens in SFT stage. Such low-to-high image resolution significantly enhances the MLLM pre-training efficiency and does not hurt the high-resolution image understanding of the final MLLM after SFT stage.
Open-Qwen2VL does not adopt advanced designs of 2d-Multimodal RoPE (Wang et al., 2024a) and naive dynamic resolution (Wang al., 2024a) to save computes and ensure the training efficiency. Moreover, for academic computing resources, downloading images and saving in original resolution require huge disk space, which unavailable in most of academic institutions. During our data downloading process with img2dataset (Beaumont, 2021), we resize the smaller side of the image to 512 pixels and keep the aspect ratio, which makes us not able to adopt naive dynamic resolution in the pre-training stage.
For both the pre-training and SFT stages, we freeze the parameters of vision encoder and make the parameters of projector and LLM backbone trainable to save more computes. However, recent studies (Wang et al., 2024a; Zhang et al., 2024) demonstrate that making the vision encoder trainable can further enhance the visual understanding capabilities of the MLLMs. We leave as an ablation study for investigations.
CE Loss
CE Loss
Large Lenguage Model
Large Language Model
. dz
M dr
AugPooling Projector
Word Embeddings
MLP Projector
Word Embaddings
di
x di
SIgLIP
Caption
SigLIP Image Encoder
Caption
Image Encader
Pre- Tre-Training Stage
SFT Stage
Figure Model Architecture of Qpen-Quen Here M = N 144 the number of image patch tokens and number of projected visual tokens during the pre-training stage, respectively.
2.3 Multimodal Sequence Packing
Because the large-scale image-text data are varied in its length, simply batchfying a set of examples based on similar length and padding them to the longest sequence will lead to large portion of padding tokens in each training batch. Such a large amount of padding tokens will result in heavy compute-waste and training inefficiency. Thus, we introduce multimodal sequence packing to regroup the image-text caption data into sequence groups closest to 4096 context length.
The algorithm for the multimodal sequence packing is presented in Algorithm 1. Since we download and pack all image-text caption data into webdataset format and each webdataset tar file contains exactly 10k image-text caption data, the proposed multimodal sequence packing intends to regroup such 10k pairs into a set of multimodal sequences with 4096 context length.
4

---

Published as a conference paper at COLM 2025
Algorithm 1 Multimodal Sequence Packing
Require:
1: D: Set of image-text caption data
Maximum context length
3: Padding token
Ensure: Packed sequences within context length L
4: function P'ACKSEQUENCIS(D,L])
I+ Ø
b Initialize items dictionary
6:
for all d E D do
7:
(Ta, Va) 4- ProcessCaption(d)
b. Get text tokens and visual tokens
8:
len, [Tal + [Va
VV 144
9:
[[d] + Iena
10:
end for
11:
{ SortByLenghh()
Sort in descending order Initialize bins
12:
B Ø
13:
for all len,) E do
14:
placed false for all bin E do
15:
16:
if I Eaii + + Iena S then
17:
(Id,
18:
placed <- true break end if end for
19:
20:
21:
22:
if ppaaced then
23:
(dd,
24:
end if end for
25:
26:
P P+C
P Initialize packed sequences
27:
for all bin E B do
28:
Thin Ø Voin f Ø
Concatenated text tokens Concatenated PIL images
29:
30:
for bin
31:
apin- (T)
32:
appen Va)
33:
end for if Tain < then
34:
35:
Tain-pad L)
Pad to context length
36:
end if P.append ((Tbir Vbiu)) end for
37:
38:
return
39:
end function
The multimodal sequence packing involves three major steps: computing the multimodal length of each image-text caption sample, regroup data into several bins in which the total length of each bin closest to 4096, and concatenate the input_ide ids vectors and pillow- format images. We adopt First-fit-decreasing (FFD) bin packing algorithm (Johnson, 1973) to pack each image-text caption data into several bins. We also follows LLaVA to insert an <image> placeholder token at the beginning of each image-text caption. We use the default end-of-sequence token, <I immmmne end > token as the separator between each image caption text.
We store each packed multimodal sequence to pickle file, because pickle support storing data in different formats like pillow-image and torch input_io ids tensor one file. Finally, each pickle file contains the following dictionary for each packed multimodal sequence:
"images": list of pillow image objects;
"input_ids jpput torch Long Tensor with image placeholder token;
"lengths"; a list of integers to record the multimodal length of each image-text caption data.

---

Published as a conference paper at COLM 2025
Data Mixture
Benchmark
1 + (23.5M)
1
1+2+3
1
(23.5
(38.5M)
(28.4M)
General Benchmark
MMMU
38,9
37.2
36.7
38.0
MMBenchaey
75.6
75.9
75.9
77.3
SEEDBench Imgdev
68,9
69.6
68.9
68.7
MMStar
39.6
41.7
41.7
41.3
OCR VQA
A12Dast
56.3
55.5
57.3
56.8
TextVQAval
55.1
57.4
57.4
57.0
Math Reasoning
Math Vistagsttn
28.6
28.1
28.7
28.6
Hallucination
POPE
79.2
80.1
77.7
80.1
Average
55.3
55.4
55.5
56.0
Table 3: Benchmark performance of MLLMs pre-trained on different pre-train data mixture and fine-tuned controlled LLaVA-665k instructions. Remarks for each dataset: 1: CCS- CLIP; 2: DataComp-DFN; 3: LAION-CLIP; 4: DataComp-MLM-Filter & DFN.
2.4 Training Infrastructure and Codebase
We develop our training codebase based on Prismatic-VLM, (Karamcheti et al., 2024). The original codebase only supports SFT on single-image instructions and we heavily modify its dataloader and batch preparation to support the multimodal packed sequences with multiple-images in one sequence. We retain its Fully-Sharded Distributed Parallel (torch- FSDP) trainer, which we find that it significantly accelerates the training compared with LLaVA codebase using DeepSpeed--ee Zero3.. Although FSDP and DeepSpeed-Zero3 utilize the same model sharding algorithm, our FSDP-based implementation achieves approximately 17% faster for each training step than the DeepSpeed implementation, consistent with findings reported by Karamcheti et al. (2024).
2.5 Ablations on the Data Mixture
After preparing and sequentially-packingg the 4 image-text caption dataset, we conduct ablation studies investigate the effects of different data mixtures to the performance of final MLLMs. Since there is 16 combinations between the four datasets, we only consider 4 combinations. The CCS-CLIP data is fixed and we incrementally add other three datasets with it. For each dataset group, pre-train the MLLM for epoch on packed multimodal sequences and then fine-tune the base MLLM on LLaVA-665k instruction dataset (Liu et al., 2023). The training details and hyperparameters are available in Appendix Table 8. Then we evaluate each model ablations on multimodal benchmarks of AI2D-test (Kembhavi et al., 2016), TextVQA-val (Singh al., 2019), POPE (Li al., 2023b), MMMU-val (Yue et al., 2024), MMBenchh-v..04 00-dev (Liu et al., 2024), SEEDBench-inge-dev (Li et al., 2023a), MMStar (Chen et al., 2024a), and VithWisst-test-mii (Lu et al., 2023).
Results. The benchmark results of each pre-trained and fine-tuned MLLM using different data mixtures are presented in Table 3. Since the DataComp-DFN and LAION are both web-crawled data and adopt similar CLIP-based data filtering techniques, the data mixtures of these two dataset with CCS achieves very similar model performance. Moreover, simply mixing three CCS-CLIP, DataComp-DFN, and LAION-CLIP does not achieve better per- formance, which might be caused by the high data homogeneity between DataComp-DEN data and LATON- CLIP data. Surprisingly, adding very small amount of high-quality data

---

Published as a conference paper at COLM 2025
22
E
2
8
X
3
Lorth Pil
Tramin 10.-
1:0 Pastraic
19p02ar
24
30
02
PAREETHAAON
Taching Dots
Tohtin 2m
Thinig
Thiiing
The scaling effects of visual SFT data from LLaVA-665k to MAmmoTH-VIL-s--00 We evaluate the checkpoints every 2M training samples.
Figure 2:
(5M) curated by a different efficient-MLLM based data filter, MLM-Filter can significantly enhance the model performance, achieving +0.5 average performance improvements. We suppose that MLLM-based data filter may introduce a different data distribution into the pre-training set, which brings new knowledge for enhancing the MLLM capabilities.
Finally, the pre-training of GPU hours, and the SFT LLaVA-665k instructions takes 48 A100-40G GPU hours.
on the best data mixture takes about 220 A100-40G
Scaling-Up Supervised Fine-Tuning
3.1 SFT Dataset
After the ablation studies on the pre-training mixture, we further scale-up the visual SFT data from LLaVA-665k (Liu et al., 2023) to MAmmoTH--LL-10M (Guo et al., 2024) further enhance the understanding and reasoning capabilities of MLLM. We only use the 10M single-image subset for visual SFT and do not include the additional LLaVA-OneVision for further SFT mixed image and video data. The MAmmoth- Mintmot---LL-1O. data requires 200GB CPU memory if the original LLaVA-style dataloader is adopted to load the full 10M json file data into memory in distributed multi-process. To comply with the limited CPU memory of our server, we store each data sample in the 10M full json data into single json file, and meanwhile generate a 10M-indices file for loading into the memory. Each index contains the path to the data sample json, the boolean value of text-only or image-text data, and the pre-computed image-text data length for batchfying. The SFT hyperparameters also follow Appendix Table 8.
3.2 Scaling Effects and Results
We save the checkpoints for every 2M sft instructions, which 15625 steps under the batch size of 128. We illustrate the benchmark performance of each saved checkpoint Figure We can conclude that scaling-up SFT remarkably improves model performance on various multimodal benchmarks. Most of the benchmarks like POPE, MMMU, MMBench, and SEEDBench performance converges at the SFT scale of 8M instructions and do not improve for the final 2M data. The curves TextVQA and Math Vista performance vary from others, which present a steady improvement over the data scale. It might be caused by the lack of such pre-training math or OCR data in our curated pre-training caption dataset, making the visual math reasoning and text-based VQA become aut-of-distribution tasks. For the general- -pupos knowledge-based benchmarks of MMMU, SEEDBench, and MMStar, we even observe the slight performance degradation in the final 6M instruction tuning data.
7

---

Published as a conference paper at COLM 2025
Models
intemV122.528-MMPP
De-2SeekVL-22-inn Qwen2-V1-20B-ns Open-Qruen2VL
# Pretrain Tokens
277B
8.1T
1.4T
5B
General Benchmark
MMMU
41.2
39.6
41.1
39.8
MMBench
72.5
68.3
68.8
80.9
SEEDBenchdev
73.2
72.5
72.0
72.5
MMStar
54.3
49.9
46.3
49.7
OCR VQA
A12Drest
75.3
74.6
72.3
66.3
TextVQAval
77.2
80.5
78.8
63.3
Math Reasoning
Math yestannt
55.3
54.5
48.0
Hallucination
POPE
89.8
88.8
87.6
84.4
Table 4: Benchmarks performance of Open-Qwen22V and other 2B-parameter state-of-the-art MLLMs.
We compare the final Open-Qwen2VL model with the state-of-the-art partially-oppn MLLMs of InternVI.2.-2BBMMPPO (Wang et al., 2024b), DeepSeekVL-22Ting (Wu et al., 2024), and Owen2-VL--B-Instruuc (Wang et al., 2024a) on a set of general multimodal benchmarks, OCR VQA datasets, multimodal math reasoning benchmark, and hallucination benchmark. Con- cluded from the results in Table 4, Open-Qwen2Ve VL demonstrates competitive performance across benchmarks compared to other 2B- 2B-parameter state-of-the-art MLLMs. It particularly excels in MMBench, achieving the highest score of 80.9, while maintaining comparable performance in SEEDBench and MMStar benchmarks. Moreover, Open-Quen2211 outper- on MMBench, SEEDBench-img,
forms the most relevant competitor Qwen2-VL-2BBB - MMStar, and Math Vista, while it is only trained on 0.35% tokens of Qwen2-VI.. However, it shows relatively weaker results in OCR VQA tasks of Al2D and TextVQA. This is because the pre-training data of Open-Qwen2V does not include the OCR-specific caption dataset like SyGthDoG (Kim et al., 2022) or LAIONCOCO-OCK (Schuhmann et al., 2022). Simply introducing such OCR-related pre-training data will significantly enhance the OCR-VQA task performance of MLLMs, as shown in Appendix Table 9.
4 Analysis
Impacts of Sequence Packing on Multi-Image In-Context Learning and Reasoning. Flamingo (Alayrac et al., 2022) proposes MultiModal MassiveWeb (M3W) dataset to con- struct pseudo interleaving data structure using caption data to elicit the multimodal in- context learning capabilities of MILMs. The multimodal sequence packing also constructs similar pseudo image-text interleaving data strcture. Thus, we conduct experiments to evaluate the few-shot multimodal in-context learning capabilities of the pre-trained base MLLM trained on packed multimodal sequences. We evaluate the base non-sft MLLM trained on the caption data mixture of CCS-CLIP and DataComp-ML.M GQA, VQA-v2, VizWiz, OKVQA, and Text-VQA datasets. This base model is the best model we get based on the ablation studies on the pre-training data mixture in Table 3. We select 5 random seeds for the 8-shot multimodal in-context learning experiments and report the average performance over the random seeds. The results in Table 5 demonstrate that the base MLLM trained with packed multimodal sequences can learn from multimodal demon- stration examples for completing the task well. The 8-shot in-context learning can gain +3% to +12% performance improvements on VQA dataset compared with (-shot reasoning.
DFN on
8

---

Published as a conference paper at COLM 2025
shots
GQA
VQA-v2
VizWiz
OKVQA
Text-VQA
27.1
40.2
24.7
30.4
35.4
51.8
31.2
27.1
30.6
Table 5: Results of the 0-shot and 8-shot multimodal in-context learning capabilities of pre-trianed base MLLMs.
Vision
A12D test
MMMU
MMBench Dev
SEEDBench Img-Dev
MathVista test-mini
TextVQA
POPE
MMStar
Avg.
Encoder
val
Prozen
56.8
57.0
80.1
38.0
77.3
68.7
41.3
28.6
56.0
Trainable
57.4
57.6
82.3
36,1
76.5
69.7
41.4
29.3
56.3
Table 6: Ablation study on the trainable or frozen vision encoder during the SFT stage on LLaV6-665k data. We freeze vision encoder for pre-training stage due to limited computing resources.
This also demonstrates the necessity and significance of performing multimodal sequence packing as it can enhance the multimodal in-context learning capabilities of the MLLMs.
Impact of Unfreezing Vision Encoder Parameters. Most of the state-of-the-arr MLLMs like Intern (Chen et al., 2024b) demonstrate that making the vision encoder trainable during the SFT stages will enhance the multimodal understanding capabilities of MILMs. Therefore, we perform such ablation studies on our base non-sft MLLM pre-trained on DapaaaCoopp-MM--Flleerrere.
data mixture. We use the SFT dataset and evaluate the two SFT-ed MLLMs with frozen and trainable vision encoder during the SFT stage. The results in Table demonstrates that make the vision encoder parameters trainable during SFT stage can achieve better average performance while there significant performance degradation on MMMU benchmark.
data the visual
Advantage and Necessity of Large-Scale Efficient Pre-training. Tn investigate the effects and necessity of performing large-scale efficient pre-training proposed in Open-Qwen22V we reproduce LLaVA using its original 595k pre-training caption data and 665k instruction data and then compare it with Open- Quenn The reproduced LLaVA adopts the architecture design and module choices and only differs with Open- -wen-VQ in the pre-training pipeline and data. Concluded from Table 7, our model significantly outperforms LLa VAVAReprod uced on all benchmarks, demonstrating the significance and necessity of the large-scale efficient pre-training.
Related Work
Open-Source Multimodal Large Language Models. Close-source MLLMs like GPT- 4o (Achiam et al., 2023) and Claude-3.7SSonnet (Anthropic, 2025) have strong multimodal understanding and reasoning capabilities. To replicate the strong capabilities of close-source MLLMs, research teams from industry develop the open-weights strong MLLMs including LL225 (Chen et al., 2024b), DeepSeek- VL2 (Wu et al., 2024), and Qwen2.5-Vi LL (Bai et al., 2025), which can achieve equivalent capability with close-source models. However, the training data, codebase and data filtering details of such models are not open-sourced for reproduction.
additional to open-weights, the fully open-sourced MLLM including MOLMO (Deitke et al., 2024), SmolVLM (Marafioti et al., 2025), and our model open-source full training data and codebase to ensure the reproduction from scratch from the community.
Large-Scale Image Text Data. Starting from ImageNet (Deng et al., 2009), the large-scale image dataset has significantly driven advances in both computer vision and multimodal foundational models. MSCOČO (Lin et al., 2014), SBU (Ordonez et al., 2011), Conceptual Captions (CC) (Sharma et al., 2018h (bbb) scales up the image dataset size to near million level,

---

Published as a conference paper at COLM 2025
Model
MMMU
MMBench
SEEDBench
MMStar
A12D
Text-VQA
Math
POPE
LLaVA*
49.9
24.7
75.7
Open-QweenVVL 38.0
77.3
68.7
41.3
56.8
57.0
28.6
80.1
Table 7: Comparisons between Open- Tpen--ween2VI. and reproduced LLaVA various multi- modal benchmarks. Both Been-QwennVV. and LLaVA adopts LLaVA-665k as the SFT data, while differing in pre-training pipeline and pre-training data.
which significantly enhances the image captioning performance of VLMs. OpenAI pre- trained contrastive VLM, CLIP with 400M Weblmage data without releasing them. Then LAION-400M and COYO-700M are open-source efforts to further scale up the image-text dataset to hundreds of million level. Then LAION-5B and DateComp-commonpol 12.8B scale up the image-text dataset size to billions level for supporting the data-intensive MLLM pre-training- Most of SOTA MLLMs like DeepSeek-VL VL (Wu et al., 2024), Qwen-VL (Bai et al., 2023), Intern-VL (Chen et al., 2023), SAIL (Dong et al., 2025) construct and curate their own large-scale image-text dataset with more then 10B image-text data, while such dataset will not be released for public research.
High-Quality Image-Texx Text Data Filtering. Beyond the conventional rule-based heuristic- based data filtering methods in constructing image-text dataset, image-text dataset with much larger scale for training contrastive VLMs adopts CLIPScore-basse filtering methods for high-quality data curation. LAION-400M (Schuhmann et al., 2021) set a hard filtering threshold using OpenAI-CLIP model for its data filtering. Later, DataComp (Gadre al., 2023) is the first effective benchmark for fairly evaluating the effectiveness of each data fil- tering method on selecting high-quality data for CLIP pre-training. Various methods (Maini et al., 2023; Yu et al., 2023; Wang et al., 2024d) try to combine CLIPScore filtering with other metrics to achieve better filtering performance on DataComp, while DFN (Fang et al., 2023) directly scales up the CLIP-based data filtering model and achieves the top-1 performance. Moreover, another line of data filtering method based on efficient MLLM (Wang et al., 2024c; Huang et al., 2024) emerges and presents better capabilities in selecting high-quality data for MLLM pre-training
Conclusion
We demonstrate that the efficient MLLM-based high-quality data filtering techniques and well-designed data mixture strategies can achieve compute-efficient pretraining for develop- ing SOTA MLLMs. Adopting the multimodal sequence packing and dynamic image token number with average-pooling layer can further enhance such pre-training efficiency. The induced final MLLM, Open-Qwen2U1 outperforms the partially-open MLLM Qwen2-VL-28 on various multimodal benchmarks, in which Open-Qwwen2V1. is trained on only 0.36% pre-training tokens of Qwen2-VL. The training is conducted on academic-level comput- ing resources and demonstrates that the advanced training pipeline and data filtering can overcome the limitation of computing resources. We wish Open- Qwnn-Qu motivate the fully-open compute-efficient multimodal pre-traie traininin research from academic community.
Acknowledgments
We would like to thank for Facebook (now Meta) for donating the 8xA100-40G GPUs for conducting the experiments. We appreciate the open-source contributions of prismatic- vlms Karamcheti al. (2024) and vlm-evaluation on which we build our training and evaluation codebases. This work was partially supported by the BioPACIFIC Materials Innovation Platform of the National Science Foundation under Award No. DMR-1933487.
10

---

Published as a conference paper at COLM 2025
References
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, al. Phi-3 technical report: A highly capable language model locally your phone, ar preprint Xii2:2444.14219, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. Xiv preprint ar xrx::23033.08774 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-233 23736, 2022.
Anthropic. Claude 3.7 sonnet and claude code. claude-3-2 3-7-eonnet, 2025. Accessed on Feb 24, 2025.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint aix::22308 12966, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. ar preprint axXie:2502.13393, 2025.
Romain Beaumont. img2dataset: Easily turn large sets of image urls to image dataset. http:://gggguuu..cooooo//mm///// 504/ mttg///amtt/mmmmtasaatttt,t 2021.
https://github com/
Lin Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large 2024a.
vision-language models? ar Xiv preprint ar arXiv:2403.20333
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. Xiv preprint ar Xrx::2231 14238, 2023.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint ar axxi::2412.05221,, 2024b.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arv e-prints, pp. arXiv-2409, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Hongyuan Dong, Zijian Kang, Weijie Yin, Xiao Liang, Chao Feng, and Jiao Ran. Scal- able vision language model training via high quality data curation. arXiv preprint ar MxXim:2501.05955 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arxiv preprint arXiv:2407.2172 21..88,, 2024.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309. 17745, 2023.
11

---

Published as a conference paper at COLM 2025
Samir Yitzhak Gadre, Gabriel Ilharco,, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXio preprint ar XrX::22304.14108, 2023.
Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint ar ariio22412.0523/ 2024.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint ar arXfo:2104.08718, 2021.
Han Huang, Yuqi Huo, Zijia Zhao, Haoyu Lu, Shu Wu, Bingning Wang, Qiang Liu, Weipeng Chen, and Liang Wang. Beyond filtering: Adaptive image-text quality enhancement for mllm pretraining. arXiv preprint arXiv:2410.66166 2024.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
David S Johnson. Near-optimal bin packing algorithms. PhD thesis, Massachusetts Institute of Technology, 1973.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. In Forty-first International Conference on Machine Learning, 2024.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 235-251. Springer, 2016.
Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Conference on Computer Vision (ECCV), 2022.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXit:2405.02246 2024.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal 1lms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 11888-12900 PMLR, 2022.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXio preprint arXiv:2305.10355, 2023b.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. (16689-26699 2024.
Tsung-Ys Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. ppp.700-55 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. ar Xix preprint arXiv:2310.03744 2023.
12

---

Published as a conference paper at COLM 2025
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2024.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathe- matical reasoning foundation models in visual contexts. arXiv preprint urXiv:2310.02255, 2023.
Pratyush Maini, Sachin Goyal, Zachary C Lipton, J Zico Kolter, and Aditi Raghunathan. T-mars: Improving visual representations by circumventing text feature learning. arviv preprint arXi::2307.033 00312,, 2023.
Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, et al. Smolvlm: Redefining small and efficient multimodal models. arXiv preprint ar xix::2504.00599, 2025.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis insights from multimodal Ilm pre-training, arXiv preprint arXiv:2403.09611, 2024.
OpenAL. Gpt-4v(ision) technical work and authors. 2023. URL htttt://///dn. www.ad....ppii...oommm contributions/ggt-4v pdf.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 1m2text: Describing images using million captioned photographs. Advances in neural information processing systems, 24, 2011.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
Christoph Schuhmann, Andreas Köpf, Richard Vencu, Theo Coombes, and Romain Beau- mont. Laion coco: 600m synthetic captions from laion2b-en. URL https://laion. ai/blog/laion- coco, 5, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 2556-2565, 2018a.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 2556-2565, 2018b.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruy Batra, Devi Paríkh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. //qwenlm. tith tt/blog///ooooo ho/blog/qwen2..5..5
URL https:
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. ar preprint arXim:2409.12199 2024a.
13

---

Published as a conference paper at COLM 2025
Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXio:2411.10442, 2024b.
Weizhi Wang, Khalil Mrini, Linjie Yang, Sateesh Kumar, Yu Tian, Xifeng Yan, and Heng Wang. Finetuned multimodal language models are high-quality image-text data filters. arXiv preprint arXiv:2403.02667, 2024c.
Yiping Wang, Yifang Chen, Wendan Yan, Alex Fang, Wenjing Zhou, Kevin Jamieson, and Simon S Du. Cliploss and norm-based data selection methods for multimodal contrastive learning. Advances in Neural Information Processing Systems, 37:15028- 1-0188155069, 2024d.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-v12: Mixture-of- experts vision-language models for advanced multimodal understanding. ar ar preprint #rXiv:2422.10302, 2024.
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint ar grXtm:2408.08872 2024.
Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, and Lu Hou. Deco: Decoupling token compression from semantic abstraction multimodal large language models. arviv preprint arXiv:2405.2098 2024.
Haichao Yu, Yu Tian, Sateesh Kumar, Linjie Yang, and Heng Wang- The devil is in the details: A deep dive into the rabbit hole of data filtering. arXiv preprint ar 10o::239.11954, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesníkov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11975-11986, 2023.
Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mm1. 5: Methods, analysis insights from multimodal Ilm fine-tuning. arXiv preprint ar Xio:2409.20066 2024.
A Training Settings of MLLM Pre-Training
The training details and hyperparameters for MLLM pre-training are presented in Tab. 8. The pre-training only one-stage process. We do not follow Qwen-V WL or DeepSeek- -VL to split the MLLM pre-training into two stages of VL alignment and VL pre-training.
B Effects of Additional OCR Data in Pre-Training
The deficiency of OCR performance of Seeen-Qee22VL.. are mainly because the pre-training data of Open-Qwen2VL. does not include any OCR caption data due to the pre-training efficiency considerations. We perform another ablation study and include the SynthDoG OCR (Kim et al., 2022) pre-training dataset, which introduces additional 0.13B tokens into the pre-training dataset. Then after SFT on LLaVA-665k, we observe that the model achieves significant improvements on OCR benchmarks of TextVQA and AI2D. It demonstrates that the deficiency of OCR performance is not caused by the methodology disadvantages but only the composition of pre-training data.
14

---

Published as a conference paper at COLM 2025
Details
Pre- TreinTrinnng
Visual
Vision Encoder
SigLIPP-o440m-384pp
SigLIP-sor sp0_IP-s4000m-384x
Visual Projector
2d Adaptive Average Pooling + MLP
MLP
LLM Backbone
Swwen.5-115B-Insttee
Fren2.5-1.B-Inntruue
Tokens per Image
144
729
Context Length
4096
4096
Sequence Packing
Yes
No
Precision
BF16
BF16
Global Batch Size
256
128
Training Epoch
1
1
GPUs
8 A100-40G
A100-40G
Peak LR
5e-5
2e-5
Warmup Steps Ratio
3%
3%
LR Scheduler
linea-warmmp---oiiedddaayy
linear
Weight Decay
0.01
0.1
Table Training details and fine-tuning.
for MLLM pre-traingg and visual supervised
Pre-Training
MMMU
MMBench
SEEDBench
MMStar
A12D
Text-VQA
MathVista
POPE
Data
CCS DataComp
38.0
80.1
SynthDoG
37.9
77.0
69.6
41.0
57.6
59.7
28.8
82.0
Table 9: Ablation study on the effects of introducing additional OCR data in pre-training.
C
Examples
rtoouue Itas painting?
This painting is boautifa repesseration of Vincnt va Oughts famous work, 'Woss The painting is vibrort troes,
and deradle depicion is gardinn
van Gegt's un rtyle uerg boMt saturaked than of lero croad sense dosth moroment The bookground fostures mie grocn follage and riettsoo of yollow and orange, odding to the ove vooanny the stone. painting coptures the essonce of boauly
aorissal gaidon, making coptivatin
Cloudo 7
Bonnet Resporse
one most works, created during hra stay The paintl ripkis ganden vibort purple rites againet bookrinp of grown fáäggg and fowers of
Vincent Gogl's "rises"
carves. Gogh paimte single staniiing
thunacerrass add trick, supresa brualstrckes and bo use of color.
kirung nustine, Muscure Los Angelos und considooee one of most valuublo parl parrding world
hortenes
"he
housed Peex Gatty
Figure 3: An example for generating caption towards famous painting, which requires the background knowledge.
15

---

Published as a conference paper at COLM 2025
Salleg for donnet sores each rencee offiren oppect siraat ouraer
imggg" mage dacouss pepple destal Shore inkar stris Butt so stesting
MONGALL
Becoalaa light segiee de sap Aual daly poo
"Tg prage" daccs budding uta aheer prie striss lira
tus lis sosld bor my ane
provplly
comee yoars decise
not
aie teaiial binng pleaed the
qhay comkers oleer decrstio taan maro lon son'
aucic for odlect the
bemel Opmn
sibuad for dhouting coderr Hig descraier
pull chetr hatrall diecioees.
facioval cull comaabtry at big
cos lostred seatly Chlese Voa Year Modd Autern Feainal tenorns Her Raye coli and
Figure 4: An example for generating long captions for a complicated 2x2 image.
16


Reviews:
[{"review_id": "3dElfC1O6z", "reviewer": "Reviewer_aZwW", "summary": "The paper introduces Open-Qwen2VL, a 2B-parameter multimodal LLM pre-trained efficiently on 29M image-text pairs using academic-grade resources (8×A100-40G GPUs, 442 GPU hours). Key contributions include dynamic image resolution scaling (144→729 tokens), multimodal sequence packing to reduce padding, and hybrid data filtering (CLIP + MLM-Filter). The model outperforms Qwen2-VL-2B on benchmarks (MMBench, SEEDBench) despite using 0.36% of its pre-training tokens. The authors emphasize full openness, releasing code, data, and methodologies.\n\nThe paper is well written and makes a huge impact for the academia. The drawbacks like OCR underperformance, are mentioned in the limitations. The only point I want to outline - the more comparisons make the paper more strong.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Achieves SOTA performance with minimal resources, \n2. The release of the training codebase and pre-training/SFT data democratizes MLLM development for academic researchers.  \n3. Low-to-high resolution (144→729 tokens) and sequence packing improve training efficiency without sacrificing quality.", "reasons_to_reject": "1. Incomplete OCR Analysis. While TextVQA/AI2D performance lags (Table 4), the paper does not explore integrating OCR-specific pre-training data (e.g., SynthDoG).  \n2. Only 4 of 16 possible data combinations tested (Table 3), leaving optimal mixtures understudied\n3. The impact of freezing the vision encoder during pre-training vs. SFT is underexplored.\n4. Limited comparisons with other open-source MLLMs (e.g., LLaVA, OpenFlamingo)", "questions_to_authors": "Why exclude interleaved image-text data during pre-training despite its potential for in-context learning, as seen in Flamingo?\nWhat motivation is to use of SigLIP-SO-400M over other vision encoders, and how does it impact training efficiency?", "extra_scores": {}}, {"review_id": "qWfmp1FhFk", "reviewer": "Reviewer_RPfn", "summary": "The paper introduces Open-Qwen2VL, a 2B-parameter multimodal large language model (MLLM) trained with only 442 A100-40G GPU hours on 29M curated image-text pairs. It emphasizes full openness of data, code, and filtering pipelines, and demonstrates competitive performance against state-of-the-art models using significantly fewer resources. Key contributions include efficient data filtering using MLLMs, multimodal sequence packing, and a low-to-high visual resolution scheme. The final instruction-tuned model surpasses Qwen2-VL-2B in several benchmarks, while being more resource-efficient.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "All components of the project—including the training dataset, filtering pipeline, codebase, and final model checkpoints—are released, which significantly contributes to the transparency and usability of multimodal model research.\n\nThe model achieves competitive or even superior performance while using a fraction of the training budget required by similar models, making it highly relevant to resource-constrained settings.\n\nTechniques such as sequence packing and low-to-high image resolution scheduling are well-motivated and demonstrably improve training efficiency without architectural modifications.", "reasons_to_reject": "The backbone and visual encoder are largely reused; the work focuses on data and training improvements rather than introducing new model designs or training objectives.\n\nSome engineering details lack theoretical justification, and could benefit from more systematic analysis.\n\nWhile the ablations are helpful, the work would benefit from additional analysis, such as statistical significance tests, longer training runs, or cross-dataset generalization evaluation.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GUamTkOKeG", "reviewer": "Reviewer_eK8t", "summary": "The paper presents Open-Qwen2VL, a fully open-source 2B-parameter multi-modal LLM pre-trained on just 5B tokens. The authors rely on careful data curation using both MLM-based filtering techniques and CLIP-based filtering methods, and increase training efficiency by packing tokens into sequences and leveraging low-to-high dynamic image resolution. The resulting model outperforms Qwen2-VL on select benchmarks. The authors promise to release the model along with the full training data and training code.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Demonstrates how to increase pre-training efficiency,  which makes such research more accessible. Promised open release of training artifacts will be valuable for the community.", "reasons_to_reject": "The reported benchmarks in Table 4 are a substantially smaller list than e.g. Table 2 in Qwen2, leaving open how the model performs on the remaining tasks. That raises the question whether the claimed training efficiency might come from improving on some (the reported) tasks at the cost of other (non-reported) tasks. The results and related discussion regarding OCR tasks also hint in this direction.", "questions_to_authors": "Please respond to the questions raised in \"Weaknesses\"", "extra_scores": {}}, {"review_id": "biRQyNMDKl", "reviewer": "Reviewer_G2dF", "summary": "This paper demonstrates that Multimodal LLMs can be effectively trained with academic-level resources. Specifically, it presents Open-Qwen2VL, a 2B-parameter MLLM trained with only 442 A100-40G GPU hours. The model was pre-trained on 5B tokens, which is just 0.36% of the 1.4T tokens used for Qwen2-VL. Despite the much smaller compute and data, Open-Qwen2VL achieves comparable or even better performance compared to Qwen2-VL.\nTo enable compute-efficient training, the authors introduce techniques like low-to-high dynamic image resolution and multimodal sequence packing. The training dataset was also carefully curated using various filtering methods.\nImportantly, this paper fully open-sources the entire training pipeline, including the model, data, code, and filtering scripts, contributing significantly to the research community by making MLLM pretraining accessible to those with limited resources.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper demonstrates that it is possible to train a strong MLLM using only academic-level resources.\n2. It effectively leverages techniques such as low-to-high dynamic resolution to maximize cost-efficiency, highlighting strategies that are often overlooked by industry models with abundant compute.\n3. It makes a major contribution to the community by fully open-sourcing not just the checkpoints but all the components necessary for training.", "reasons_to_reject": "1. The paper does not provide a comparison with concurrent works targeting similar goals, such as Molmo [1] and SmolVLM [2]. A detailed analysis comparing Open-Qwen2VL against these models in terms of training cost, model performance, and degree of open-sourcing for reproducibility would substantially strengthen the paper.\n2. Models like LLaVA, which do not rely on multimodal pretraining, have demonstrated that strong MLLMs can be trained using only projector training and visual instruction tuning with academic-level resources. In this context, it is important for the paper to justify why full multimodal pretraining is necessary under similar resource constraints, and to show whether it provides meaningful advantages over visual instruction tuning alone.\n3.  The paper does not explicitly evaluate the model on multi-image or video tasks. It remains unclear whether Open-Qwen2VL achieves comparable performance to Qwen2-VL in these settings.\n\n[1] Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models  \n[2] SmolVLM: Redefining small and efficient multimodal models", "questions_to_authors": "Please refer to the Reasons to Reject section above.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Open-Qwen2VL,  a compact 2B-parameter multimodal model trained with minimal compute, emphasizing efficiency and full transparency. The authors leverage techniques like dynamic resolution scaling, sequence packing, and curated data filtering to reduce resource requirements while maintaining strong performance. The model achieves competitive results across diverse benchmarks and demonstrates meaningful improvements over comparable models, such as Qwen2VL, LLaVA, SmolVLM, etc. These results demonstrate the effectiveness of the proposed training recipe to attain strong yet cost-efficient VLM. The full release of code, data, and training pipelines will significantly enhance reproducibility and accessibility for academic research.

During rebuttal, the authors further effectively addressed reviewer concerns through additional experiments, ablations, and broader comparisons. As such, all reviewers reached a consensus about the contributions of this work and all gave a high rating (7, accept) on this work. The Area Chairs agreed with all reviewers and appreciate the authors' work and rebuttal, and recommend a clear acceptance.
Decision: Accept