Should VLMs be Pre-trained with Image Data?
Authors: Sedrick Keh
Jean Mercat
Samir Yitzhak Gadre
Kushal Arora
Igor Vasiljevic
Benjamin Burchfiel
Shuran Song
Russ Tedrake
Thomas Kollar
Ludwig Schmidt
Achal Dave
Venue: ICLR 2025 Poster (2025)
Keywords: vision language models
pre-training
fine-tuning
Primary area: foundation or frontier models, including LLMs
TL;DR: We vary amount of image data used in pre-training VLMs, questioning the conventional formula of fine-tuning pre-trained LLMs into VLMs

Abstract: Pre-trained LLMs that are further trained with image data perform well on vision-language tasks. 
While adding images during a second training phase effectively unlocks this capability, it is unclear how much of a gain or loss this two-step pipeline gives over VLMs which integrate images earlier into the training process. 
To investigate this, we train models spanning various datasets, scales, image-text ratios, and amount of pre-training done before introducing vision tokens.
We then fine-tune these models and evaluate their downstream performance on a suite of vision-language and text-only tasks.
We find that pre-training with a mixture of image and text data allows models to perform better on vision-language tasks while maintaining strong performance on text-only evaluations.
On an average of 6 diverse tasks, we find that for a 1B model, introducing visual tokens 80\% of the way through pre-training results in a 2\% average improvement over introducing visual tokens to a fully pre-trained model.

Published as a conference paper at ICLR 2025
SHOULD VLMS BE PRE-TRAINEL WITH IMAGE DATA?
Sedrick Keh*! Jean Mercat Samir Yitzhak Gadre' Kushal Arora' Igor Vasiljevic' Benjamin Burchfiel' Shuran Song" Russ Tedrake" 1,3 Thomas Kollar Ludwig Schmidt Achal Dave
Toyota Research Institute, Stannoord, MIIT, sedrick keh, jean. mercat @etri.global
ABSTRACT
Pre-trained LLMs that are further trained with image data perform well on vision- language tasks. While adding images during a second training phase effectively unlocks this capability, it is unclear how much of a gain or loss this two-step pipeline gives over VLMs which integrate images earlier into the training process. To investigate this, we train models spanning various datasets, scales, image-text ratios, and amount of pre-training done before introducing vision tokens. then suite of
fine-tune these models and evaluate their downstream performance Assion-language and text-only tasks. We find that pre-training with a mixture image and text data allows models to perform better vision- vision-language tasks while maintaining strong performance on text-only evaluations. On an average of 6 diverse tasks, find that for 1B model, introducing visual tokens 80% of the way through pre-training results in a 2% average improvement over introducing visual tokens to fully pre-trained model.
DCLM
DataComp-DR
LLaVa
Data sources
TXT
pleces information per deconal Soued impossible? That's esactly how feet [-]"
"inaghe processing 400xbilion
Ice view the electrnnic board a computer
"Human": "What could polental empty
'GPT: "A potental reason L..]
LLM Pre-training
Image-text Pre-training
Fine-tuning
28B tokens
1B tokens
training
3.4T tokens
jo
10
npochs
Mix of data source
Figure 1: An overview of our VLM pre- vre-training data recipe. We investigate data mixes and design choices for text-only pre-training. image-text pre-training, and fine-tuning. Note that while we depict "LLM Pre-training" and "Image-text Pre-training" as two separate steps in this diagram, in practice, we continuously transition from the first stage the second.
INTRODUCTION
Numerous work have extensively studied how to train a VLM from pre-trained LLM (Liu et al.
2024
Alayrac et al.
2022
Bavishi et al.] 2023: Li et al. 2024a Karamcheti et al. 2024 Tong et al.
Laurençon al., 2024 Lu et al., 2024] Wu et al. 2024a bj Chen al. 2025). ). Most existing VLMs follow roughly the same leve- recipe: 1) Start with pre-trained
2024 Beyer et al.
2024
LLM, 2) continue training intermediate layers on image-text tokens (e.g., Flamingo, Pali-Gemma,
Idefics, DeepSeek-VDD VL) align image projection layer (e.g.. LLaVA, Cambrian, Prismatic), and
3) fine-tune the model on multi-task instructions or chat templates. Notably, training with images included in the initial pre- pre-training stage (i.e., step 1) is largely undocumented in these papers, though a number of large models (with undocumented training procedures) are known to be natively multimodal (e.g., Pixtral, Gemini, Fuyu).

---

Published as a conference paper at ICLR 2025
In this paper, investigate the significance of each step in the pipeline for downstream performance on vision-language and text benchmarks. Additionally, we examine whether increasing the amount of image data in one step reduce the data requirements in other steps. To understand when image data should be introduced to VL. training, we train suite of 300 models over various numbers of parameters, varying the amount of text-only pre-training data, well as the amount, type, and ratio of image pre-training data (Figure D).
Our experiments suggest several key findings:
First, incorporating image data during pre-training generally helps, especially after a model has many text tokens. However, the timing and method of introducing visual information also important. Specifically, we find that it is beneficial to add image data before the LLM is fully pretrained, during the cooldown phase. Our strategy to interrupt the text pre-training 80% completion and add image-text data outperforms the popular alternative of fully training the LLM, then re-warming and cooling the model with image data (Section 3.1).
Second, the fraction of visual data introduced during cooldown is another key parameter for strong performance across domains. At the IB parameter regime, our experiments reveal that 10% to 20% of tokens should be visual. Going above or below this ratio results in worse downstream performance. However, this fraction appears to function of scale. the smaller 79M parameter regime, larger fractions of visual data are preferred (Sections 3.2 and 3.3).
Third, the timing of when to introduce instruction fine-tuning image tokens is crucial for downstream performance, both for pure text and vision-language tasks. We observe that mixing together instruc- tion fine-tuning during the image-text pre-training process actively hurts the model (Section (.4). Meanwhile, adding instruction tokens during fine-tuming improves dision-language task performance up to 4 epochs at the IB parameter scale, at the cost of slightly hurting performance on text-only tasks (Section 3.5
2 EXPERIMENTAL SETUP
Image encoder
MLP
TXT
Decoder-only Transformer
top view of the electronic board a
TXT
Text embedding
computer
Figure 2: The commonly used framework we apply to add vision capabilities to a transformer model.
Our experimental setup is designed to study the impact of image-text data the model's 's downstream results on both text and vision tasks. To this end, we implement a commonly used model architec-
ture (Liu al., 2023 Tong al. 2024 McKinzie et al. 2024 Beyer et al. 2024 Bai et al. 2023; Li
ct al. 2023a Laurençon et al. (2024) consisting of pre-trained image encoder, a projection block. and decoder-only transformer. Unless stated otherwise, the image encoder is SigLIP 400M (Zhai ct al. 2023), the projection block is two-layer MLP, and the transformer is the 1.4B parameter model described in Gadre al. (2024b). Our training procedures (Section 2.1 are applied to the model prior to its downstream evaluation (Section 2.2
2.1 TRAINING PROCEDURE
We train all our models with three-step process illustrated in Figure
Partial text-only pre-training
2. Image-text pairs mixed with text-only continued pre-training
3. Image-text pairs multi-task fine-tuning
We can view the first and second steps as single pre-training stage, where we introduce images before the model has finished pre-training, opposed previous methods where images introduced
2

---

Published as a conference paper at ICLR 2025
Table I: The conventional approach VLM training first fully trains language model text-only data, and adds images in a second or third stage of training. In this work, we instcad introduce image data earlier in pre-training. To do this efficiently, we resume a language model during the course of its pre-training at various stages (e.g,, 20% of the way through training) and introduce images midway through training.
Text-Omly Pre-training
Image- Pree-training
Tine-Tuning
BLIP
Fully trained (Phi3-mini)
warmup
Re warmupp
Caytion interle text-image ditta, no pure text
Plamingo
Fully truined
warmup
(Skipped)
Alayyac
Caption
image no pure
DEFICS
Fully trained (Mistral: v0.1)
warmup
Re warmuu
Laurençon
image no pure text
MMT
Fully pre-rrineed
Re- Re-warmup
Re-warmup
McKinze
image
(100:9,
66:33)
DeepSrok-V1
V1.2
Fully prstrained (DeapSesk)
Two-stage (re- Pro-aammup then re-warmep)
Re-warmup
[l 0024
Waee
Mulitask; 50.700 image-test
Qwan-Vi
Fully pretrained (Qwas)
Two-stge (re-warmmp then re-wammpp)
Re-warmup
0023
Caption muhtioek; pure
Palioeemia
Fally pre me-triined
Two stage (ze- warnuup then contimee LR schedule)
Continued LK schedule
[Reyerat
2024
Mulittsk; pure
Jamus Janus Pro
Fally pre unined (DeopSeck LLM)
Two stage (re- waraspp then re waramup) Muliitsk;;
No warmup
al.
MM2 Chca 2025
followed by 30:70 image ratio No image- text pre-iraining.
Prismatic
Fully pe-trained (LLaMA)
Re Re-warmup
(Karamcheti
Ours
Partially pre-trineed (Opent
Coninnee LR schedllee
Re-warmup
Various image-lext ratios
to model which had already been pre-trained. In practice, this is implemented by resuming from the same learning rate schedule (Section 2.1.2p. more detailed comparison overview of different models can be found in T
2.1.1 TEXT-ONLY PRE-TRAINING
During the initial pre-training process, only the LLM backbone is trained with the training recipe from Gadre al. (2026)). Rather than re- traininng from scratch, we instead leverage existing open checkpoints from SCLM-1B (Li et al. 2024b) [ This is a 1.4B parameter model trained on text tokens from M-LM-slliile mixed with StarCoder (Li al. 2023b) and ProofPile (Azerbayev et al., 2024), for total of 4.3T tokens, and it outperforms the recent LLaMA-3.2 IB model [ on several commonly used text benchmarks.
The pre-training learning rate schedule is warmup-cosine with a peak learning rate of 10-2 and final learning rate 10 10-5,1 In addition to using the final model, we also use checkpoints from 20%, 40%, 60%, and 80% through training, which correspond 860B, 1.72T, 2.58T, and 3.44T training tokens respectively. Since these checkpoints are taken from the middle training, their learning rates higher than the typical last learning rates of most released model weights, allowing to effectively resume pre-training (Figure 3)
Additional hyperparameters for the text-only pre- pre-training stage be found in Appendix A
2.1.2 IMAGE-TEXT PRE-TRAINING
We take checkpoints from the text-only pre-trained model and resume the cosine learning rate schedule with a cooldown phase, as illustrated in Figure
For experiments, perform multi-modal training using both caption data and task-related instruction data. As opposed to other VLMs like MM1 (McKinzie et al. 2024), BLIP3 (Xue et al.) 2024h, and IDEFICS (Laurençon al. 2024) which use interleaved data in addition to caption data, we limit our setting to either text only or text + single image. Our minimal setting allows us to control as many factors as possible isolate the most fundamental factors to train a performant multi-modal model.
httoss//nuggioofcs..co//tt
https://hugginggaaccccom
www.aaaaaaaammm 3.2-11
3

---

Published as a conference paper at ICLR 2025
0.010
Main Schedule Warmup-cosine 0%
Learning Rate
Cosine 20%
0.005
Cosine 40%
Cosine 60%
Cosine 80%
0.000
4.3T
Warmup-cosie 100%
Token Count
Figure 3: Representation of the different learning rate schedules used for our experiments. 'Main schedule' corresponds the learning rate for the initial, text-only pretraining. Other colored schedules are the ones used for image-text training and extend over 28B tokens each. They have been upscaled and appear as extending over 280B tokens for readability.
There are number of design choices involved in this stage. We give overview of these design choices below:
Text dataset selection: During the image-text pre-training stage, train with the ACL.-baseline dataset, mixed with StarCoder and MathPile. This is selected to match the dataset used during the text-only pre-training step to ensure continuity from the LLM checkpoints that we resume.
Caption dataset selection: We train with the DataCompDR-I caption dataset (Vasu et al. 2024), DataComp-11 (Gadre et al. 2024a) by regenerating higher quality
which is enhancement captions. To arrive at this selection, we ablated over several caption datasets (Appendix I.Tp.
Image-text ratio: The effect of mixing text and images has been thoroughly documented and is one of the key questions we explore in this paper. McKinzie et al. (2024) show that adding some text tokens helps with few-shot and text-only performance, though they show only up 33% text and operate at slightly different settings (fully pre-trained vs partially pre-trained model). DeepSeek- VL (Lu al., (024) uses a fixed 70% text ratio but does not discuss other ratios. We conduct thorough investigation on the image-text ratio in Sections 3.3 and 3.2
Amount of training: We train all our models with token multiplier 20, following approximate Chinchilla optimal scaling (Hoffmann al.) 2022). For a 1.4B model, this equates to around 28B tokens. From our experiments at smaller scale, we observe that trends generally hold across higher token multipliers (Appendix G
We found the SigLIP encoder to work well for our setting, which confirms findings from Beyer al. (2024). The DINO+SigLIP combination from Karamcheti et al. (2024) gave slightly better results on some benchmarks; for simplicity, however, we use SigLIP for most of our experiments. See Appendix J.2 for full ablations.
Image encoder selection:
Frozen encoder weights: The vision encoder can be kept frozen KKramchhtti et al. 2024) (i.c., its parameters not trained) trained end-to-end with the language model (Tong et al., [2024] Beyer et al. 2024). In our small-scale ablations, we found more effective to freeze the image encoder weights in both the pre-training and fine-tuning stages (Appendix [}. freeze the weights for the rest of this section.
Each image is encoded to 729 tokens, and train all models for sequence length of 1024. We provide additional implementation details in Appendix
2.1.3 INSTRUCTION FINE-TUNING
We fine-tune pre-trained models with the LLaVA dataset (Liu et al. 2024) using the Prismatic framework (Karamcheti et al., 2024p. For each model trained in the previous step, we fine-tune for {1, 2, 4} epochs. We cosine learning rate schedule with warmup. To account for different epoch numbers having different learning rate schedules, we treated each epoch number as its own separate run. A full list of hyperparameters can be found in Appendix
4

---

Published as a conference paper at ICLR 2025
2.2 EVALUATION
We want our VLMs to perform well in vision-language tasks while still retaining their performance in text-only tasks. As such, conduct evaluations over a suite of diverse tasks, both image-based and text-based. In our early experiments, we trained a 79M parameter model with text and used it to search the training space before scaling the 1B experiment presented in this work. We only keep the downstream text tasks where a 79M parameter model performs better than random chance. We test our model without any in-context learning, thus all results reported are O-shot.
Vision-language tasks:
We evaluate on subset of vision- tasks used by Karamcheti al. [224 These tasks test general visual reasoning, spatial reasoning, ability to read text. object localisation and hallucination. For a full set of vision-language evaluation tasks, see Appendix B
In addition to reporting the scores on individual tasks, we also report an aggregate metric. All our results are accuracy scores (between [0, 1)) To aggregate them, we subtract the random baseline accuracy, then average the result over all tasks. We call this aggregate metric the stable score. Its absolute value can be interpreted as the percentage points of advantage over random guessing. Its trend can be understood as average percentage points on the suite of benchmarks.
Text tasks: We evaluate on suite of tasks taken from Gadre al. (2024b) and conduct evaluations with Eleuther's LM Harness (Gao et (2024). These tasks test for general and specific knowledge, reasoning, and commonsense. We also compute the stable score previously described. For a full list of our text evaluation tasks, see Appendix
3 RESULTS
This section presents a series experiments exploring different aspects of multi-modal training. We aim to find important choices that affect the downstream performance of the model on set of diverse visual tasks and text-only question answering.
All experiments below conducted the 1.4 billion parameter regime. We make of various checkpoints of the DCLM-1B model along its text-only pre-training. This is explored in more detail in Section 3.1 below.
All experiments use in different proportions three dataset sources. (1) A text-only dataset, DCLM- baseline (Li al. 2024b), which was used pre-train the IB language model. (2) An image caption dataset, DataCompDR-1B Vasu et al. 2024), and (3) An image instruction tuning dataset, mix of openly available data described in A. A.I/Multimodd Instruct Tuning (Karamcheti al. (2024).
Each section below discusses different set of experiments with given setup summarized in green box labeled Setup. The setup describes which checkpoint of the text-only pre-trained model is used, the mix of text and image captioning data used for image-text pre-training, and the number of fine-tuning epochs performed on the image instruction tuning dataset. Within each description, we use denote the variable that we are changing for that particular experiment, while keeping the rest of the variables constant.
For each of the sections below, there will be two plots, each with two vertical axes, representing our experiment results on laiion-languag and text tasks. Each point on a plot represents a IB model initialized from one of the text-only pre-trained checkpoints, pre-trained with text and image captions, and fine-tuned with image instruction tuning data. The plot on the left represents the stable score, an aggregate of performance results, across the vision benchmark the other. The plot on the right represents one of each set of benchmark, the VQA-v2 (Goyal et al.) 2017) results and the ARC-easy (Clark et al. 2018) results. These might be easier to interpret but do not convey the full extent of the evaluation. The full list of benchmarks as well as detailed graphs for the vision performance can be found in Appendix
hand and text benchmarks in
5

---

Published as a conference paper at ICLR 2025
3.1
THE IMPACT OF AMOUNT OF TEXT-ONLY PRE- TRAINING
Setup
For this experiment, we use different checkpoints along the text-only pre-training. The initial text model was trained with warmup-cosine learning rate schedule. All continued image-text pre-training runs follow the learning rate schedules represented in Figure
x% checkpoints of 1B model
Image-text pre-trained for 28B tokens:
90% Text-only
10% Image captions
3 We continue training starting from the last text-only learning rate value, if not too low, and con- tinue training for 28B, For 0% and 100% the learning rate would be too low, thus adopt linear warmup-cosing decay with maximum learning rate of 3 x 10-3 See Figure
Fine-tuning 4 epochs
Figure below shows the progression of the model performance function of the text pre-training completion of the initial checkpoint. We see that up to 80%, a larger amount of text-only pre-training improves performance for both text and vision tasks, suggesting that it's helpful to start with stronger model which has been trained for greater number of text tokens. However, at 100% the learning rate schedule is different and could affect the results. Many similar works all initialize their model to 100% text-only pre-traines model and all re- warmmup the learning rate during the image-tent pre- pre-training phase. From our experiments, we observe decrease in performance at 100%, suggesting that continued training is preferable to re-training a 100% fully pre-trained text model.
Vision stable-score
Text stable-scor
Vision vqa-v2
0.54
easy
0.44
0.76
0.53
0.42
0.13
0.40
0.11
0.74
0.50
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Text pretraining completion ratio
Text pretraining completion ratio
Figure Varying the length of text-only pre-training. We analyze the impact adding image data after varying amounts of text-only pre-training, showing results vision benchmarks (green) and text benchmarks (blue). On the left, we show results across a suite of vision and text benchmarks; on the right, plot two common benchmarks, VQA-v2 and ARC-easy. Introducing images around 80% the way through training performs best, maintaining high vision and text task performance. Note: The points at 100% marked with hollow circles highlight that they trained with different learning rate schedule, as shown in Figure
Longer text pre-training improves performance but, when adding image input, continued training gives better performance than re-weet warming up.
3.2
THE IMPACT OF ADDING IMAGES BEFORE THE END OF PRE-TRAINING
Building on the results from Figure we take the 80% checkpoint and vary the image-text ratio we add during pre- mre-training. Figure S shows the evolution of the model performance as we pre-train (from text- only pre-trained model) with more image data. We keep the total number of pre-training tokens constant at 28B, adjusting only the image-text ratio.
Setup
80% pre-trained IB model
Image-text pre-trained for 28B tokens: 2% text
x%% image captions
Fine-tuning 4 epochs
From Figure 5 we see that as more image data is added to the mix, better downstream performance is obtained vision tasks. However, text-only downstream performance decreases. When only image data is used for image pre-training, both text and vision task performance drop significantly. We observe that from 0% images (i.e. train 100% text) to 1% images, there slight drop in vision stable score performance. We believe this can possibly be attributed the model being confused at the addition of this new task, and that it requires

---

Published as a conference paper at ICLR 2025
some minimum number of tokens in order to truly learn the image representations. Interestingly, per- formance on text-only tasks gets slightly better when some image data added to the pre-training mix.
Introducing image data at late stage during pre-traine pre-training improves vision performance while maintaining text performance.
Vision stable-score stable-score
Text stabie-score
0.770
0.15
Vision vqa-v2
Text arc easy
0.765
0.5
0.10
0.44
0,760
0.43
0.05
0.755
0.3
0.42
5% 10%
100%
10%200%
Pretraining Image Ratio
Pretraining Image Ratio
Figure 5: Varying the ratio of image text data, after some text-only pretraining. We analyze the impact of the ratio of image to text data in pre-training, after the model has seen text-only data for most of pre-training (80%). Unlike when training from scratch (Figure we find that adding vision data significantly helps vision performance, while maintaining high text accuracy.
3.3
THE IMPACT OF ADDING IMAGES WHEN TRAINING FROM SCRATCH
Setup
This section mirrors Section 3.2 with the only change being starting the image-text pre-training from scratch opposed from an 80% checkpoint. Figure [ shows the evolution of the model perfor- mance as we pre-train the model from scratch with more image data.
0% randomly initialized IB model
Image-text pre-trained for 28B tokens:
2% text
x%% image captions
Fine-tuning 4 epochs
We observe slighly different trends for models trained from scratch (Figure 6) compared to the models trained from the 80% checkpoint (Figure(3). Here, as more image data are added to the mix, we obtain worse downstream performance on both text and vision tasks. These models were trained for 28B tokens each and would likely benefit from more training; is possible that at larger scales, these trends will more closely resemble those in Figure Interestingly, just like in Figure adding 1% image data strongly degrades the ability of the model to be fine-tuned for vision tasks.
Vision stable-score
Text stable-scoeo sabbe--cooee
Vision vga-v2
0.74
Text arc easy
0.38
0.10
0.4
0.05
0.34
0.72
0.3
0.32
10%
1%
5% 10%
Pretraining Image Ratio
Pretraining Image Ratio
Figure 6: Varying the ratio of image text data, when training from seratch. We analyze the impact of the image-text ratio in pre-training from scratch without any language-only pre-training. Perhaps surprisingly. when training from seratch, adding vision data consistently hurts both vision and text performance, suggesting period of language-only training early is important for VLMs.
Introducing image data when training from scratch at small scales does not improve down- stream vision performance.
7

---

Published as a conference paper at ICLR 2025
3.4
THE IMPACT OF INSTRUCTION TUNING DATA IN PRE-TRAINING
We train a 1B model with mix of 10% image-text and 90% text-only data and vary the source of the image-text data from purely image captions to purely instruction tuning data. Figure 7 shows the evolu- tion of the model performance as we increase the proportion of image instruction tuning of image-text data. We observe that the best perfor- mance on vision-language tasks occurs when there are no instruction tuning data included in the pre- training mix, although adding instruction-tuning data seems to positively affect scores on text-only tasks. From vision-langaus standpoint, conclude that image instruction tuning data should not be added to the pre- trainiiinng mix and only be used a separate fine-tume senettunin stage.
Setup
80% pre-trained IB model
Image-texi pre-trained for 28B tokens:
90% text
10 x% image captions
the mix
2%% image instruction tuning data
Fine-tuning for epochs
Vision stable-score
0.46
Text stable-score
Vision vga-v2
0.765
0.62
Text are .a.sssy
0.44
0.17
0.760
0.42
0.58
0.755
0.40
0.58
0.750
0.38
0.15
Liava Image Ratio
Llava Image Ratio
Figure 7: Varying the proportion of instruction tuning data in the image mix. Is including instruction tuning data during pre-training helpful for VLMs? Surprisingly, we find that adding this data to pre-training harms performance. We hypothesize that this may be due to overfitting, or because mixing instruction tuning data with image-caption pairs degrades learning this scale.
The presence image instruction tuning data in pre-training harms downstream evaluations.
3.5
THE IMPACT OF FINE-TUNING ON VISION AND TEXT PERFORMANCE
Figure g shows the evolution of the model IB perfor- mance as it is fine-tuned for I to 6 epochs. Looking the stable score, observe that up to 4 epochs, downstream vision task performance improves and the downstream text performance degrades. Beyond
Setup
80% pre-trained IB model
Image-text pre-trained for 28B tokens: 90% text
10% image captions
4 epochs of fine-tuning, both vision and text perfor- mance degrade, likely due to overfitting.
Fine-tuning for epochs
Vision stable-score
0.46
0.16
Text stable-score
Vision vqa-v2
0.775
0.65
Text arc easy
0.44
0.15
0.770
0.55
0.14
0.765
0.50
Fine-tuning epochs
Fine-tuning epochs
Figure 8: Varying the number of fine-tuning epochs. We find that fine- finettnnig for 2-4 epochs after
pre- trainiain performs best for vision tasks, with 2 epochs being a sweet spot for maintaining text performance while achieving high vision performance.
Fine-tuning for image instruction improves dision-language tasks even beyond one epoch, up point, but degrades text-only task performance.
8

---

Published as a conference paper at ICLR 2025
4 RELATED WORK
The related work covers three subjects, first, two-stage vision language models (VLMs), which rely on pre-training with text-only data, then recent VLMs which introduce vision data earlier in pre-training, and finally, multimodal datasets for VLM training.
Two-stage VLMs. The standard approach for training vision language models involves first training large language model on text-only data, and then incorporating vision data in a second fine-tuning phase. This strategy employed effectively by models including Liu al. (2024) 2023) (LLa VA and variants), (llyyac et al. (2022) (Flamingo), Li et al. (2023a); Xue et al. (2024) (BLIP-2 and BLIP-3), Laurençon et al. (2024). Laurençon et al. (2023) (IDEFICS and Obelics), Chen et al. (005)) (Janus and Janus-Pro) among others. MccKinzie et al. (2024) explore design choices for training VLMs complementary to the ones we assess, namely the vision-language connector choice, the pretraining strategy for the image encoder, and the image resolution, while using two-stage training strategy with a fixed, pre-trained language model.
Pre-training with visual data. Bavishi et al. (2023) (Fuyu-8B) is a notable exception which trains a Lssion-anguuge model from scratch, but the training strategy is not publicly released. Team (024) introduces series of early-fusion VLMs, which, similar to our work, introduce image data during pre-training but choose a specific, handcrafted strategy for training. For multi-modal pre- pre-training. Aghajanyan et al. (2023) find scaling laws for pre-training from text, code, image-text, image and speech data in terms of unimodal and bi-modal combinations. They investigate training dynamies for various mixtures of pre-traini pre-training data and model sizes in terms of perplexity. Their study is related to ours in the sense of tre-trainig with mixtures of data, but it differs in that it does not involve any downstream evaluations. We focus specifically on the setting of VLM training, and thoroughly investigate the results on standard benchmarks.
Datasets Collecting image-text pre-training and instruction data is challenging, due to the low quality of image-text pairs on the internet. Most datasets use combination of curation and re- captioning to address this, including Desai et al. (2021) (RedCaps). or Thomee et al. (2015) (YFCC- 100M). In this work, we specifically rely on recent large-scale, high-quality datasets that perform well for training multimodal models: [Changpinyo et al. (202T) (ConceptualCaptions 122M follows the ConceptualCaptions-3M (Sharma et al. 2018) pipeline, curating images and alt-text pairs from the internet, with relaxed filters to increase the dataset size. Gadre et al. (2024a) (DataComp-I) IB) uses CLIP-based filtering to curate a high-quality set of 1 billion image-text pairs, and Vasu et al. (0024) (DataCompDR-11) uses rision-languan DataComp-1
model to generate synthetic captions for the original 1B) further improves on this by using a LLaMA3-based LLaVA-1.5 model (Liu et al.) 2023) perform the recaptioning. For finetuning, we use the LLaVA-1.5 (Liu et al. 2024 2023) mix, which consists of high quality, curated image datasets, along with instruction-tuning data generated using large language model. Recent datasets further explore interleaved data, where the same sequence counts several images and text piece that relate to each other. Interleaved datasets include OBELICS (Laurençon et al., (003), the Cauldron (Laurençon) et al.) 2024), and MINT-IT (Awadalla et al. 2024). We leave the analysis of this data for VL.M pretraining to future work.
1B IB dataset. Li al. (2024c)
5 CONCLUSION
In this paper, we address some key gaps and challenge several common assumptions in vision- language model (VLM) pre-training. Specifically, our work questions the traditional practice of separating text and image pre-training phases, demonstrating that a more integrated approach of incorporating image data during pre- pre-training can yield superior downstream results. We recommend future VLM efforts leverage intermediate pre-training checkpoints to incorporate images before completing the pre-training process. In this setting, it's important to carefully manage the image- to-text ratio: unlike the typical approach where a separate image training stage is done with 100% caption images, this integrated setting requires balanced image- ratio avoid performance degradation. We plan to make our code and our testbed of models publicly available, and we hope that our findings will provide strong empirical foundation for open-source VLM pre-training.

---

Published as a conference paper at ICLR 2025
REFERENCES
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning, pp. 265-279. PMLR, 2023.
Jean- -aptittt Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Concel-Kedzions'e Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation- -based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 2357-2367 Minneapolis, Minnesota, 2019. Association for Computational ac http:://cccaannooooooooro//////
Linguistics. doi: 10.18653/11/N19- 1245.
Anas Awadalla, Le Xue, Oscar Lo, Manli Shu, Hannah Lee, Btash Kumar Guha, Matt Jordan, Sheng Shen, Mohamed Awadalla, Silvio Savarese, et al. Mint- lt: Scaling open-source multimodal data by 10x: A multimodal dataset with one trillion tokens. arXiv preprint arXiv:2406. 11277, 2024.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2024. URLhttts:://azxt www./org/aaa. abs
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond, August 2023. URL
2308. 12966v3
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye. Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Fuyu-8b: A multimodal architecture for ai agents, 2023. URL https:
/ww..adeptt. http//log//uuuu/
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. In Transactions on Machine Learning Research (TMLR), 2023. https
//openreview net/ htt:://ormmmdidime Lote/orrrrri-ddy-sssss
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint 24Xi::2407.07726,, 2024.
Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA:
reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432-7439 AAAI Press, 2020. URL
https://aaai. /tw///////////////////////////// http://sssninedeeee index. pwpp/AAI/araaaaaaapaaappaapaaaaa wareAIaariica
6239
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro pro: Unified multimodal understanding and generation with data and model scaling, arXiv preprint 250Iv:2501. 17811, 2025.
Christopher Clark, Kenton Lee, Ming- Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2936. Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/ 019-1300) TRI.httss:.0 acl hnt oog.oogg..oooooooooo.ooooooo..oo M0wWw19111 9-1000
10

---

Published as a conference paper at ICLR 2025
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.054:: 2018.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image-text data created by the people, for the people, 2021.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smymis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang. et al. Datacomp: In search of the next generation of multimodal datasets. In Advances Neural Information Processing Systems (NeurIPS), volume 36, 2024a. htt.ss:/arxiv..rr abbs/ 2304. 1
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403. 08540, 2024b.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang. and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL http:/::://zenooooorrr zwwodo.odooooooooooooooooooo. org/
www.:/oorrrrrroorroooeoo records 112608666666 602]
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Suchin Gururangan, Mitchell Wortsman, Samir Yitzhak Gadre, Achal Dave, Maciej Kilian, Weijia Shi, Jean Mercat, Georgios Smyrnis, Gabriel Ilbarco, Matt Jordan, Reinhard Heckel, Alex Dimakis, Ali Farhadi, Vaishaal Shankar, and Ludwig Schmidt. OpenLM: a minimal but performative language modeling (Im) repository, 2023.
https: /ttp://gtttbbb com/ ml Coundations: /one/opensnnn_n__o.. open__mn
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optmml large language models. In Advances in Neural Information Processing Systems (NeurIPS). 2022.
org/ wwtpaab......e/ 15556
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and patfern recognition, pp. 6700-6709, 2019.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP,, pp. 2567-2577 Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10. 111111//1/111111111 1259.. URL https::/
org/D19-1255
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic VLMs: Investigating the Design Space of Models, February 2024. URL http::ttt/aaatt org/ /bs//4/222222 07865 arXiv:2402.07865 [cs].
Language
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Hugo I Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when build- ing lasion-language models?, May 2024. URL hett:://axxi..org/aa arXiv:2405.02246 [cs].
02246

---

Published as a conference paper at ICLR 2025
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejum Ma, and Chun- yuan Li. Llava-next-imterleavee Tackling multi-image, video, and 3d in large multimodal models. (arXiv:2407.00 007800777.7 July 2024a. URL http::/arxiv..ott org. abs/ wwwp///....... 0207/20077099 arXiv:2407.07895 [cs].
Jeffrey Li, Alex Fang, Georgios Smymiis Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Kch, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang. Khyathi Chandu, Thao Nguyen, Igor Vasiljevic. Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zeutlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang. Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave. Ludwig Schmidt, and Vaishaal Shankar. DataComp-LM: In search of the next generation of training sets for language models, June 2024b. URL http:://taatt http://aawxvv...gbbbbbb//saa//// wwt./org/////a/// 11794 arXiv:2406. 11666.1194444 1794 [cs].
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learming, pp. 19730-19742. PMLR, 2023a.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! ArXiv preprint, abs/2305.06/// 2023b. URL hattps::/axxiv..rrgggg. wbw.oorrg/bbbbbaaooooo
06161
Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang. Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, Yuyin Zhou, and Cihang Xie. What if we recaption billions of web images with Ilama-3? arXiv preprint arXiv:2406.084777 2024c.
Yifan Li, Yifan Du, Kun Zhou, Jimpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprini arXiv:2305. 10355, 2023c.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, al. Deepseek-vi: towards real-world language understanding. arXiv preprint ar 2xXX::2443.0552. 2024.
Brandon McKinzie, Zhe Gan, Jean- Phinlppi Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre-training. arXiv preprint arXiv.2403.061/ 2024.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alterna- tives: An evaluation of commonsense causal reasoning. In Association for the Advancement of Artificial Intelligence (AAAI) Spring Symposium, 2011. https:: /tppp:://pepppe... ict wsw. edu/ -gordon/ copa. html
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2556-2565, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18- 1238. URL htt.attss//acaaaatcc..c
-138
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
12

---

Published as a conference paper at ICLR 2025
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:240S.09818. 2024.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data in multimedia research. ArXiv preprint, abs/1503.01817, 2015. /ttps::////axxx
abs/ 1503.01817
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang. Shusheng Yang, Adithya Iyer, Xichen Pan, al. Cambrian-1: fully open, vision-centric exploration of multimodal Ilms. arXiv preprint 240.::6000.668000 2024.
Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel
Tuzel. Mobileclip: Fast image-text models through multi-modal reinforced training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15563- 15974, 2024.
Ke-Jyun Wang. Yun-Hsuan Liu, Hung- Ting Su, Jen-Wei Wang, Yu-Siang Wang, Winston H Hsu, and Wen-Chin Chen. Ocid-ref: A 3d robotic dataset with embodied language for clutter scene grounding. arXiv preprint wxxi:2103.07000 2021.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal 2024a.
understanding and generation. arXiv preprint ar 2rXiv:2400..3888
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang. Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li. Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseek-v12. Mixture-of-experts vision-language models for advanced multimodal understanding. 2024b. httthhtts:/aan
org/aba/ 2412. 1.e.10332. 10302
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint 2XXx:22008.08872, 2024.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 69-85. Springer, 2016.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800 Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/119/ 1472. URL
1472
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Lan- guage Image Pre-Training. In 2023 JEEE/CVE International Conference on Computer Vision (ICCV), pp. 11941-11952, Paris, France, October 2023. IEEE, ISBN 9798350307184. doi:
10. 10.11000CCV110002222001011 100. h http:tts/eeeeeelrrreet iereeeeieee..ooo. org/ ootpmmceet//////////o 103775507
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. ArXiv preprint, abs/2304.06366 2023. htLLhtts::/ http://axxvv.r..g.. httpg//ab/233000 06364
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuue images with one multi-modal model, 2024. URL hothttt/:// /t/a:///xiiii wwt.xiiivrrrr/ooooooo abs////////// //bs/220 abs/240s/// /400 019
13

---

Published as a conference paper at ICLR 2025
A TEXT-ONLY PRE-TRAINING HYPERPARAMETERS
We preformed initial exploratory experiments with 79M parameter model and scale our final experiments to 1B model with settings from Gadre et al. (2224)) given in Table 2 Our text models are trained with the OpenLM (Gururangan et al. 2023) codebase for next token prediction on the DCLM-baseline dataset (Li al. 2024b).
Table 2:
The two models and set of hyperparameters used in our experiments. Models have number of parameters N, with number of layers Maayers number of attention heads Fheadss model width dmodel. and width per attention head doead Batch sizes global and in units of sequences. Each sequence has 2,048 tokens. A A100 GPU hours are at M = 150. For the 1.4B scale, batch size of 256 performs slightly better than 512.
Mlayen
Theess
duodd
dhed
Warmup
Learning rate
Batch size
Training tokens
A100 hours
79M
8
512
128
400
3e-3
512
237B 4.3T
1.2k
24
2,048
128
5000
le-2
256
106k
B IMAGE-TEXT PRE-TRAINING HYPERPARAMETERS
We add an image encoder, loaded from pre-trained model (or randomly initialized, in the case of patch projection). We define 2-layer MLP with GELU activation to project the image patches from the image encoder dimension to dimension 4 times larger and then down to the LLM input dimension.
Image-test pre-training follows the same procedure text pre-training except that the image patches are not predicted by the auto-regressive model and are masked out of the auto-regressive cross-entropy loss.
We use the learning rate schedules defined in Figure 3
C IMAGE ENCODING IMPLEMENTATION DETAILS
We encode each image to the vocabulary space the language model, with each image taking 729 tokens. When dealing with caption data, we always place the image first before the text, and we add separator token to separate the image and text. We then mask out the image and only compute loss over the text tokens. However, we do not use a PrefixLM architecture as used in Beyer et al. (2024) and Zhou et al. (2024). We believe this is a promising direction to explore in the future.
Mcanwhile, when dealing with LLaVA data, we do similar process of adding the image before the text and masking the image. This time, we also make sure to mask out any system prompt and human question, only keeping the model responses unmasked. We also add separator token the end of every human question and model response.
D FINE-TUNING HYPERPARAMETERS
As described in Karamcheti et al. (2024). our image-text instruction tuning dataset is composed of 665K examples. Each sequence that is fed to the model begins with 729 items matching image patch embedding followed by different numbers of embedded text tokens. The model is trained using the Prismatic codebase Karamcheti al. (2024). Instruction tuning follows the same training strategy as pre-training except that the image patch and question tokens are masked out the cross-entropy loss.
We the following non-default hyperparameters:
Learning rate: 3.10
Warmup ratio: 0.05
Adam optimizer: A2 0.95
Finetune epochs: 4
14

---

Published as a conference paper at ICLR 2025
Batch size: 256
Image resize strategy: "letterbox"
E EVALUATION BENCHMARK DESCRIPTIONS
VQA benchmarks:
VQAv2 (Goyal al. 2017):
General visual reasoning
GQA (Hudson & Manning
2019): Spatial reasoning
TextVQA (Singh et
2019): Text-based reasoning
POPE Li ct al. 2023c): Yes/No hallucination test
Object localisation benchmarks:
ReiCOCO (azzemzadeh et al. 2014 Yu et al. 2016): Object localisation
OCID-Ref Waag et al. 2021): Cluttered object localisation
These subset of the tasks considered in Karamcheti al. (2024). We evaluate and report VizWiz results but leave them out of the average performance computation because its system prompt is of the training distribution and is often misunderstood by the lb model. The "unanswerable" answer suggested in the prompt is rarely used by the model, which would often answer reasonable but different responses such as empty responses or "I don't 't't know". The exact match scoring does not reflect that, causing a large variance of the results. With different initial random seeds, most results stay similar, but VizWiz results vary by several percentage points.
We also left out challenge tasks VSR, Ta and AI2D because the difficulty of these tasks make them out of scope for small scale models.
Text benchmarks: The following benchmark descriptions are taken from Li (002bb)
AGI Eval LSAT-EN (Zhong al. 2023 tests for model knowledge in the legal domain and
evaluates analytical reasoning capabilities.
ARC-easy CClark et al. 2018) contain four-way multiple choice questions taken from grade 3-9 science exams, where questions in the easy dataset require knowledge of basic science, and the challenge questions require some procedural reasoning.
BigBench (bench authors, 2023) Conceptual combinations 4 4-way multiple choice ques- tion answering dataset which discriminate combinations of objects and attributes that are appropriate to each other not.
BoolQ (Clar et al. 2019) binary question answering dataset where the model is expected to answer questions about relevant passages.
COPA (Roemmele et al. 2011) causal reasoning questions where the model given two possible outcomes to scenario and must. commonsense to select the outcome that is more likely.
conversational question answering dataset where the model is given passage and conversation between two participants and then expected to extract an answer from the passage to a question from one of the participants.
HellaSwag (Zellers et al. 2019)
MathQA (Amini al.) 2019) 5-way multiple choice question answering dataset that evalu- ates math word problem solving capabilities, built on top of AQuA.
PIQA Bisk et al. 2020) binary multiple choice question answering dataset that requires the model to physical commonsense reasoning answer correctly.
PubMedQA (Jin et al. 2019) 3-way multiple choice question answering dataset which evaluates the model's 's ability to answer biomedical research questions given context from a relevant research article.
15

---

Published as a conference paper at ICLR 2025
F DETAILED RESULTS
F.1 THE IMPACT OF TEXT-ONLY PRE-TRAINING
Figure 9 shows the detailed vision benchmarks results and global text benchmark results as the initial text-only pre-trained model is trained to completion. All tasks except VizWiz perform the best with the text backbone trained to 80% of completion.
Vision: pope
Vislon: OCIDRe
Text stable-seeee stable-score
Vision OCIDRe
stable- Rable-score
0.14
0.14
0.865
0.13
0.35
0.13
Vision
0.30
0.855
20%
Text pretraining completion ratio
Text pretraining complation ratio
Vision: RefCoco
Vision: TextVQA
Refcoco
Text stable-seeee stable-score
Vision TextvQA
Rable-score
0.14
0.14
stable-
0.50
0.13
0.13
0.45
0.40
0.12
0.35
Text pretraining completion ratio
Text pretraining completion ratio
Vision: VizWiz
Vision: gqa
0.20
0.610
Vision VizWiz
0.18
0.14
Text Text stable-score
Vision gga
0.605
0.14
stable-scom
0.16
0.13
0.13
0.14
0.11
0.590
0.12
0.585
20%
Text pretraining completion ratio
Text pretraining completion ratio
Vision: vqa-v2
Vision: z-score
0.14
Text stbble-score
Vision z-score
1.2
1.0
0.14
Text stable-score
0.13
0.13
0.6
0.4
0.11
0.11
0.2
80%
20%
Text pretraining completion ratlo
Text pretraining completion ratlo
Figure 9: Evolution of the performance of the 1b model vision benchmarks and text benchmarks
as functions of the text-only pre-training completion.
16

---

Published as a conference paper at ICLR 2025
F.2 THE IMPACT OF ADDING IMAGES BEFORE THE END OF PRE- TRAINING
For some tasks such as POPE and RefCOCO, the best performance is obtained without any image in the second stage. Most other tasks do benefit from image-text pre-training.
Vision: pope
Vision: OCIDRe
0.150
0.150
0.125
0.125
0.39
0.100
Text statsttabee
0.100
Vision
0.38
0.868
0.075
Vision
0.075
0.866
0.050
0.050
0.36
0.864
0.025
0.025
10% 20%
100%
10% Pretraining Image Ratio
Pretraining Image Ratio
Vision: RefCOCo
Vision: TextVQA
0.625
Vision Refcoco
0.150
Vision TextVOA
0.49
0.150
0.125
Text stable-sco.
0.40
0.125
0.575
0.550
0.075
0.075
0.525
0.050
0.45
0.050
0.025
0.025
196
10% 20%
100%
10%
100%
Pretraining Image Ratio
Pretraining Image Ratio
Vision: VizWiz
Vision: gqa
Vision VizWiz
0.150
0.125
Text stable-scon
gqa
0.610
0.125
0.150
0.100
0.605
0.100
stabb
0.075
0.075
0.050
0.600
0.050
0.025
1%6
10% 20%
100%
1%
5% 10%
100%
Pretraining Image Ratio
Pretraining Image Ratio
Vision: vqa-v2
Vision: 2-score
0.150
Text stable Selesccom
Vision 2-score
1.2
0.150
stabl bble-scon
0.125
1.1
0.125
0.100
1.0
0.100
Vision
0.760
0.075
0.9
0.075
0.755
0.050
0.8
0.050
0.7
0%
15
5% 10%
100%
1%
5%
10%220%
100%
Pretraining Image Ratlo
Pretraining Image Ratlo
Figure 10: Evolution of the performance of the lb model on vision benchmarks and text benchmarks functions of the ratio of image caption data the image pre-training phase.
17

---

Published as a conference paper at ICLR 2025
F.3 THE IMPACT OF FINETUNING ON VISION AND TEXT PERFORMANCE
The trend of progress over finetuning for individual benchmark only clear for some benchmarks such RefCOCO and OCID. For VQA-v2, GQA, TexIVQA, and POPE, the best results 4 epochs but less than one percentage point is lost between their maximum performance and their performance epoch 4.
Vision: pope
Vision: OCIDRe
Vision OCIDRe
Text stable stable-scrre
Vision pope
0.875
0.375
0.870
0.15
Text stable
0.350
0.15
0.865
0.325
0.14
0.300
0.860
Fine-tuning epochs RefCOco
Fine-tuning epochs
Vision:
Vision RefcOCo
0.60
0.16
Text stable-scor
Vision TextVOA
0.495
0.16
Text stable-scae stable-score
0.490
0.15
0.50
0.485
0.14
0.14
0.45
0.480
Fine-tuning epochs
Fine-tuning epochs
Vision: VizWiz
Vision:
Vision VizWiz
0.275
0.16
Text stable-score
0.16
Text stable-scon
0.250
Vision gga
0.610
0.225
0.200
0.605
0.14
0.14
0.175
0.150
0.600
Fine-tuning epochs
Fine-tuning epochs
Vision: vqa-v2
Vision: 2-score
Text stable store
1.2
Text stable-scoe sabree-score
Vision vGa-v2
Vision z-score
0.7725
1.1
0.7700
0.15
1.0
0.15
0.7675
0.8
0.14
0.7625
0.7
Fine-tuning epochs
Fine-tuning epochs
Figure 11: Evolution of the performance of the 1b model on vision benchmarks and text benchmarks functions of the number of fine-tuning epochs.
18

---

Published as a conference paper at ICLR 2025
F.4 THE IMPACT OF ADDING IMAGES FROM THE START WHEN TRAINING FROM SCRATCH
Although not by a large margin, in this pre-training from scratch experiments, best results are obtained without image in the pre-training mix.
Vision: pope
Vision: OCIDRe
0.8725
Text stable -ccoe
Vision OCIDRe
0.0700
0.10
Text stable-scon
0.8675
Vision
0.8650
0.05
0.20
0.05
0.8625
0.15
0.00
0%
1%
5% 20%
100%
0%
1%
5% 10%
100%
Pretraining Image Ratio
Pretraining Image Ratio
Vision: RefCoco
Vision: TextVQA
RefCOCO
0.45
Text staale-scooooooo
Vision TextVQA
0.42
stable-scorre
0.10
0.40
0.40
0.05
0.05
0.30
0.25
0.00
0.34
0.00
056
1%
5%
20%
100%
0%
5% 10% 20%
100%
Pretraining Image Ratio
Pretraining Image Ratio
Vision: VizWiz
Vision:
Vision VizWWiz
0.58
Text stable-score
0.10
Text stable-stsee
Vision gga
0.10
0.12
0.57
0.10
0.05
0.56
0.05
0.00
10%
Pretraining Image Ratlo
Pretraining Image Ratio
Vision: vqa-v2
Vision: z-score
Text stable -ccorc
Vision vqa-v2
Vision z-score
0.25
Text stable-score
0.10
0.00
-0.25
0.05
0.05
0.72
0.71
-0.75
0.00
05
1%
5%
10%
100%
0%
1%
5%
10%
100%
Pretraining Image Ratio
Pretraining Image Ratio
Figure 12: Evolution of the performance of the 1b model trained from scratch on vision benchmarks and text benchmarks as functions of the ratio of image caption data in the image pre- vee-rrining phase.
19

---

Published as a conference paper at ICLR 2025
F.5 THE IMPACT OF INSTRUCTION TUNING DATA DURING TRAININIG
Including instruction tuning data degrades VQA performance of the downstream model for all tested benchmarks. However, surprisingly, it improves the overall performance on downstream text tasks. We believe this is due to the instruction tuning data that help the model understanding question answering tasks.
Vision: pope
Vision: OCIDRe
0.873
Text stable-score
Vision OCIDRe
Text blabre-coore
0.17
0.17
0.872
Vision
0.16
0.16
0.15
0.15
0.869
Llava Image Ratio
Llava Image Ratio
Vision: RefCOCO
Vision: TextVQA
RefCOCo
Text stable-score
Vision TextvQA
Text stable-Scorf
0.17
0.48
0.17
0.16
0.46
0.18
0.15
0.44
0.15
Llava Image Ratio
Llava Image Ratio
Vision: VizWiz
Vision: gqa
Text stable-score
Vision VizWiz
0.15
0.610
Text stable-Score
0.17
0.17
Vision
0.16
0.606
0.16
0.13
0.604
0.12
0.15
0.602
0.15
2%
Llava Image Ratio
Llava Image Ratio
Vision: vqa-v2
Vision: z-score
Text stable-score
1.2
Vision vqa-v2
0.765
Vision z-score
Text stable-score
0.17
1.0
0.17
0.760
0.16
0.16
0.6
0.750
0.15
0.4
0.15
2%
Llava Image Ratio
Llava Image Ratio
Figure 13: Evolution of the performance of the lb model on vision benchmarks and text benchmarks
as functions of the ratio of image instruction tuning the image-text pre-training phase.
20

---

Published as a conference paper at ICLR 2025
VQAv2 performance vs Number pretraining chinchillas
Epochs
Scales
70
epochs
CX1
epochs
epochs
epochs
10-2
10
100
Percentage of image tokens
Figure 14: We train for {1,2,4) times the Chinchilla optimal number of tokens for 79M models. The above plots taken for 60% checkpoint, with VQA-v2 the y-axis. Plots from other checkpoints look similar.
G TRAINING FOR HIGHER CHINCHILLA SCALES
In Figure 14] we train a 79M model for up to 4x the Chinchilla scale, which approximately an 80x token multiplier. Overall, the trends for Cx1, Cx2, and Cx4 look quite similar. Consider, for example, the blue (epochs=l) trend lines for Cx1 (circle), Cx2 (cross), and Cx4 (star). We see that all three trend lines generally follow the same shape. The same can be said for the trend lines of the other epoch counts. We believe that this will be very interesting (albeit costly) to test at the 1B level observe what role model size plays in these scale calculations.
H LARGER MODEL SCALES (7B)
In this paper, we mostly conducted experiments on 1B scales and below. One natural question to ask whether the results will continue hold at larger scales.
Experiments at the 7B scale can be costly, especially for pre-training. For text-only pre-training, we checkpoints from DataComp-LM (Li al. 2024bb. However,
were able leverage the pre-trained DCLM-7B was set to a schedule of 4.3T tokens but trained for only 2.3T tokens then cooled down. As such, the checkpoints for the 60%, 80%, and 100% do not exist. To test if our main claim still holds true in this modified setting, we train and compare the following models:
1. Introduce images (at a 90: 10 text:image ratio) around 50% of the way through pre-training for DCLM-7B checkpoint (2.3T tokens a 4.3T token schedule). This is done at 1x Chinchilla tokens (140B tokens). Then fine-tune with LLaVA data for 4 epochs.
2. Take the final cooled-down DCLM-7B checkpoint (2.3T tokens cooled-down for 270B more tokens). For a fair tokenematched comparison, we continue training with just text at the
http:s//huggingffcccc.o////
21

---

Published as a conference paper at ICLR 2025
scale the model above (i.e. 1x Chinchilla tokens). Then fine-tune with LLaVa data for 4 epochs.
same
Here, model is basically the method that we are proposing (i.e., adding images the pre-trained checkpoint before it's fully cooled down), and model 2 is very similar to how many VLMs are conventionally trained.
Our results are shown in Table 3 below:
Dataset
Model 1 (with image data in pre-training)
Model 2 (no image data in pre-training)
VQAv2
76.39
74.69
GQA VizWiz
61.08
60.6
10.88
11.69
TextVQA
0,46828
0,41624
RefCOCO
0.64168
0,58889
OCID
0,46107
0.39696
POPE
0.87478
0.87113 13
Table Performance of 7B models on various vision-language tasks. We see that model (with images in pre-training) generally outperforms model 2 (no images in pre-training). This supports the findings that we observed the smaller scale experiments.
I
WHAT DO THE RESULTS LOOK LIKE IF WE UNFREEZE ALL THE WEIGHTS?
There has been a debate about whether or not to freeze the image encoders when training VLMs. Certain papers such Prismatic (Karamcheti et al. 2024) claim that the performance is better if the encoders are frozen, whereas other papers such as Cambrian (Tong et al. 2024) and PaliGemma (Beyer et al. 2024) claim the opposite.
We conduct a few experiments to test this ourselves. Our results on the frozen vs unfrozen were conducted 79M models. We report results in Table As the results for the unfrozen model were significantly worse across the board, we decided to stick with frozen image encoders for the rest of our experiments.
ABLATIONS
J.1 EFFECT OF IMAGE DATASET
In Table we ablate over the caption dataset use. We find that DataComp-DR is the best for both vision and text, though the gap is larger in text than in vision. This was one key reason we decided to use DataComp-DD for the majority of experiments.
Table 4: All results are taken with the mix of DataComp-DR caption data and DCLM text data and fine-tuned on 3 epochs. Across multiple settings, the unfrozen models perform worse in general. Checkpoint Frozen? Text-Image Ratio VQA-y2 score
80%
yes
5%
58.71
80% yes
30%
60.22
80% yes
50%
60.85
80%
5%
68.55
80%
no
30%
69.42
80%
50%
71.0
22

---

Published as a conference paper at ICLR 2025
Table 5: Effect of image dataset
Table 6: Effect of image encoder
Vision Sta-
Text Stable
Vision Sta-
Text Stable
Dataset Mix
ble Score
Score
Image Encoder
ble Score
Score
DataComp
0.4603
0.1
SigLIP
0.4607
0.1503
DataComp-DR
0.4607
0.1503
SigLIP + DINO
0.4696
0.1347
CC12M
0,4556
0.1298
Patch
0.1564
0.1503
ShutterStock 0.4518
0.1310
Projection
Table 7: All models above were trained from 20% checkpoint mix of DataComp-DR caption data and DCLM text data. Random Seed
Image Ratio
FT epochs
VQA-v2 score
1%
73.83
75.39
7
5%
1
75.36
5%
2
76.39
7
10%
1
75.39
10%
2
76.5
365
1%
74.46
365
1%
2
75.9
365
5%
75.39
365
5%
2
76.47
365
10%
75.55
365
10%
2
76.58
J.2 EFFECT OF IMAGE ENCODER
In Table 6 we ablate image encoders. Here, observe that SigL.P and DINO-SigLIP quite close to each other for vision, with DINO-SigLIP narrowly edging out SigLIP. However, SigLIP is better by large margin in text. We hence select SigLIP for the majority our experiments.
K RANDOM SEEDS
One question that arises when comparing our different experiments is to what extent small variation of the result is significant. To control this factor, experiment with different random seeds. The resulting variation of the result, shown in Table bbloww, representative of the variability of our pipeline. Overall, observe from Table that the variation across seeds is quite small, although the variation may get slightly larger as we go to smaller image ratios such as 1%.
L LIMITATIONS AND FUTURE WORK
While we strive to be thorough in our experiments, the space of exploration of multi-modal trans- formers is large and costly cover. Many promising directions remain be explored in this area.
Interleaved images This is something several recent papers have done (Laurençon et al. 2024] Awadalla al., 2024 McKinzie al. 2024). There are currently also several datasets available to such as Awadalla et al. (2024) and Laurençon et al. (2023). It is possible that using interleaved images will lead to new interesting results. To better handle this usage, would need to re-think our image encoding scheme reduce the sequence length of sample counting several images.
Higher token multipliers for 1B models Right now, have experiments with higher token multipliers (up to times the Chinchilla (Hotfmann et al., 2022) optimal) for 79M models (Section G). However, do not have these for 1B, they be costly run.
23

---

Published as a conference paper at ICLR 2025
would also be interesting to go beyond 4x Chinchilla, this would bring our training lot closer to full pre-training as opposed to the cooldown that we are currently doing.
Scaling laws Through our experimentation at 79M parameter scale and parameter scale, we discovered large differences in the model behavior and results through similar training pipelines. might sign that scaling laws of VLMs are more complex than those of LLMs. More experimentation across model sizes would be needed to confirm if these behavioral shifts persist across scales and data, uncovering the unique scaling laws of Vision-Language Modell (VLMs) compared Large Language Models (LLMs).
Image resolution training- This well as other tricks from PaliGemma (Beyer et al. 2024) contribute to improve performance. However, they add complexity to the model and training pipeline, departing from the mainstream VLM training recipe. While we do not study the compounding effect these choices could have on the factors that we study, we believe these to be orthogonal improvement directions.
PrefixLM - Using PrefixLM allows the model to attend bidirectionally over image patches and the task definition. This can possibly interplay with other factors such as freezing vs unfreezing the model, ultimately affecting the downstream performance after pre-training. By bringing part of the model closer to the usual VIT architecture, this could unlock some ability of the model to better capture the image input information, and potentially remove the need for a pre-trained image encoder.
24

---

Published as a conference paper at ICLR 2025
M BENCHMARK RESULTS
To provide further analysis, we detail below the result per benchmark at different stages of our training process. Most text benchmarks see a decreasing score the model is trained images and then fine-tuned for visual question answering. Interestingly, AGIEval score actually increases with image training. while PubMedQA sees sharp decrease the score.
Metric
Ours
Ours (No FT)
Base 80%
Base 100%
LLaMA 3.2 IB
Qwen 2.5 1.5B
AGIEval
32.5
27.7
19.4
28.2
21.4
63.6
ARC Basy
59.3
72.3
74.7
77.1
69.5
80.6
BigBench CC
27.2
30.1
40.8
47.6
27.2
57.3
BigBench CS
35.8
46.2
44.6
46.7
46.5
56.5
COPA
71.0
83.0
86.0
92.0
83.0
85.0
HellaSwag
58.2
66.5
69.2
72.8
65.1
67.78
MathQA
22.9
25.9
26.9
27.3
30.52
40.84
PIQA
70.5
74.4
76.6
79.1
76.0
76.22
PubMedQA
35.0
38.6
66.2
69.6
65.8
66.6
Stable-Score
15.44
21.26
25.73
29.67
23.52
35.68
Table Benchmark results different stages of the training process. "Ours" is our model trained with 10% image captioning and 90% text before 4 epochs of LLaVA fine-tuning.
To help provide a qualitative assessment of our model performance, we also produce some data samples from VQAv2 with different question types. These samples are available at Zenodo
N COMPARISON WITH PRIOR SOTA MODELS
While our primary focus in this paper is on studying training recipes, grounding our findings with a comparison to prior state-of-the-art (SOTA) visual language models (VLMs) provides valuable context and strengthens the validity of our results. Below, present comparison of our results with those reported by Prismatic VLM and PaliGemma. Although these models are larger, we were unable to find similarly sized models reporting the same set of evaluations ours. We also plan to evaluate the Qwen
Owen-VVL-2B-- VL.-BB- model and include it this comparison.
Metric
Ours
Prismatic 7B
PaliGemma 3B
TextVQA
49.05
51.78
73.2 + 0.2
RefCOCO
61.29
73.62
77.9 + 0.1
POPE
87.33
88.28
87.0
GQA
61.11
64.16
67.0 + 0.3
VQAv2
76.82
79.05
85.6 + 0.2
OCID
40.80
50.56
N/A
Stable Score (No OCID)
47.13
51.38
58.14
Stable Score (With OCID)
46.08
51.25
N/A
Table 9: Comparison of evaluation metrics between our model, Prismatie 7B, and PaliGemma 3B. "Ours" is our model trained with 10% image captioning and 90% text before 4 epochs of LLaVA fine-tuning.
25


Reviews:
[{"review_id": "31rS0i1hjr", "reviewer": "Reviewer_Z7KL", "summary": "Brief Summary: This is an experiment heavy paper tackling the question of how to best pre-train Vision-Language Models (VLMs). In particular, the authors investigate the best time to provide vision-text data to the existing LLM model, how much visual data should be provided to maintain scores on language-only tasks. \n\nThe experiments on multiple VQA datasets and text-benchmarks (noted in appendix E) suggest some key takeaways such as timing of introducing the vision data in the cooldown phase of the LLM training, the fraction of the vision data being limited compared to text data, and using instruction fine-tuning tokens for downstream tasks during fine-tuning is better.", "questions": "Q1. Is there a particular reason for choosing OpenLM instead of other models such as LLaMa and Qwen adjacent models (those in Table 1)?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Pros:\n\n1. This is an experiment heavy paper aiming to empirically provide a guideline for training with image data for image-language downstream tasks. The authors suggest they have performed over 300 runs (some on 79M some on 1B model) which is generally sufficient.\n\n2. The takeaways make sense. The improvements by following the guidelines are significant (~2% on vqa benchmarks). \n\n3. The authors provide heavy ablations, the experiments are well thought out in general.", "weaknesses": "Cons:\n\n1. The authors should provide some qualitative analysis, of where the model scoring lower misses compared to other models. Currently, it is not directly clear what kind of questions the 2% drop come from. \n\n2. One concern is that the chosen VQA benchmark is a bit restrictive. The authors should chose a larger set of evaluation benchmark such as MMMU, OCR-Bench to name a few (Qwen2-VL has a good set of possible evaluation benchmarks [Ref1]). \n\n3. Related to previous point, the authors should also include some caption/description as well as OCR heavy datasets as well. \n\n4. (Minor) The provided methodology is shown on 1B models but not on larger models such as 7B and 70B. While in general it is expected the findings will carry forward, an explicit experiment would be better. However, it might be prohibitively expensive. \n\n5. (Minor) One issue with the guidelines is that these can be only applied at training time by the authors of the original work. Thus, if someone else wants to improve an existing VLM, it requires the original authors to have open-sourced their checkpoints including the optimizers weights and schedule which may be difficult to obtain. \n \n[Ref1]: Wang, Peng, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen et al. \"Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution.\" arXiv preprint arXiv:2409.12191 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W2djmZE2MC", "reviewer": "Reviewer_7Ue5", "summary": "Authors investigate the common multi-step training procedure of VLMs, where first an LLM is trained, and then this LLM is further trained with image-text data to turn it into a VLM. They probe when and how to exactly add the image-text data into the training, and how to set the learning rate schedules. Also, they research the impact of setting different ratios of image/text tokens in the visual pretraining stage, and how to exactly use the image instruction data for the last last finetuning stage. \n\nTheir findings show that image-text data should be added during the cooldown phase of the LLM training, and not as a second step with a second warmup phase. They further find that adding instruction fine-tuning data during the pretraining stage hurts performance, and that 10-20% of tokens should be visual during the image-text pretraining stage.", "questions": "- The LLM is only trained for 28B tokens in this work, while it is originally trained for 4.3T tokens in the DCLM-1B work, a factor 150x more data. Authors argue with the scaling laws from the Chinchilla paper, which would suggest “20 times number of params” of tokens is enough. But then, why use a LLM that is trained on “3070 times number of params” tokens?\n\n- How many (approximate) GPU hours were used for each experiment, or at least, how expensive is one run of the 1.4B model on 28B tokens? Appendix A shows GPU hours for the text-only training but not for the image-text training.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "The paper is very well presented visually and also very well written, as it is fluent to read and understandable. This combination of nice figures and well written text makes it well digestable. The authors highlight potential weaknesses or difficulties, e.g. in figure 4 highlighting the points that are trained with a different learning rate schedule.\n\nThe research question is interesting and valuable, since most VLM pretraining happens in (semi-)proprietary setups where details are hidden, or in setups were many training choices are not discussed since ablating them would be difficult.\n\nThe finding that image-text data should be injected earlier into the training process is relevant for many existing VLM training pipelines.", "weaknesses": "Note that these weaknesses are in descending order of importance.\n\n1) The comparison to other VLMs is missing: The authors introduce a new metric “vision stable-score” as their main empirical evidence, but then evaluate only their own trained models on this metric. This makes comparing this work to other works difficult. Authors should show the benchmark numbers for some of the models in table 1. This work does not need to be top 1 in this table since the goal is not to train a SOTA model but to investigate training behaviour. However the comparison table is needed to evaluate how close this work is to the SOTA VLMs, because if the model is too far away, the findings might not relevant for SOTA models.\n\n2) Similarly, the text stable score is very low, below 20% everywhere. This might be due to subtracting the random baseline. However there needs to be a comparison to the DCLM-1B text model before any image training, as well as maybe one more SOTA/popular LLM.\n\n3) In 3.1 the peak learning rate is set to 3e-3 for the 100% checkpoint model. This choice is not argued for or ablated. The best peak learning rate in this chapter seems to be the green cosine 80% curve in figure 3 (around 1e-4 however it is hard to read the exact number from the graph), therefore authors should have at least tried this best peak learning rate also for their 100% checkpoint model. Otherwise, the risk remains that the main finding of the paper in chapter 3.1 can be just attributed to different hyperparameters.\n\n4) Evaluating only on accuracy metric benchmarks has significant drawbacks, since only short and exactly matching answers can be correct. Considering that this model is trained on image-caption pairs, there should be at least a captioning benchmark added.\n\n5) In the abstract: “a 1B model (retains) performance within 2% of our text-only baseline”. I could not easily find this in the paper. Assuming you compare the 80% model from figure 4 with ~15% text stable score and ~54% text arc easy. The text-only baseline should then be the DCLM-1B model without any image-text training and without image-text finetuning. However that model reports ~75% text arc easy results. So: Which models are you comparing to make this statement, on which metrics? If the text-only baseline is not the DCLM-1B model, why?\n\n6) The decision to freeze the vision encoder was done on a 79M model. However there is no argument why this should hold for bigger models, it would be good to have at least one unfrozen run on the 1.4B model.\n\n7) Training on interleaved image-text data is omitted in this work. This is a weakness acknowledged by the authors but it is reasonable to first evaluate the findings on image-text pairs and potentially in the future extend to interleaved data.\n\nSummary: I believe that points 1 to 3 must be addressed before the paper can be published, because otherwise the scientific evidence for the findings is not strong enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t87ZGq6v7E", "reviewer": "Reviewer_ans3", "summary": "This paper studied when to integrate image tokens to the training process of LVLM. The authors trained models spanning various datasets, scales, image-text ratios, and amount of pre-training done before introducing vision tokens. It experimentally found that incorporating image data during pre-training generally helps and the fraction of visual data and time of introducing visual instruction tuning are also important.", "questions": "1. This is a clarification question: When incorporate image at a early time, does that mean more visual data is trained or seen by model? \n2. I think the name of \"LLaVa\" should be \"LLaVA\".", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper systematically studies the time of introducing visual tokens during language pre-training, image-text pre-training and instruction finetuning. \n2. It demonstrates several key findings such as incorporating image data during pre-training, and the time of introducing instruction fine-tuning image tokens. \n3. Compared with fully pre-trained model, introducing visual tokens 80% of the way through pre-training results in a 2% average improvement in VQA tasks.", "weaknesses": "1. This study is very limited to the parameter size of the LLM. It seems hard to generalize to different size of LLMs. It seems this paper adopted 1B LLM, commonly used 7B and 13B LLM should also be studied. How do authors expect the findings might change at different model scales? \n2. Some findings are already well-studied, such as mixing together instruction fine-tuning during the image-text pre-training can hurt the model. \n3. Current LVLM starts with a fully pre-trained LLM. But it seems not many strong LLMs can release their weight at 80%. What are the practical implications of this paper's' findings given the limited availability of partially trained LLM checkpoints? Could the authors explore ways to approximate the approach using fully trained models? \n4. Would like to see the final model trained by this paper compared with existing LVLMs such as mentioned in paper's Table 1 or well-recognized models such as LLaVA-Next, LLaVA-One-Vision etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vmJtkivN0F", "reviewer": "Reviewer_sMqo", "summary": "This paper investigates the effect of image-text pretraining in large vision-language models using a one-stage approach, in contrast to the typical two-stage method where a text-only language model is adapted into a vision-language model. The study examines various factors during pretraining, such as the amount of text-only pretraining data, and the quantity, type, and ratio of image pretraining data. Specifically, after text-only training, the authors investigate the impact of when to start image-text training, the ratio of image-text pairs when performing image-text training, the impact of instruction-tuning data, and the finetuning epochs on image instruction tuning. After each factor is implemented, the model is followed by image instruction-based tuning, and each factor is then evaluated on VQA and text-only QA tasks. Experiments are conducted on a 1B model, with 28B tokens following the scaling law of (parameters x 20). Interesting findings in multimodal pretraining are deduced.", "questions": "My biggest problem is W1, and im afraid no general conclusions can be drawn. If time permits, I would like that the authors verify their insights on at least another task (e.g., captioning, VQA-X, e-SNLI-VE). I will increase my score if the authors address this concern. W2 is also concerning.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Generally, many strong insights that can advance the research of vision-language pretraining and Multimodal LLMs are deduced. \n\n- I find that the insight of continued training at 80% preferable to training from the 100% pretrained text model, with a 90-10 image-text ratio, very interesting and will likely have a good impact on the community. \n\n- The training from scratch insight (3.3) is also very interesting. This will highlight the importance of having a strong text-only model before interfering images. Interestingly, many old (2020-2022 century) vision-language models such as OSCAR, UNITER...etc, were following this \"from-scratch\" paradigm. \n\n- I also find the insight on image instruction finetuning to also be degrading text-only task performance (3.5), interesting. This allows the community to deduce that when multimodal LLMs are instructed with text-only, they dont perform at their best. This could be helpful for systems to select the non-finetuned model when the prompt does not include an image.", "weaknesses": "- [W1] My main problem is that these insights are only demonstrated on QA and VQA tasks, which I dont find enough for deducing generalizable insights. Good VQA performance does not necessarily reflect general performance. There are a suit of benchmarks (e.g., those used in LLava) used to evaluate the vision-language models. At least include other vision-language tasks such as image captioning and its variations? Visual entailment? VQA with Textual Explanations (VQA-X) and visual entailment with explanations (e-SNLI-VE) can also be good dataset for testing. \n\n- [W2] Something in Figure 5 caught my attention. When the image ratio is 0% (as if no image data is added, and we continue training on text-only data, as in a normal text-only LLM), the VQA performance is already around 75%, even though the model has seen no images. This makes me wonder that the model is answering questions without really looking at the image (this is a well-known bias problem in VQA models, presented in [R1]). Therefore, i am afraid these results may reflect biased models. I wonder what the results are without using any images? What about other VQA reasoning datasets such as A-OK-VQA? \n\n- [W3] The authors mention in L739 in the appendix: “This time, we also make sure to mask out any system prompt and human question, only keeping the model responses unmasked.” If no question is there in the image instruction tuning stage, then this is not *instruction* finetuning (this is just a different dataset of image-text pretraining, similar form to the DataCompDR-1B). What is the point of conducting *instruction* tuning then? Im not sure exactly what is the rationale behind masking out the question from the instruction? \n\nOther minor points (do not affect my decision):\n\n- How is the image-text training on DataCompDR-1B dataset done? Is it to generate the caption autoregressively? Is it via Masked VL modelling? The authors dont mention this. I understand from section C that the authors dont use prefixLM (no image-image attention), and only casual modeling and text-image attention. Therefore, I assume that they predict the caption autoregressively? If so, please mention this, as this is not a trivial point, since many works in the literature use different VL pretraining objectives. \n\n- In 3.4, when x% of image instruction tuning data is added during the image-caption training, is the same data repeated during the image instruction finetuning stage? \n\n- L182-187, I feel this should go to the related work. The authors describe other works here.\n\n\n[R1] RUBi: Reducing Unimodal Biases for Visual Question Answering", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)