What matters when building vision-language models?
Authors: Hugo Laurençon
Leo Tronchon
Matthieu Cord
Victor Sanh
Venue: NeurIPS 2024 poster (2024)
Keywords: vision-language model
multimodal
Primary area: machine_vision

Abstract: The growing interest in vision-language models (VLMs) has been driven by improvements in large language models and vision transformers. Despite the abundance of literature on this subject, we observe that critical decisions regarding the design of VLMs are often not justified. We argue that these unsupported decisions impede progress in the field by making it difficult to identify which choices improve model performance. To address this issue, we conduct extensive experiments around pre-trained models, architecture choice, data, and training methods. Our consolidation of findings includes the development of Idefics2, an efficient foundational VLM of 8 billion parameters. Idefics2 achieves state-of-the-art performance within its size category across various multimodal benchmarks, and is often on par with models four times its size. We release the model (base, instructed, and chat) along with the datasets created for its training.

What matters when building vision-language models?
Hugo Laurençon":l4
Léo Tronchon":
Matthieu Cord
Victor Sanh
"Hugging Face Sorbonne Université valleo.ai, Paris, France "The order was chosen randomly.
Abstract
The growing interest in vision-language models (VLMs) has been driven by im- provements large language models and vision transformers. Despite the abun- dance of literature on this subject, we observe that critical decisions regarding the design VLMs are often not justified. We argue that these unsupported decisions impede progress in the field by making it difficult to identify which choices im- prove model performance. To address this issue, we conduct extensive experiments around pre-trained models, architecture choice, data, and training methods. Our consolidation of findings includes the development of Idefics2, an efficient founda- tional VLM of 8 billion parameters. Idefics2 achieves state-of-the-art performance within size category across various multimodal benchmarks, and often on par with models four times its size. We release the model (base, instructed, and chat) along with the datasets created for its training.
Prompt
Idefics2 output
expense was 2014, and the long-term is 10% higher than its 2015 level. Can you calculate the combined total of the interest and long-term debt for 2024?
Yes, help you calculate that, First, let's out interest expense for 2024:
Interest Expense (2024) = Interest Expense (2014) 2 630
1260
2014
2015
find the
debt 2024:
Long-Tern Debt (2024) Long-Term Debt (2015) 10% of Long-Term Debt (2015)
Sales
$ 9,102
$10,091
Depreciation
1,350
1,351
19 19,260 0.1 19.260
Cost of goods sold
19.260 1,926
Other expernes
767
611
21.188
Interest
630
721
Now, add these two amounts together to get combined
Cash
4931
6,211
interest long-term debt
Accounts recoivable
6,527
7,352
Short term notes payable
953
895
Combined Total (2024) Interest Expense (2024) Long-Term Debt (2024)
debt
16,152
19,260
Net
41,346
1260 21,196
Accounts payable
5,179
5,072
22.446
Inventory
11,604
11,976
the combined total the interest and long
Dividends
1,147
1.261
for $22.446.
Figure 1: Idefics2-chatty analyzes the table compute and answer the query.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Vision-language models (VLMs) that take images and texts as imputs and output texts, are useful for many tasks, like retrieving information in a scanned PDF (Hu et al. (024). explaining charts or diagrams (Carbune al., 2 transcribing the text in an image (Blecher al., 2023), counting objects in a picture (Goyal et al. 2017) or turning screenshots of webpages into code (Laurençon
al. 2024). The development of powerful open large language models (Touvron et al., 1023:Jiang
et al. 2023 TTam et al., 2024) and image encoders (Zhai et al. [023] Sun et al., 2023] Radford et al.
2021) enables researchers to build upon these unimodal pre-trained models to create advanced VLMs
2023 Bai et al. 2023 Lin et al. 2023] [i et al., 2023: Wang et al., 2023). Despite the progress in the field, the literature
that solve these problems with increasing accuracy (Dai et al. [023 Liu et al.
reveals many disparate design choices which are often not justified experimentally, or very briefly.
This situation makes challenging to distinguish which decisions truly account for model perfor- mance, thereby making it difficult for the community to make meaningful and grounded progress. For instance, (Alayrac et al., 2022 Laurençon et al., 2023) use interleaved Transformer -eroooeeer---dddddde cross- attentions to fuse the image information into the language model, while (Li et al.) 2023] Liu et al.
2023) concatenate the sequence of image hidden states with the sequence of text embeddings, and feed the concatenated sequence to the language model. To our knowledge, this choice has not been properly ablated, and trade-offs in terms of compute, data efficiency and performance are poorly understood. In this work, we aim to bring experimental clarity to some of these core design choices and pose the question: What matters when building vision-language models?
We identify two areas where various works adopt different design choices: (a) model architecture, and in particular, connector modules that fuse the vision and text modalities and their impact on inference efficiency, (b) multimodal training procedure and its impact on training stability. For each of these areas, we rigorously compare different design choices in a controlled environment and extract experimental findings. Notably, we find that (a) the progress of vision language models is in large part driven by the progress of pre-trained unimodal backbones, (b) the more recent fully autoregressive architecture outperforms the cross-attention architecture, although it requires modifications to the optimization procedure to ensure stable training, (c) adaptation of the pre-trained vision backhone and the modules connecting the text and vision modalities allow for more efficiency at inference time on one side, and handling images in their original ratio and size without harming downstream performance on the other side, and (d) modifications to the image processing enables trading inference cost for downstream performance.
Our results are complementary with those presented in (Karamcheti et al. 2024 McKinzie et al., 2024, Lin al. 2023) which derive insights about multi-stage training, selective unfreezing of the pre- trained backbones. data repetition, and impact of training mixture on zero and few-shot performance. We specifically analyze unexplored aspects such as model architecture, training methods, stability, and efficiency improvements at inference. For example, we introduce the task of text transcription from PDFs directly the pre-training, we justify the benefits of using synthetic captions in image-text pair datasets, we detail our multi-stage training procedure to save computational resources, and we create a large-scale instruction fine-tuning dataset.
Learning from these insights, we train Idefics2, a foundational VLM with 8 billion parameters. 1defies2 achieves tate-of-the-ar performance in its size category on various benchmarks while being more efficient at inference, for both the base and the fine-tuned version. It is on par with state-of- the-art models times larger on some vision-language benchmarks and matches the performance of Gemini 1.5 Pro some challenging benchmarks. We release the base, instructed, and chat versions of Idefics? as resources for the VLM community along with the data created to train the model.
2 Terminology
We first establish shared terminology for discussing the different design choices. Training VL.MS typically requires gluing together a pre-trained vision backbone and a pre-trained language backbone by initializing new parameters to connect the two modalities. Training these new parameters is done during the pre-training phase. This stage commonly leverages a large multimodal dataset such as image-caption pairs. We note that even though it is most common to start from two separate

---

unimodal pre-trained backbones, the parameters of these two backbones can be optionally shared and initialized from scratch done in (Bavishi et al. 2023). As the large language models literature. the pre-training stage is followed by an instruction fine-tuning stage, in which the model learns from task-oriented samples.
Recent works explore two main choices to combine the visual inputs and the text inputs. In the cross-attention architecture (Alayrac al. 2022 Laurençon et al 2023 Awadalla et al. 2023). the images encoded through the vision backbone are injected at different layers within the language model by interleaving rross-attention blocks in which the text cross- cross-attents attends to the image hidden states. In contrast, in the fully autoregressive architecture (Koh et al. 20233 Driess et al. 2023 Liu et al. 2023). the output of the vision encoder directly concatenated to the sequence of text embeddings, and the entire sequence is passed as input to the language model. The input sequence of the language model is thus the concatenation of visual tokens and text tokens. The sequence of visual tokens can be optionally pooled into shorter sequence, providing more compute efficiency. We refer to the layers that maps the vision hidden space to the text hidden space as modality projection layers. Figure highlights the aully-autoregressive architecture we ultimately use for Idefics2.
Up to 980x980
Hidden states
64
Modality projection Pooling
Vision encoder
LLM
TT
Figure 2: 1defics2 fully-autoreegressive architecture: Input images are processed by the Vision encoder. The resulting visual features are mapped (and optionally pooled) to the LLM input space to get the visual tokens (64 in standard configuration). They concatenated (and potentially interleaved) with the input sequence of text embeddings (green and red column). The concatenated sequence is fed to the language model (LLM), which predicts the text tokens output.
3 Exploring the design space of vision-language models
In this section, we compare recurrent design choices the vision-language model literature and highlight findings. Unless specified otherwise, the average score of the 4-shot performance on downstream benchmarks measuring different capabilities: VQAv2 (Goyal al. 2017) for general visual question answering, TextVQA (Singh et al. (019)) for OCR abilities, OK VKA (Marino et al. 2019) for external knowledge, and COCO (Lin et (2014) for captioning. We the ablations eight nodes containing eight H100s each, for up to five days.
the ablations for 6'000 steps and report
3.1 Are all pre-trained backbones equivalent for VLMs?
Most recent VLMs start from pre-trained unimodal backbones. How does the choice of the backbones (vision and text) influence the performance of the resulting VLM?
We fix the size of the pretrained backbones, the data used for multi- modal pre-training and the number of training updates. Under the cross-attention architecture, we observe that the greatest improve- ment in the performance on vision-language benchmarks comes from changing the language model to better one. More specif- ically, replacing L--M--1-77B (Touvron et al.) 2023) (35.1 on MMLU (Hendrycks al. (02I)) by Mistral-7B Jiang et al. 2023
LM backbone
Avg.
Llama-1-77
62.5
Mistral-7B 67.6
Table 1: Ablation on the language model backbone.

---

(60.1% on MMLU) yields boost of 5.1 (see T.ble Additionally, switching the vision encoder
from CLIP- --T- (Radford al. 202T) (78.0% on Dmage et al. 2009)) to SigLIP-S0400X 2023 (83.2% on ImageNet) yields a 3.3 increase in performance on the benchmarks (see
(Zhai et al.
Table 2 2024).
This result on better vision backbones corroborates observations from (Karamcheti et al.
We note Chen and Wang (2022) reports stronger increase in performance by scaling the size of the vision encoder com- pared to scaling the size of the language model even though scaling the vision encoder leads to smaller parameter count in- crease. Although EVA-CLIP-55 (Sun
VE backbone
Res.
Avg. score
CLIP-ViT-H
224
57.4
EVA-CLIP-SB
224
60.2
2023) is ten times bigger in parameter counts than Sig S0000M (Zhai et al. 2023), we obtain similar performance across benchmarks, suggesting that EVA-CLIP-55 could be heavily under-trained, and we acknowledge that the open VLM community is missing large well-trained vision encoder.
SigLIP-
384
60.7
Table 2: Ablation on the vision encoder backbone.
Finding 1. Better pre-trained LLMs and vision encoders lead to better performance on multimodal tasks. However, with the best current models for both, the LLM has a higher impact.
3.2 How does the fully autoregressive architecture compare to the cross-attention architecture?
To our knowledge, there is no proper comparison between the fully autoregressive and the cross- attention architecture. We aim to fill this gap by considering their trade-offs, namely performance, parameter count, and inference cost.
Following (Alayrac et al. 2022), first compare the two architectures by freezing the uni- modal backbones and training only the newly initialized parameters (cross-attention on one side, and modality pro- jection along with learned pooling on the other side), while fixing the amount of training data. Alayrac et al. (2022) shows that the more frequently the cross-attention blocks are interleaved with the language model layers, and the higher the vision-language performance. As such, we note that under this setup, the cross-attention architecture has 1.6B more trainable parameters (2.3B trainable parameters in total) than the fully autoregressive architecture. Additionally, at inference time, the former uses 10% more flops than the latter. Under these conditions, observe that the cross-attention architecture performs points better in Table
Backbones training
trainable params
Architecture
params
Avg.
Fully autoreg. no Perceiver
Frozen
7.6B
5M
51.8
Fully autoreg.
Frozen
8.3B
740M
60.3
Cross-attention
Frozen
2.3B
66.7
Cross-attention
LoRA
10B
2.5B
67.3
Fully autoreg.
LoRA
8.3B
950M
69.5
Tahle Ablation for the architecture method of training.
Out of the total number of parameters, approximately 10% for the fully autoregressive architecture and 25% for the cross-attentio are trained. We hypothesize that this low proportion limits the expressivity of the training and hinders performance. To test that hypothesis, we compare the two architectures by unfreezing parameters (newly initialized parameters and parameters of pre-trained unimodal backbones). Under these conditions, training the fully autoregressive architecture would yield loss divergences, and we were not successful in stabilizing the training even by aggressively lowering the learning rate gradually unfreezing various components. To overcome this stability challenge, we leverage Low-Rank Adaptation (Hu et al. 2022) to adapt the pre-trained parameters while using standard full fine-tuning for the newly initialized ones.
This setup yields significantly more stable trainings, and more importantly, we observe 12.9 points increase under the fully autoregressive architecture, and 0.6 point under the cross-attention architecture. While the cross-attention architecture performs better than the fully autoregressive architecture with frozen backbones, it is worse when we add degrees of liberty for the pre-trained backbones. Besides, using LoRA allows training the unimodal backbones a fraction of the GPU memory cost of full fine-tuning, and LoRA layers can be merged back into the original linear layers

---

yielding no additional cost at inference. We therefore choose the fully autoregressive architecture in the rest of this work.
is interesting to note that this finding contradicts (Karamcheti et al. 2024) in which the authors observed that unfreezing the pre-trained visual backhone would significantly degrade the performance. We hypothesize that using parameter-efficien fine-tuning methods is key difference.
The cross cross-attention architecture performs better than the fully autoregressive one when unimodal pre-trained backbones kept frozen. However, when training the unimodal backbones, the fully autoregressive architecture outperforms the cross- cross-attention one, even though the latter has parameters.
Finding 2.
Finding 3. Unfreezing the pre-trained backbones under the fully autoregressive architecture can lead to training divergences. Leveraging LoRA still adds expressivity to the training and stabilizes it.
3.3 Where are the efficiency gains?
Number of visual tokens Recent VLMMs typically route the entire sequence of the vision encoder's hidden states directly into the modality projection layer, which subsequently inputs into the language model, with no pooling. This is motivated by previous works which adding pooling strategy. like average pooling, was found to deteriorate the performance (Vallaeys et al. 2024). This results a high number visual tokens for each image ranging from 576 for DeepSeek- VL (Lu al.) 2024) to 2890 for SPHINX-24 (Lin al. 2023). With the resulting sequence lengths, training is computationally costly, and in-context learning with interleaved images and texts is challenging because it requires modifications to the language models to handle very large context windows.
We reduce the sequence length of each image's hidden states by using a perceiver resampler (Jaegle et al. 202T as form of trainable Transformer-bases pooling. The number of queries (also referred to as latents) corresponds to the number of resulting visual tokens after the pooling (from one for classification Touvron et al. (202T) to several in |laarac et al. (2022) or growing number for continual learning Douillard al. (2022). ) We observe that the learned pooling is effective in two ways: it increases the performance by 8.5 points on average and reduces the number of visual tokens necessary for each image from
729 to 64 (see TableB)
In contrast to (Vallaess et al. 2024 McKinzie et al., 2024 which find that the more visual tokens the higher the perfor- mance, we observe no gains when using more than 64 visual tokens. Other variations over the Perceiver architecture (Mañas et al. 2023 Darcet et al. 2024 decreased performance.
Pooling
# vis. tok.
Avg. score
Perceiver
Perceiver
64
71.7
Vallaeys et al. 2024 resulted
Table 4: Ablation on pooling strategy.
Reducing the number of visual tokens with learned pooling significantly improves compute efficiency at training and inference while improving performance on non-OCR downstream tasks.
Finding 4.
Preserving the original aspect ratio and image resolution Vision encoders, such as SigLIP, are typically trained on fixed-size square images. Resizing images alters their original aspect ratio, which problematic, for instance, for tasks requiring reading long texts. Furthermore, conditioning the training single resolution size inherently introduces limitations: low resolution omits crucial visual details, while a high resolution leads to inefficiency training and inference. Allowing the model to encode images at various resolutions allows users to decide how much compute is spent each image.
Following Lee et al. (2023): Dehghani et al. (2023), we pass image patches to the vision encoder without resizing the image modifying its aspect ratio. Given that SigLIP was trained on fixed-size low-resolution square images, we interpolate the pre-trained positional embeddings to allow for
Images
Res.
Avg. score
Square images
768
73.1
AR preserving
378-768
72.1
Table 5: Ablation on the aspect-ratio preserv- ing strategy.
5

---

higher resolution and train the vision encoder with LoRA parameters adapt to these modifications
Our findings indicate that the aspect ratio preserving strategy maintains performance levels on downstream tasks while unlocking computational flexibility during both training and inference (see T. In particular, not having to resize images to the same high resolution allows for saving GPU memory and handling images at the resolution they require.
Finding 5.
Adapting vision encoder pre-trained on fixed-size square images to preserve images' original aspect ratio and resolution does not degrade performance while speeding up training and inference and reducing memory.
3.4 How can one trade compute for performance?
LLin et al. 2023 et al. 2023 Liu et al., 2024 McKinzie et al., 2024) show that splitting an image into sub-images allows boosting the downstream performance with no changes the model's signature. An image is decomposed into sub-images (for instance 4 equal sub-images), which are then concatenated to the original image to form a sequence of 5 images. Additionally, the sub-images are resized to the original image's size. This strategy however comes at the cost of much higher number of tokens to encode the images.
We adopt this strategy during the instruction fine-tuning stage. Each single image becomes a list of 5 images: crops and the original image. This way, at inference, the model is able to deal with standalone images (64 visual tokens per image), as well as artificially augmented images (320 visual tokens in total per image). We notice that this strategy is particularly useful for benchmarks like VeX and VOc, which require a sufficiently high resolution to extract the text in an image (see Table
Moreover, when we apply image splitting to only 50% of the training samples (instead of 100% the samples), observe that does not impair the performance increase that image splitting provides. Surprisingly, we find at evaluation time that increasing the resolution the sub-images (and the standalone image) provides only minor boost in performance compared to the improvement yielded by sole image splitting: 73.6% when increasing the resolution of the sub-images to the maximum 73.0% accuracy
validation set of TextVQA, and respectively 72.7 vs 72.9 ANLS on the validation set of DoeVQA.
Finding 6. Splitting images into sub- sub-images during training allow trading compute efficiency for more performance during inference. The increase in performance is particularly noticeable in tasks involving reading text an image.
Idefics2 an open state-of-the-art vision-language foundation model
With these learnings in hand, train an open 8B parameters vision-language model: Idefics2. This section describes the construction of the model, the choice of the dataset, the sequence of training phases and compares the resulting model against VLMs baselines.
4.1 Multi-stage pre-training
We start from SigLIP-S0400X, and Mistral-7B-vO.1 and pre-train Idefics2 on 3 ttpes of data.
Interleaved image-text documents We use OBELICS (Laurençon et al. 2023). 3) open web-scale dataset of interleaved image-text documents with 350 million images and 115 billion text tokens. As shown by the authors, the long documents of OBELICS allow for preserving the performance of the language model while learning to deal with an arbitrary number interleaved images and texts and long context. Additionally, the authors show that interleaved image-tett documents are the biggest driving factor in increasing the performance on visual question answering (VQA) tasks, in particular
"Since SigLIP is trained with a fixed resolution, the positional embeddings can be interpreted both as absolute or relative positions. With the aspect ratio and resolution preserving, these positions become relative positional embeddings.

---

in the in-context learning setup. We perform an additional removal of newly opted-out content in January 2024 using the Spawning AP even though OBELICS had already been filtered to exclude opted-out content as of September 2023. We also removed the 5% of documents with the highest perplexity scores, as computed by Falcon- (Penedo et al. 2023).
Image-text pairs Training on image-text pairs allows the model to learn the alignment between images and their associated texts. We use a combination of high-quality human-annotated image-text pairs from
Captions
Avg. score
Alt-texts Synthetic 52.9
49.8
PMD (Singh et al., 2022) and higher-noise web-scale image-text pairs from (Schuhmann et al.j (022). To limit the amount of poor poor-quality data, we opt for the synthetic captions obtained through the LAION COCO version of the dataset where images have been captioned with model trained on COCO. This improves the quality of the training samples and thus the quality of the resulting model (see Table 6). We use NSFW classifie Twth a high recall and remove 7% of the samples in LAION COCO. We manually inspect 5'000 examples and found 28 pornographic images in the original LAION COCO and only I after filtering. This filtering does not negatively impact the downstream performance.
Table 6: Ablation on synthetic captions against alt-text for image-text pairs.
PDF documents
Sun et al. (2023) shows that a large pro- portion of mistakes of state-of-the art VLMs stem from their
OCR data
Res.
Doc
failure to accurately extract text in images or documents. In
Wlo
384
22.6
order to obtain strong OCR and document understanding abil- ities, we train Idefics2 on different sources of PDF documents: 19 million industry documents from OCR-IDI. (Biten et al.) 2022) and 18 million pages from PDFA Moreover, we add Rendered Tex{" to complement the dataset with texts writ- ten with wide variety of fonts and colors and on diverse backgrounds. These integrations significantly boost the per- formance on benchmarks that require reading text without decreasing the performance on other benchmarks (see Table 7
Wio
768
42.9
W/
768
49.9
Table 7: Ablation on the synergy be-
tween OCR data and image resolution. We pre-trained the models for 5'500 steps, followed by 500 steps of fine- tuning on DoeVQA.
To maximize compute efficiency. we decompose the pre-training in two stages. In the first stage, we limit the max image resolution to 384 pixels, which allows us to use a large global batch size of 2'048 (17k images and 2.5M text tokens on average). We sample OBELICS for 70% of the examples with a maximum sequence length of 2' 2'048, and the image-text pairs datasets for 30% of the examples with maximum sequence length of 1'536. In the second stage, we introduce PDF documents. Since they require a higher image resolution for the text to be legible, we increase the resolution to a maximum of 980 pixels. We use the same global batch size, but have to decrease the per-device batch size and use gradient accumulation to compensate for the additional memory cost. OBELICS represents 45% of the examples with maximum sequence length of 2'048, image-text pairs represent 35% of the examples with a maximum sequence length of 1'536, and PDF documents represent the remaining 20% of the examples with a maximum sequence length of 1'024. Additionally, we randomly scale up images to adequately cover the distribution of potential image sizes. We emphasize that the training stages are different than the ones ablated in (Karamcheti al. 2024): instead of selectively freezing/unfreezinn parts of the model, we train the entire model during both stages (some parameters are trained with LoRA) and increase the image resolution from one stage the other.
We use a learning rate of 10-4 with AdamW for the optimizer, and do around 2 epochs on our training data. It corresponds to approximately 1.5 billion images and 225 billion text tokens. We note that this is orders of magnitude more training data than other open VLMs. For example. ShareGPT (Chen et al. 2023) uses 1.2 million images, while Monkey (Li et al. 2023) uses 1.4 million for training. In total, we use 32 nodes of eight H100s each for 3 weeks for the multi-stage pre-training.
To evaluate the base model, we consider VQAv2 (Goyal et al. 2017), TextVQA (Singh et al. 2019 OKVQA (Marino et al., 2019), and COCO (Lin et al.) 2014). Table presents the results. While having fewer tokens per image, and thus being more efficient, Idefics2-base performs favorably
https://spawmingg.a
ltape://laioo.aa//llg/llaaoo
http:///hgggaattecco//aaassssstr eng wds
7

---

tokens per image
Model
Size
Archi.
VQAv2
TextVQA
OKVQA
COCO
OpenFlamingo
9B
CA
54.8
29.1
41.1
96.3
I
9B
CA
56.4
27.5
47.7
97.0
Flamingo
9B
CA
58.0
33.6
50.0
99.0
MM1
7B
FA
144
63.6
46.3
51.4
116.3
Idefics2-base
8B
FA
64
70.3
57.9
54.6
116.0
Table 8: Performance of Idefics2-base against state-of-the-ar base VLMs. The evaluations were done with random in-context examples, and in an open-ended setting for VQA tasks.
FA: fully autoregressive architecture. CA: cross-attention architecture.
(Task, Metric, Split): (VQAv2, VQA acc., testdev), (TextVQA, VQA acc., val), (OKVQA, VQA acc., val), (COCO, CIDEr, test)
compared to the other current best base VLMs (0 (OpenFlamingo (Awadalla al.) (023), Idefics
(Laurençon et al., 2023). Flamingo (Alayrac et al. 2022) and MM1 (McKinzie et al. (024)). is notably much better reading texts in image. Figure shows example of output from the base model on task similar to the pre-training.
Prompt:
<image> The content of the letter is:kn
tellgge
Dea Chan
hett dettlaadig Soo ii witer the in apod homent and stily
Completion:
Dear Craig,
Actually, I've been big letter writing, though nice receive hand written note from someone. am afraid that the drawing. and the related ability sketch idea, also being lost. But perhaps, the cliche goes, what goes around comes around. Good luck with your project
des the have and for partage po, old caees wonke palegge yot Thabinss
Tom Cianfichi
Figure 3: An example of text transcription with Idefics2-base.
4.2 Instruction fine-tuning
We continue the training with an instruction fine-tuning phase.
To do so, we create and release The Cauldror
massive collection of 50 vision-languass datasets, covering wide range of tasks: general visual question answering, counting, captioning, tran- scription, document understanding, chart/figure understanding, table understanding, visual reasoning, geometry, spotting differences between images or converting a screenshot a functional code. Sim- ilarly to (Sanh et al. 2022) Wei et al. 2022: Bach et al. [202 Dai et al.) 2023 Li et al. (202)). each dataset prompted into shared question/an /nnwer format. When there are multiple question/answer pairs image, we concatenate the pairs into multi-turn conversation. We deduplicate the training set against the evaluation sets, ensuring that there is minimum contamination from the training the evaluation.
B
In addition these vision-language datasets and following insights from (McKinzie et al. 2024). add text-only instruction datasets to the mixture. The datasets aim at teaching the model to follow complex instructions, solve mathematical problems, do arithmetic calculations. We give more details about the chosen datasets, the number of images, question-answer pairs, and size of each of the subsets, as well as our selected mixture proportion in Table 14 Appendix A.2.1
We instruction-tun the base model using DoRA (Liu et al. 2024) (a variant LoRA). During the fine-tuning, only compute the loss the tokens of the answers in the Q/A pairs. Since doing many epochs over some of the datasets, we employ several strategies to lower the risk of overfitting. First, add noise to the embeddings with the NEFTune (Jain at. 2024 technique.

---

# tokens per image
Model
Size
MMMU
Math Vista
TextVQA
MMBench
LLaVA-NeXT
13B
2880
36.2/-
35.3
67.1
70.0
DeepSeek-VL
7B
576
36.6/-
36.1
64.4
73.2
MM1-Chat
7B
720
37.0/35.6
35.9
72.8
72.3
Idefics2
8B
64
43.5/37.9
51.6
70.4
76.8
Idefics2 8B
320
43.0/37.7
51.4
73.0
76.7
Table 9: Performance of Idefics2 against state-of-the-art VLMs up to size of 14B parameters. The evaluations are done zero shot. Idefics2 with 64 320 tokens per image is the same model (same weights), only the inference differs. The full table is present in Appendix AA.32 (Benchmark, Split, Metric): (MMMU, val/test, MMMU score), (Math Vista, testmini, MMMU score). (TexvVQA, val, VQA acc.), (MMBench, test, accuracy).
Then, we scale randomly the resolution of the images during the training. Finally, when applicable, we shuffle the multiple user/assistant turns randomly before feeding the example to the model.
We evaluate Idefics2 on commonly adopted benchmarks: MMMU (uue et al. 2024) for multidis- cipline college-level problems, Math Vista (Lu et al., 2024) for mathematical reasoning, TextVQA (Singh et al., 2019) for text reading on natural images, and MMBench I Liu et al. (2023) for various perception and reasoning tasks. Table 9 presents the results (see Table [5 for the complete result table) of Idefics2 against the current strongest VLMs in its class size: LLa LA VA-Next (Liu et al. (2024), DeepSeek-VL. (Lu al. 2024) and MMI-Chat (McKinzie al. 2024). While being computation- ally much more efficient at inference, Idefics2 exhibits strong performance on various benchmarks, outperforming the current best foundation VLMs its size category.
on par with state-of-the th-tt-
models 4x its size, or with closed-soures models like Gemini 1.5 Pro on several benchmarks like Math Math or TextVQA. The detailed performance of Idefies2 across each category of MMMU (Yue) et al.
2024 is present in Table 16
4.3 Optimizing for chat scenarios
The evaluation benchmarks expect very short answers, but humans prefer long generations when inter- acting with model. We find that Idefics2 can exhibit difficulties in precisely following instructions about the expected format, making difficult to reconeile "chattiness" and downstream performance. As such, after instruction fine-tuning we further train Idefics2 on dialogue data. We fine-tune Idefics2 for a few hundred steps on LLaVA-Conv (Liu et al., 2023) and ShareGPT4V (Chen et al., (023). with large batch size. Our blind human evaluations reveal that Idefics2-chatty is overwhelmingly
preferred over its instruction fine-tuned version many user interactions. We also adversarially
stress-tested the model to generate inaccurate, biased, or offensive responses and reported the findings
in Appendix A.5 We show examples of generations with Idefics2-chatty Figure and in Appendix
in Figures
6 and 7
5 Conclusion
In this work, we re-examine common choices made in the VLM literature and rigorously compare these choices in controlled experiments. Our findings touch upon the effectiveness of different architectures, their performance//nferrncee cost trade-offs as well as training stability. With these learnings at hand, train Idefics2, an open 8B parameters vision-language model. Idefics2 is state-of-the-art on various benchmarks in its category size and is much more efficient at inference. By releasing findings, well models and our training dataset, we aim contribute the ongoing evolution of VLMs and their applications in solving complex real-world problems.
Acknowledgement
We thank Mustafa Shukor for helpful suggestions on the paper, and Yacine Jernite, Sasha I Luccioni, Margaret Mitchell, Giada Pistilli, Lucie-Aimée Kaffee, and Jack Kumar for red-teaming the model.

---

References
Acharya, M., K. Kafle, and C. Kanan (2019). Tallyqa: Answering complex counting questions. In
AAAI.
Achiam, J., S. Adler, S. Agarwal, L. Ahmad, Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, S. Anadkat, al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
Agrawal, H., K. Desai, Y. Wang, X. Chen, R. Jain, M. Johnson, D. Batra, D. Parikh, S. Lee, and
P. Anderson (2019, October). nocaps: novel object captioning at scale. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE.
Alayrac, J.-B., J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. L. Menick, Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkkowsi, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan (2022). Flamingo: a visual language model for few-shot learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Advances in Neural Information Processing Systems, Volume 35, pp. 23716-23736 Curran Associates, Inc.
Antol, S., A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh (2015). VQA: Visual Question Answering. International Conference on Computer Vision (ICCV).
Awadalla, A., I. Gao, J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, S. Sagawa, al. (2023). Openflamingo: An open-source framework for training large autoregres- sive vision-language models. arXiv preprint vrXivv2308.01333
Bach, S., V. Sanh, Z. X. Yong. A. Webson. C. Raffel, N. V. Nayak, Sharma. T. Kim, M. S. Bari, T. Fevry, Z. Alyafeai, M. Dey, A. Santilli, Z. Sun, S. Ben-david, C. Xu, G. Chhablani, H. Wang, Fries, M. Al-shaibani, Sharma, U. Thakker, K. Almubarak, X. Tang, D. Radev, M. T.-j. Jiang, and A. Rush (2022, May). PromptSource: An integrated development environment and repository for natural language prompts. In Basile, Kozareva, and S. Stajner (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Dublin, Ireland, pp. 93- 93-104. Association for Computational Linguisties.
Bai, J., S. Bai, S. Yang, S. Wang, S. Tan, P. Wang. J. Lin, C. Zhou, and J. Zhou (2023). Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.
Bavishi, R., E. Elsen, C. Hawthorne, M. Nye, A. Odena, A. Somani, and S. Taşırlar (2023). Introduc- ing our multimodal models.
Belouadi, J., A. Lauscher, and S. Eger (2023). Automatikz: Text-guided synthesis of scientific vector graphics with tikz. arXiv preprint 1xXix:2310000367
Biten, A. F., R. Tito, L. Gomez, E. Valveny, and D. Karatzas (2022). Ocr-idl: Ocr annotations for industry document library dataset. European Conference on Computer Vision, pp. 241-252. Springer.
Biten, A. F., R. Tito, Mafla, L. Gomez, M. Rusiñol, C. Jawahar, E. Valveny, and D. Karatzas (2019). Scene text visual question answering. In 20/9 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4290-4300
Blecher, L., G. Cucurull, T. Scialom, and R. Stojnic (2023). Nougat: Neural optical understanding for academic documents. arXiv preprint arXiv:2308.1344 /3418.
Brown, T., B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020). Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Advances Neural Information Processing Systems, Volume 33, pp. 1877-1901. Curran Associates, Inc.
10

---

Carbune, V., H. Mansoor, F. Liu, R. Aralikatte, G. Baechler, J. Chen, and A. Sharma (2024). Chart- based reasoning: Transferring capabilities from Ilms to vlms. arXiv preprint arXiv:2403.
Chang, S., D. Palzer, J. Li, E. Fosler-Lussier and N. Xiao (2022). MapQA: A dataset for question answering on choropleth maps. In NeurlPS 2022 First Table Representation Workshop.
Changpinyo, S., Sharma, N. Ding, and R. Soricut (2021). Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR.
Chen, L., J. Li. X. Dong, P. Zhang. He, J. Wang, F. Zhao, and D. Lin (2023). Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:231 1.
Chen, X., J. Djolonga, P. Padlewski, B. Mustafa, S. Changpinyo, Wu, C. R. Ruiz, S. Goodman, X. Wang, Y. Tay, et al. (2023). Pali-x: On scaling up multilingual vision and language model. arXiv preprint arXiv:2305. 18565.
Chen, X. and X. Wang (2022). Pali: Scaling language-image learning in 100+ languages. In Conference on Neural Information Processing Systems (NeurIPS).
Chen, X., X. Wang, L. Beyer, Kolesnikov, J. Wu, Voigtlaender, B. Mustafa, Goodman, I. Alabdulmohsin, P. Padlewski, et al. (2023). Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199.
Chen, Z., W. Chen, C. Smiley, S. Shah, I. Borova, D. Langdon, R. Moussa, M. Beane, T.-H. Huang, B. Routledge, and W. Y. Wang (2021, November). FinQA: A dataset of numerical reasoning over financial data. In M.-F Moens, X. Huang, L. Specia, and S. W.-t. Yih (Eds.), Proceedings the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 3697-3711 Association for Computational Linguistics.
Cheng, Z., H. Dong, Z. Wang. R. Jia. J. Guo, Y. Gao, S. Han, J.-G. Lou, and D. Zhang (2022, May). HiTab: A hierarchical table dataset for question answering and natural language generation. In S. Muresan, P. Nakov, and A. Villavicencio (Eds.), Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), Dublin, Ireland, pp. 1094-11 11--1110,,.01 Association for Computational Linguistics.
Chu. X., L. Qiao, X. Zhang, S. Xu, F. Wei, Y. Yang, X. Sun, Y. Hu, X. Lin, B. Zhang, et al. (2024). Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766.
Conover, M., M. Hayes, A. Mathur, J. Xie, Wan, S. Shah, A. Ghodsi, Wen- dell, M. Zaharia, and R. Xin (2023). Free dolly: Introducing the world's first truly open instruction-tuned 1lm. https: //wws: databricks www.ccommoooo.. http//oooo/2////12/22 doolly first-ir----eer open- wommer commmmrriillyl.... viableialeeeeeeee instruction tuneediooon-nnnn--e. 11m 06-30.
Accessed: 2023-
Dai, W., J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi (2023). InstructBIIIP: Towards general-purposs vision-language models with instruction tuning. In Thirty-seventh Conference Neural Information Processing Systems.
Darcet, T., M. Oquab, J. Mairal, and P. Bojanowski (2024). Vision transformers need registers. In The Twelfth International Conference Learning Representations.
Dehghani, M., J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, R. Jenatton, L. Beyer, M. Tschannen, A. Arnab, X. Wang, C. Riquelme Ruiz, M. Minderer, J. Puigcerver, U. Evci, M. Kumar, S. V. Steenkiste, G. F. Elsayed, A. Mahendran, F. Yu, A. Oliver, Huot, J. Bastings, M. Collier, A. A. Gritsenko, Birodkar, C.N. Vasconcelos, Y. Tay, T. Mensink, A. Kolesnikov, F. Pavetic, D. Tran, T. Kipf, M. Lucic, X. Zhai, D. Keysers, J. Harmsen, and N. Houlsby (2023, 23-29 Jul). Scaling vision transformers to 22 billion parameters. In A. Krause, F. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of the 40th International Conference on Machine Learning, Volume 202 of Proceedings of Machine Learning Research, pp. 7480-7512, PMLR.

---

Dehghani, M., B. Mustafa. J. Djolonga, J. Heek, M. Minderer, M. Caron, A. P. Steiner, J. Puigcerver, R. Geirhos, I. Alabdulmohsin, A. Oliver, P. Padlewski, A. A. Gritsenko, M. Lucic, and N. Houlsby (2023). Patch n' pack: Navit, vision transformer for any aspect ratio and resolution. In Thirty- seventh Conference on Neural Information Processing Systems.
Deng, J., W. Dong, R. Socher, L.-J. Li, Li, and L. Fei-Fei (2009). Imagenet: A large-scale hierar- chical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp-
Desai, K., G. Kaul, Z. Aysola, and J. Johnson (2021). Redcaps: Web-curated image-text data created by the people, for the people. In J. Vanschoren and S. Yeung (Eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Volume 1. Curran.
Douillard, A., A. Ramé, G. Couairon, and M. Cord (2022, June). Dytox: Transformers for continual learning with dynamic token expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9285-9295.
Driess, D., F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence (2023), Palm-e: an embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, ICML' 23.. JMLR.org.
Gao, P., R. Zhang, C. Liu, L. Qiu, S. Huang, W. Lin, S. Zhao, S. Geng, Z. Lin, P. Jin, et al. (2024). Sphinx-x: Scaling data and parameters for family of multi-modal large language models. arXiv preprint 202.0524...
Goyal, Y., T. Khot, D. Summers-Stay D. Batra, and D. Parikh (2017). Making the in vqa matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6325-6334.
He, X., Y. Zhang, L. Mou, E. Xing, and Xie (2020). Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.1043
Hendrycks, D., C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2021). Measuring massive multitask language understanding. In International Conference on Learning Representations.
Hong, W., W. Wang, J. W. Yu, J. Ji, Y. Wang, Z. Wang. Y. Dong, M. Ding, al. (2023).
Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.08914
Hu, A., H. Xu, J. Ye, M. Yan, L. Zhang, B. Zhang, C. Li, J. Zhang, Q. Jin, F. Huang, et al. (2024). mplug-docow 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint 2rXiv:2403.12295
Hu, E. J., yelong shen, Wallis, Z. Allen-Zhu, Y. Li, Wang, L. Wang, and W. Chen (2022). LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Huang, S., L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, Lv, L. Cui, O. K. Mohammed, B. Patra, Q. Liu, K. Aggarwal, Z. Chi, Bjorck, V. Chaudhary, S. Som, X. Song, and F. Wei (2023). Language is not all you need: Aligning perception with language models. In Thirty-seventh Conference on Neural Information Processing Systems.
Hudson, D. A. and C. D. Manning (2019). Gqa: A new dataset for real-world visual reasoning and compositional question answering. In 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 6693-6702.
Iyyer, M., W.-t Yih, and M.-W. Chang (2017, July). Search-based neural structured learning for sequential question answering. In R. Barzilay and M.-Y. Kan (Eds.), Proceedings of the 55th Annual Meeting of the Association for Computational SirimaaLiinn Linguistics (Volume I: Long Papers), Vancouver, Canada, pp. 1821-1831 Association for Computational Linguistics.
12

---

Jaegle, A., F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira (2021, 18-24 Jul). Per- ceiver: General perception with iterative attention. M. Meila and T. Zhang (Eds.), Proceedings of the 38th International Conference on Machine Learning Volume 139 of Proceedings of Machine Learning Research, pp. 4651 16644464. PMLR.
Jain, N., P. yeh Chiang, Y. Wen. J. Kirchenbauer, H.-M. Chu. G. Somepalli, B. R. Bartoldson, B. Kailkhura, A. Schwarzschild, Saha, M. Goldblum, Geiping, and T. Goldstein (2024). NEFTune: Noisy embeddings improve instruction finetuning. In The Twelfth International Conference on Learning Representations.
Jhamtani, H. et al. (2018, October-November), Learning to describe differences between pairs of similar images. In E. Riloff, D. Chiang. J. Hockenmaier, and J. Tsujii (Eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 4024 4034. Association for Computational Linguistics.
Jiang, A. Q., A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. 1. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. (2023). Mistral 7b, arXiv preprint arXiv:2310.06825.
Johnson, J., B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitmick, and R. Girshick (2017). Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1988-1997.
Kafle, S. Cohen, B. Price, and C. Kanan (2018). Dvqa: Understanding data visualizations via question answering. In CVPR.
Kahou, E., V. Michalski, A. Atkinson, Á. Kádár, A. Trischler, and Y. Bengio (2017). Figureqa: An annotated figure dataset for visual reasoning, arXiv preprint arXiv:
Karamcheti, S., S. Nair, A. Balakrishna, Liang, T. Kollar, and D. Sadigh (2024). Prismatic vlms: Investigating the design space of visually-conditioned language models. arXiv preprint 24Xi:22002.0785..
Kazemi, M., H. Alvari, A. Anand, J. Wu, X. Chen, and R. Soricut (2024). Geomverse: A systematic evaluation of large models for geometric reasoning. In Synthetic Data for Computer Vision Workshop @ CVPR 2024.
Kembhavi, A., M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi (2016). diagram is worth dozen images. In B. Leibe, J. Matas, N. Sebe, and M. Welling (Eds.), Computer Vision ECCV 2016, Cham, pp. 235-255-2212 Springer International Publishing.
Kembhavi, A., M. Seo, D. Schwenk, J. Choi, A. Farhadi, and H. Hajishirzi (2017). Are you smarter than sixth grader? textbook question answering for multimodal machine comprehension. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5376-5384
Kiela, D., H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, and D. Testuggine (2020). The hateful memes challenge: Detecting hate speech in multimodal memes. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Advances in Neural Information Processing Systems, Volume 33, pp. 2611-2624. Curran Associates, Inc
Kingma, D. and J. Ba (2015). Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA.
Koh, J. Y., R. Salakhutdinov, and D. Fried (2023). Grounding language models to images for multimodal inputs and outputs.
Lau, J., S. Gayen, A. Ben Abacha, and D. generated visual questions and answers about radiology images. Scientific Data 5, 180251.
(2018, 11). A dataset of clinically
Laurençon, H., L. Saulnier, T. Wang, C. Akiki, A. Villanova del Moral, T. Le Scao, L. Von Werra, C. Mou, E. González Ponferrada, H. Nguyen, J. Frohberg, M. Šaško, Q. Lhoest, A. McMillan- Major, G. Dupont, S. Biderman, A. Rogers, L. Ben allal, F. De Toni, G. Pistilli, O. Nguyen, S. Nikpoor, M. Masoud, P. Colombo, J. de la Rosa, P. Villegas, T. Thrush, S. Longpre, S. Nagel,
13

---

L. Weber, M. Muñoz, J. Zhu, D. Van Strien, Z. Alyafeai, K. Almubarak, M. C. Vu, I. Gonzalez- Dios, A. Soroa, K. Lo, M. Dey, Ortiz Suarez, A. Gokaslan, S. Bose, D. Adelani, L. Phan, H. Tran, I. Yu, S. Pai, J. Chim, V. Lepercq, S. Ilie, M. Mitchell, S. A. Luccioni, and Y. Jernite (2022). The bigscience roots corpus: 1.6th composite multilingual dataset. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Advances in Neural Information Processing Systems, Volume 35, pp. 31809-31888 -8266 Curran Associates, Inc.
Laurençon. H., Saulnier, L. Tronchon, S. Bekman, A. Singh, A. Lozhkov, T. Wang, S. Karamcheti, A. M. Rush, D. Kiela, M. Cord, and V. Sanh (2023). OBELICS: An open web-scale filtered dataset of interleaved image-text documents. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Laurençon, H., L. Tronchon, and V. Sanh (2024). Unlocking the conversion of web screenshots into html code with the websight dataset. arXiv preprini urXiv:2403.09022
Lec, B.-K., B. Park, C. W. Kim, and Y. M. Ro (2024). Moai: Mixture of all intelligence for large language and vision models. arXiv preprint arXiv:2403.07500
Lee, K., M. Joshi, I. Ture, H. Hu, F. Liu, J. Eisenschlos, U. Khandelwal, P. Shaw, M.-W. Chang, and K. Toutanova (2023). Pix2struct: screenshot parsing as pretraining for visual language under- standing. In Proceedings of the 40th International Conference on Machine Learning, ICMI'23. JMLR.org.
Li, B., R. Wang. G. Wang, Y. Ge, Y. Ge. and Y. Shan (2023). Seed-bench: Benchmarking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125.
Li, B., Y. Zhang, L. Chen, J. Wang, F. Pu, J. Yang, C. Li, and Z. Liu (2023). Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.0544
Li, G., H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem (2023). CAMEL: Communica- tive agents for "mind" exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Systems.
Li, J., D. Li, S. Savarese, and S. Hoi (2023). Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23 JMLR.org.
Li, J., D. Li, C. Xiong, and S. Hoi (2022). Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML.
Li, L., Y. Yin, S. Li, L. Chen, P. Wang, S. Ren, M. Li, Y. Yang, J. Xu, X. Sun, et al. (2023). M3it: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprini arXiv:2306.044 043877
Li, Y., Y. Du, K. Zhou, J. Wang. X. Zhao, and J.-R. Wen (2023, December). Evaluating object hallucination in large vision-language models. In H. Bouamor, J. Pino, and K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore, pp. 292-305. Association for Computational Linguistics.
Li, Y., Y. Zhang, C. Wang, Z. Zhong, Y. Chen, R. Chu, S. Liu, and J. Jia (2024), Mini-gemini: Mining the potential of multi multi-modality vision language models. arXiv preprint arXiv:2403. 18814.
Li, Z., B. Yang, Q. Liu, Z. Ma, Zhang. Yang. Y. Sun, Y. Liu, and X. Bai (2023). Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint ar 1.0600..
Lin, B., Z. Tang, Y. Ye, J. Cui, B. Zhu, P. Jin, J. Zhang, M. Ning, and L. Yuan (2024). Moe-Ilava: Mixture of experts for large vision-language models. arXiv preprint mrXiv:2401.15944
Lin, J., H. Yin, W. Ping, Y. Lu, P. Molchanov, A. Tao, H. Mao, J. Kautz, M. Shoeybi, and S. Han (2023). Vila: On pre-training for visual language models. arXiv preprint arXiv:2312.07533.
Lin, T.-Y,, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014). Microsoft coco: Common objects in context. In D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars (Eds.), Computer Vision - ECCV 2014, Cham, pp. 740-755. Springer International Publishing.
14

---

Lin, Z., C. Liu, R. Zhang, P. Gao, L. Qiu, H. Xiao, H. Qiu, C. Lin, W. Shao, K. Chen, et al. (2023). Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:231 1.07575.
Lindström, A. D. and al (2022). Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.055: 05355..
Liu, F., G. Emerson, and N. Collier (2023). Visual spatial reasoning. Transactions of the Association for Computational Linguistics 11, 635-651.
Liu. H.. C. Li, Y. Li, and Y. J. Lee (2023). Improved baselines with visual instruction tuning. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.
Liu, H., C. Li, Y. Li, B. Li, Y. Zhang. S. Shen, and Y. J. Lee (2024, January). Llava-next: Improved reasoning, ocr, and world knowledge.
Liu, H., C. Li, Q Wu, and Y. J. Lee (2023). Visual instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems.
Liu, H., Q. You, X. Han, Y. Wang, B. Zhai, Y. Liu, Tao, H. Huang, R. He, and H. Yang (2024). Infimm-hd: A laap forward in high-resolution multimodal understanding. arXiv preprint arXiv:2403.01487.
Liu, S.-Y., C.-Y. Wang, H. Yin, Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen (2024).
Dora: Weight-decomposed low-rank adaptation. arXiv preprint 2402:2240..99333
Liu, T. and B. K. H. Low (2023). Goat: Fine-tuned Ilama outperforms gpt-4 arithmetic tasks. arXiv preprint arXiv:2305.1 142001.
Liu, Y., H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. (2023). Mmbench: Is your multi-modal model all-around player? arXiv preprint wrXiv:2307.0628/1
Loshchilov, I. and F. Hutter (2019). Decoupled weight decay regularization. In International Conference on Learning Representations.
Lu, H., W. Liu, B. Zhang, B. Wang, K. Dong. B. Liu, J. Sun, T. Ren, Z. Li, Y. Sun, et al. (2024). Deepseek-vl: vI: towards real-world vision-language understanding. arXiv preprint arXiv:2443.055 055555.
Lu, J., C. Clark, S. Lee, Z. Zhang, S. Khosla, R. Marten, D. Hoiem, and A. Kembhavi (2023). Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2112.17177
Lu, P., H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao (2024). Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR).
Lu, P., R. Gong, S. Jiang, L. Qiu. S. Huang, X. Liang, and S.-C. Zhu (2021). Inter-gps. gpe: Interpretable geometry problem solving with formal language and symbolic reasoning. The Joint Confer- ence of the 59th Annual Meeting of the Association for Computational Linguistics and the 1ith International Joint Conference on Natural Language Processing (ACL-IICNLPP 2021).
Lu, P., S. Mishra, T. Xia, L. Qiu, K.W. Chang. S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan (2022). Learn explain: Multimodal reasoning via thought chains for science question answering. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Advances in Neural Information Processing Systems, Volume 35, pp. 2507-2521. Curran Associates, Inc.
Lu, P., L. Qiu, K.-W. Chang, Y. N. Wu, S.-C. Zhu, T. Rajpurohit, P. Clark, and A. Kalyan (2023). Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference on Learning Representations (ICLR).
Lu, P., L. Qiu, J. Chen, T. Xia, Y. Zhao, W. Zhang, Z. Yu, Liang, and S.-C. Zhu (2021). Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks.
15

---

Association for Mañas, O., P. Rodriguez Lopez, S. Ahmadi, A. Nematzadeh, Y. Goyal, and A. Agrawal (2023, May). MAPL: Parameter-efficient adaptation of unimodal pre-trained models for -lanuuge few-shot prompting. In A. Vlachos and I. Augenstein (Eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computcaional Linguistics, Dubrovnik, Croatia, pp. 2523 2548. Association for Computational Linguistics.
Marino, K., M. Rastegari, A. Farhadi, and R. Mottaghi (2019). Ok-vqa: A visual question answer- ing benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR).
Marti, U.-V. and H. Bunke (2002, 11). The iam-database: An english sentence database for offline handwriting recognition. International Journal on Document Analysis and Recognition 5, 39.4.6..
Masry, A., D. Long, Q. Tan, S. Joty, and E. Hoque (2022, May). ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, pp. Computational Linguistics.
Mathew, M., V. Bagal, R. Tito, D. Karatzas, E. Valveny, and C. V. Jawahar (2022). Infographicvqa. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2592-2591.
Mathew, M., D. Karatzas, and C. V. Jawahar (2021). Docvqa: A dataset for vqa on document images. In 202/ IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 2199-2208.
McKinzie, B., Z. Gan, J.-P. Fauconnier, S. Dodge, B. Zhang, P. Dufter, D. Shah, X. Du, F. Peng, F. Weers, et al. (2024). Mml: Methods, analysis & insights from multimodal Ilm pre-training. arXiv preprint 1936612433.06/11
Methani, N., P. Ganguly, M. M. Khapra, and P. Kumar (2020, March). Plotqa: Reasoning over scientific plots. The IEEE Winter Conference on Applications of Computer Vision (WACV).
Mishra, A., S. Shekhar, A. K. Singh, and A. Chakraborty (2019). Ocr-vqa: Visual question answering by reading text in images. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 947-952.
Mitra, A., H. Khanpour, C. Rosset, and A. Awadallah (2024). Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402. 14830.
Obeid, J. and E. Hoque (2020, December). Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. In B. Davis, Y. Graham, J. Kelleher, and Sripada (Eds.), Proceedings of the 13th International Conference Natural Language Generation, Dublin, Ireland, pp. 138 138-147. Association for Computational Linguistics.
Pasupat, P. and P. Liang (2015, July). Compositional semantic parsing on semi-structured tables. In C. Zong and M. Strube (Eds.), Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), Beijing, China, pp. 1470-1480, Association for Computational Linguisties.
Penedo, G., Q. Malartic, D. Hesslow, R. Cojocaru, H. Alobeidli, A. Cappelli, B. Pannier, E. Al- mazrouei, and Launay (2023). The refinedweb dataset for falcon LLM: Outperforming curated corpora with web data only. In seirth- Conference Neural Information Processing Systems Datasets and Benchmarks Track.
Pont-Tuset, J., J. Uijlings, S. Changpinyo, R. Soricut, and V. Ferrari (2020). Connecting vision and language with localized narratives. In A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm (Bds.), Computer Vision ECCV 2020, Cham, pp. 647-664. Springer International Publishing.
Radford, A., W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, G. Krueger, and I. Sutskever (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning.
16

---

Reid, M., N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055530
Ren, M.. R. Kiros, and R. Zemel (2015). Exploring models and data for image question answering. In C. Cortes. N. Lawrence, D. Lee. M. Sugiyama, and R. Garnett (Eds.), Advances in Neural Information Processing Systems, Volume 28. Curran Associates, Inc.
Sanh, V., A. Webson, C. Raffel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. Nayak, D. Datta, J. Chang, M. T.-J. Jiang, H. Wang, M. Manica, S. Shen, 7. X. Yong. H. Pandey, R. Bawden, T. Wang. T. Neeraj. J. Rozen, A. Sharma, A. Santilli, T. Fevry, J. Fries, Techan,
T. L. Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush (2022). Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations.
Schuhmann, C., R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev (2022). Laion-5b: An open large-scale dataset for training next generation image-text models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Advances in Neural Information Processing Systems, Volume 35, pp. 25278-252944
Curran Associates, Inc.
Schuhmann, C., R. Veneu, R. Beaumont, R. Kaczmarezyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki (2021). Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint
Schwenk, D., A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi (2022). A-okvqa: A benchmark for visual question answering using world knowledge. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VIII, Berlin, Heidelberg, pp. 146-162. Springer- Verlag..
Sharma, P., N. Ding, Goodman, and R. Soricut (2018). Conceptual captions: A cleaned, hyper- nymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL.
Shayegani, E., Y. Dong, and N. Abu-Ghazaleh (2024). Jailbreak pieces: Compositional adversarial attacks on muuli--ooda language models. In The Twelfth International Conference on Learning Representations.
Shukor, M., C. Dancette, and M. Cord (2023, oct). ep-alm: Efficient perceptual augmentation of language models. In 2023 JEEE/CVF International Conference on Computer Vision (ICCV), Los Alamitos, CA, USA, pp. 21999-22012, IEEE Computer Society.
Sidorov, O., R. Hu, M. Rohrbach, and A. Singh (2020). Textcaps: A dataset for image captioning with reading comprehension. In A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm (Eds.), Computer Vision ECCV 2020, Cham, pp. 742-758. Springer International Publishing.
Singh, A., R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela (2022). Flava: A foundational language and vision alignment model. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15617-15629.
Singh, A., V. Natarjan, M. Shah, Y. Jiang, X. Chen, D. Parikh, and M. Rohrbach (2019). Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
Srinivasan, K., K. Raman, J. Chen, M. Bendersky, and M. Najork (2021). Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, New York, NY, USA. pp. 2443-2449. Association for Computing Machinery.
Suhr, A., S. Zhou, A. Zhang, Zhang, H. Bai, and Y. Artzi (2019, July). A corpus for reasoning about natural language grounded in photographs. In A. Korhonen, D. Traum, and L. Mårquez (Fds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 6418 64-88. Association for Computational Linguistics.
17

---

Sun. Q., Y. Cui, X. Zhang, F. Zhang, Q. Yu, Z. Luo, Y. Wang, Y. Rao, J. Liu, T. Huang, et al. (2023).
Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.15286.
Sun. Q. Y. Fang, L. Wu, X. Wang, and Y. Cao (2023). Eva-clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303. 15389.
Sun, Z., S. Shen, Cao, H. Liu, C. Li, Y. Shen, C. Gan, L.-Y. Gui, Y.-X. Wang, Y. Yang, et al. (2023). Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525.
Tanaka, R., K. Nishida, and S. Yoshida (2021). Visualmre: Machine reading comprehension on document images. In AAAL.
Tang, B. J., A. Boggust, and A. Satyanarayan (2023). VisText: Benchmark for Semantically Rich Chart Captioning, In The Annual Meeting of the Association for Computational Linguistics (ACL).
Team, G., R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. (2023). Gemini: a family of highly capable multimodal models. arXiv preprint 1rXi::2312.118055
Team, G., T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. Kale, J. Love, al. (2024). Gemma: Open models based gemini research and technology. arXiv preprint arXiv:2403.08290
Teknium (2023). Openhermes 2.5: An open dataset of synthetic data for generalist I1m assistants.
Thiel, D. (2023). Identifying and eliminating esam in generative mi training data and models.
Touvron, H., M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou (2021, October). Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV). pp. 32-42.
Touvron, H., T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, al. (2023). Llama: Open and efficient foundation language models. arXiv preprint ar arXiv:2302. 13971.
Touvron, H. L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
Vallaeys, T., M. Shukor, M. Cord, and J. Verbeck (2024). Improved baselines for data-efficient perceptual augmentation of llms. arXiv preprint 13999.
Wang. B., G. Li, Zhou, Z. Chen, T. Grossman, and Y. Li (2021). Screen2words: Automatic mobile ui summarization with multimodal learning. In The 34th Annual ACM Symposium User Interface Software and Technology, UIST 221, New York, NY, USA, pp. 498-510. Association for Computing Machinery.
Wang, W., Q. Lv, W. Yu, W. Hong, Qi, Y. Wang, Ji, Yang, L. Zhao, X. Song, al. (2023). Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.0307
Wei, J., M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le (2022). Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
Wolf, T., L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu. T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush (2020, October). Transformers: State-of-the. art natural language processing. In Q. Liu and D. Schlangen (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online, pp. 38 45. Association for Computational Linguistics.
Xiao, J. 7. Xu, A. Yuille, S. Yan, and Wang (2024). Palm2-vadapter: Progressively aligned language model makes strong vision-language adapter. arXiv preprint arXiv:2402.10896
18

---

Young, P., A. Lai. M. Hodosh, and J. Hockenmaier (2014). From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2, 67-78.
Yu, L., W. Jiang, H. Shi, J. YU, Z. Liu, Y. Zhang, J. Kwok, Z. Li, A. Weller, and W. Liu (2024). Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference
Learning Representations.
Yue, X., Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen (2024). Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR.
Yue, X., X. Qu, G. Zhang, Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen (2024). MAmmoTH: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations.
Zhai, X., B. Mustafa, A. Kolesnikov, and L. Beyer (2023). Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975-11986.
Zhang. C., F. Gao, B. Jia, Y. Zhu, and S.-C. Zhu (2019). Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Zhang, X.. C. Wu, Z. Zhao, W. Lin, Y. Zhang, Y. Wang, and W. Xie (2023). Pmc-vqa: Visual instruction tuning for medical visual question answering, arXiv preprint WrXiv22305.10415
Zhao, Y., Y. Li, C. Li, and R. Zhang (2022, May). MultiHiertt: Numerical reasoning over multi hier- archical tabular and textual data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), Dublin, Ireland, pp. 6588 36606.00..0 Association for Computational Linguistics.
Zhao, Y., C. Zhao, L. Nan, Z. Qi, W. Zhang, X. Tang. B. Mi, and D. Radev (2023, July). RobuT: A systematic study of table QA robustness against human- annotated adversarial perturbations. In A. Rogers. J. Boyd-Graber, and N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), Toronto, Canada, pp- 6064-6081 Association for Computational Linguistics.
Zhong, V., C. Xiong, and R. Socher (2017). Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.0011
Zhou, B., Y. Hu, X. Weng, Jia, J. Luo, X. Liu, J. Wu, and L. Huang (2024). Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402. 14289.
Zhou, C., P. Liu, Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. YU, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy (2023). LIMA: Less is more for alignment. Thirty- seventh Conference on Neural Information Processing Systems.
Zhu, F., W. Lei, Y. Huang, C. Wang. S. Zhang, J. Lv, F. Feng, and T.-S. Chua (2021, August). TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 3277-3287. Association for Computational Linguistics.
Zhu, W., J. Hessel, A. Awadalla, Y. Gadre, Dodge, A. Fang, Y. Yu, L. Schmidt, W. Y. Wang. and Y. Choi (2023). Multimodal c4: An open, billion-scale corpus of images interleaved with text. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Truck.
Zhu, Y. O. Groth, M. Bernstein, and L. Fei-Fei (2016). Visual7W: Grounded Question Answering in Images. In IEEE Conference on Computer Vision and Pattern Recognition.
19

---

A Appendix
A.1 Further experimental details of the ablations
A.1.1 Cross-attention vs. fully-autoregressive architectures
We apply LoRA modules to the LLM for the fully autoregressive architecture and to the cross- attention modules and the LLM for the cross-attention architecture. In Figure report the average performance with respect to the number of steps, the number of images, as well as the number of text tokens. We see improvement across the board with the fully autoregressive architecture. Comparing the average score with these different axes is essential because the cross- -ttssnttt architecture feeds single token per image to the language model, against 64 for the fully autoregressive architecture with perceiver pooling. This implies that for the same training sequence length, the number of images and text tokens different for the two architectures. Equivalently, the same multimodal document will yield different sequence lengths. Even though we fix the batch size in the comparison, the number of text tokens and number of images grow at different paces under the two architectures.
Number Optimizer Staps
Number Images
Number Tokerss
0.7
Fully autoregressive with LORA
Cross- attention with LoRA
0.88
0.66
Avg Score
0.64
0.6
0.58
0.50
0006
000
Figure 4: Comparison of the cross-attentien and fully autoregressive architectures through the number of steps, the number of images and the number of text tokens.
A.1.2 Comparing various vision backbones
We present in Table The detailed results of comparing multiple vision backbones. While EVA-CLIP- 5B performs similarly to SigL.P-SO400M. we emphasize that it has 11 times more parameters. We also noticed in early experiments that TextVQA is the most sensitive benchmark image resolution. which accounts for the performance increase.
VE backbone
Size
Res.
Avg.
VQAv2
OKVQA
TextVQA
COCO
CLI--VIT-H
600M
224
$7.4
$2.4
41.7
28.2
107.5
EVA-CLIP-SB
4.4B
224
60.2
53.4
43.3
30.4
113.7
SigLIP-SO4000
400M
384
60.6
53.6
43,4
33,8
111.6
Table 10: Detailed results of ablation on the vision encoder backbone
A.1.3 Comparing various pooling strategies
We compare multiple pooling strategies: simple linear layer that takes the flattened sequence of vision hidden states and projects it into shorter sequence of visual tokens, as well as a Mapping Network (Mañas et al. 2023). The perceiver resampler significantly outperforms these two options (see Table II).
We also ablate the number of layers in the perceiver resampler, and find no statistically significant differences when increasing the number of layers, similarly results from Xiao al. (204).. We settle on 3 layers out of caution to avoid any potential capacity bottleneck.
Finally, we add 2-layer modality projection MLP on top of the vision encoder hidden states to project the vision hidden dimension to the language model hidden dimension prior to the perceiver resampler. These changes yield better performance well (see Table l3).
20

---

Vision-language Connector
Avg. score
Linear Projection Mapping Network (Mañas et al. 2023
44.5
518
Pereeier
60.3
Table 1I: Ablation on modality projection
Num. perceiver layers
Avg. score
69.3
3
68.6
12
69.0
Table 12: Ablation on number pereeiver resampler layers
MLP modality projection
Avg.
W/
71.4
W/o
69.6
Table 13: Ablation on the addition of modality projection before the perceiver resampler
A.1.4 Ablations OCR data
We hypothesize that adding PDF documents helps the model learn to read text from images. In Table we compare checkpoints trained with and without OCR documents, along with image resolution increase to ensure that the text is legible. We do not observe statistically significant differences when evaluating checkpoints in zero or few shot. Instead, we fine-tune the checkpoints on Doc for 500 steps with learning rate of le 5, leading checkpoints showing much stronger differences.
A.1.5 Statistical significance of the experiments
In the context of large-scale experiments, this paper does not include error bars due to the high costs and time demands of multiple runs; each experiment nodes containing eight H100s each. Nevertheless, we conducted multiple runs for some experiments and observed differences of less than one point in the average score across all cases. This minimal variation supports the validity of the conclusions drawn from our ablation studies.
take long five days eight
A.2
Details of the instruction fine-tuning
A.2.1 Statistics of The Cauldron
In 14] present the statistics of the datasets included in The Cauldron, well the text-only instruction datasets used for the supervised fine-tuning. For each dataset, we give the number of different images it contains, the number of question-answer pairs, the total number of tokens for the answers in the question-answer pairs, and the selected percentage of tokens it represents in our final mixture after upsampling or downsampling.
Dataset
images # Q/A pairs
tokens
mixture
General visual question answering
VQAv2 (Goyal et al. 2017
82,772
443,757
1,595,929
5.72%
COCO-OO QA (Ren et al. 2015
46,287
78,736
286,982
1.47%
Visual7W (Zhu et al. 2016
14,366
69,817
279,268
1.43%
A-OKVQA (Schwenk et al.
2022
16,539
17,056
236,492
1.21%
TallyQA (Acharya et al. 2019
98,680
183.986
738,254
0.57%
OK-VQA (Marino et 2019)
8,998
9,009
38,853
0.40%
HatefulMemes (Kiela et al. 2020
8,500
8,500
25,500
0.13%
VQA-RAD (Luu ct al. 2018
313
1,793
8,418
0.09%
Captioning
LNarratives
(Pont-Tusee al.
2020
507,444
507.444
21,328,731
4.56%
21

---

Gereen2Words Wang et al. (Liu et al. 2023
2021
15,730
15,743
143,103
0.37%
VSR
2,157
3,354
10,062
0.21%
OCR, document understanding, text transcription
RenderedTTxx
999,000
999,000
27,207,774
5.57%
Doc (Mathew et al.
2021
10,189
39,463
337,829
3,46%
TextCaps Sidorov
2020
21,953
21,953
389,658
2.00%
TexVVQA [Singh et al.
2019)
21,953
34,602
181,918
1.86%
ST-VQA Biten al.
2019)
17.247
23,121
127,846
1.31%
OCR-VQA (Mishra et al. 2019)
165,746
801,579
6,073,824
0.93%
VisualMRC Tanaka et
2021
3,027
11,988
168.828
0.86%
IAM (Marti and Bunke.
2002)
5,663
5,663
144,216
0.74%
Info VQA (Mathew et
2022
2,118
10,074
61,048
0.63%
Diagram image-to-tens
300
300
22,196
0.11%
Chart/figure understanding
Chart2Text (Obeid and Hoque
2020
26,985
30,242
2,852,827
4.38%
DVQA Kafle
2018)
200,000
2,325,316
8,346,234
4.27%
VisText Tang et al. 2023
7,057
9,969
1,245,485
1.91%
ChartQA Masry et
2022
18,271
28.299
185,835
1.90%
PlotQA Methann et al.
2020
157,070
20,249,479
8478299.22 .778
0.65%
FigureQ (Kahou et al.
2017
100,000
1,327,368
3,982,104
0.61%
MapQA
Chang et al.
2022
37,417
483,416
6,470,485
0.33%
Table understanding
TabMWP (Lu et 2023
22,729
23,059
1,948,166
2.49%
TAT-QA ZZu et al. 2021
2,199
13,215
283,776
2.18%
HiTab (Cheng al. 2022
2,500
7,782
351,299
1.80%
MultiHeert (Zhao al. (022
7,619
7,830
267,615
1.37%
FinQA (Chen cl al. 2021]
5,276
6,251
242,561
0.99%
WikiSQL (Zhong et al. 2017
74,989
86,202
9,680,673
0.99%
SQA (yyeer et al. (0177)
8,514
34,141
1,894,824
0.97%
WTQ Pasupat and Liang.
2015)
38,246
44,096
6,677,013
0.51%
Reasoning, logic. maths
Geom Verse (Kazemi et al.
2024
9,303
9,339
2,489,459
3.83%
CLEVR- Math (Lindström and
2022
70,000
788,650
3,184,656
3.26%
CLEVR (Johnson et al. 2017
70,000
699,989
2,396,781
1.23%
IconQA (uu et
27,315
29,859
112,969
1.16%
RAVEN Zhang et al.
2019
42,000
42,000
105,081
0.67%
Inter-GPs (Lu
202T
1,451
2,101
8,404
0.17%
Textbook/academic questions
A12D Kembhavi et al. 2016
3,099
9,708
38,832
0.80%
TQA Kembhavi et 2017
1,496
6,501
26,004
0.53%
ScienceQA (Luu et al.
2022
4,985
6,218
24,872
0.25%
Differences between 2 images NLVR2 (Suhr et al. 2019)
50,426
86,373
259,119
1.33%
GSD (Li al. 2023
70,939
141,869
4,637,229
0,48%
Spot the dill Chamtani et al. 2018
8,566
9,524
221,477
0.57%
Screenshot to code WebSight (Laurençon al. 2024 DaTikz [Belouadi et al. 2023)
500,000
500,000
276,743,299
0.28%
47,974
48,296
59,556,252
0.03%
22

---

Text-only general instructions. math problems, arithmetic calculations
OpenHermees-2.5
Teknium
2023)
1,006,223
248,533,747
12.73%
LIMA (Zhou et al. 2023)
1,052
633,867
0.81%
Dolly Conover et al. 2023)
14,972
1,329,999
0.68%
MetaMathQU Yu ct al. 2024
395,000
74,328,255
3.81%
MathInstruct (Yue et al. 2024
261,781
45,393,559
2.33%
OrcaMath (Mitra et al. 2024)
200,031
63,780,702
1.63%
CamelAlMath (Li al. 2023
0
49,744
21,873,629
0.06%
et
0
17,807,579
455,411,624
3.50%
Goat (Liu and Low 2023)
1,746,300
167,695,693
0.86%
Table 14:
The statistics of datasets used for instruction fine-tuning. # tokens is the total number of tokens for each dataset for the answers only. mixture our selected percentage of answer tokens for each dataset in the final mixture.
A.3 Details of the evaluations
A.3.1
Evaluation setup
We perform all evaluations with a batch size and greedy decoding.
For the multi-choice questions in MMMU, MathVista MMBench, we evaluate with the same prompt used for similar types datasets during the instruction fine-tuning:
Question: (question)
Choices:
A. [choice_at __
B. (choice_b
C. [choice_c]
Answer with the letter.
For the open-ended questions in TexIVQA, DocVQA, and VQAv2, evaluate with the prompt:
Question: (question)
Give very brief answer.
We the stop words Question, User, <end_ of__tfrtatteree and <eos> stop generation.
A.3.2 Expanded evaluation table
We report the expanded evaluation Idefics2 and the comparison to other models in Table 15 This includes scores VQAv2 (Goyal al. 2017), which is widely adopted for evaluation. We acknowledge, though, that the metric used for the open-ended visual question answering benchmarks strongly penalizes models that do generate in the answering "large" when the ground truth is "big" or more verbose reformulations will be counted as incorrect. Our manual qualitative analysis reveals that benchmarks like VQAv2, the generations of two models differing by points would be barely noticeable. This problem is less concerning for other open-ended benchmarks like TextVQA or VQA which require finding text in an image, making the expected answer less prone to ambiguity. The detailed performance of Idefics2 across each category of MMMU Yuu al. 2024) is present Table T6
format as the ground truth. For example,
A.3.3 Qualitative evaluation
We show in Figures 5 and
examples of generations with Idefics2-chatty.
23

---

MMMM
Math
TextvQa
MMBench
DocVQA
VQAv2
tokens per image
Model
Size
7B-14B models
LLaVA-NeXT
13B
2880
36.2/-
35.3
67.1
70.0
82.8
DeepSeek-V VI.
7B
576
36.6/-
36.1
64.4
73.2
49.6
MM1-Chat
7B
720
37.0/35.6
35.9
72.8
72.3
82.8
Idefics2
8B
64
43.5/37.9
51.6
70.4
76,8
67.3
80.8
Idefics2
8B
320
43.0/37.7
51.4
73.0
76.7
74.0
81.2
30B models
Mini-Gemini-HD
34B
2880
48.0/44.9
43.3
74.1
80.6
MMI-Chat
30B
720
44.7/40.3
39.4
73.5
75.1
83.7
LLaVA-NeXT
34B
2880
51.1/44.7
46.5
69.5
79.3
83.7
Proprietary
Gemini 1.0 Pro
47.9/-
45.2
74.6
88.1
71.2
Claude Haiku
50.2/-
46.4
88.8
Claude 3 Sonnet
53.1/-
47.9
89,5
Gemini 1.5 Pro
58.5/-
52.1
73.5
86.5
73.2
Table 15: Performance of Idefics2 against wate-of-theeart VLMs across different sizes. The eval- uations are done in zero shot. Idefics2 with 64 or 320 tokens per image only differs by the image splitting.
(Benchmark, Split, Metric): (MMMU, val/test, MMMU score), (Math Vista, testmini/test, MMMU score), (TexiVQA, val, VQA acc.), (MMBench, test, accuracy), (DocVQA, test, ANLS score), (VQAv2, testdev, VQA acc.).
Prompt
Idefics2 output
name, email, jobrent JSON format.
"name""
"name":"Yann "Yann Karpatty",
"emal": "yann "yann.kerssae wan.kkraaay@ggeil..omm
job": "Machine Learning Engineer, Hugging "education':
"year": 2002 2016",
"degree" Ph.D. Computer Science",
EXPCRIENCE
"university" "Sanffrd Universiy"
Pepharc Conssny Hagiing
COUCATION
'year" 2010 2012",
"degree" "Mster of Science in Artificaal Intelligence", "unwersity" "Massachusets insatute Technology'
SKELLS
'yerr: 2008 2010',
"dogroe" BBacholar Science Computer Science",
"nniversit": "University Toronto"
Figure 5: Idefics2-chatty finds the requested information in the resume, and organizes it in JSON format.
24

---

MMMU category
Score
Overall
45.3
Accounting
26.7
Agriculture
53.3
Architecture and Engineering
33.3
Art
76.7
Art Theory
76.7
Basic Medical Science
53.3
Biology
36.7
Chemistry
43.3
Clinical Medicine
43.3
Computer Science
43.3
Design
76.7
Diagnostics and Laboratory Medicine
36.7
Economicy
46.7
Electronics
36.7
Energy and Power
23.3
Finance
40.0
Geography
30.0
History
50.0
Literature
83.3
Manage
46.7
Marketing
40.0
Materials
33.3
Math
56.7
Mechanical Engineering
43.3
Music
40.0
Pharmacy
36.7
Physics
26.7
Psychology
36.7
Public Health
36.7
Sociology
53.3
Art & Design Business
67.5
40.0
Health & Medicine
41.3
Humanities & Social Science
55.8
Science
38.7
Tech & Engineering
38.1
Table 16: Detailed performance of Idefics2 across each category of MMMU
Yue al. 2024
Prompt
Idefics2 output
Describe the image
The image depicts surreai scone where thre robot soldiers are standing on street
soldier equipped
resembles bread roll. The robots facing towards the right side the image, and they appear to be ready for action. The sky above them filled with ciouds, and light visible distance. atmosphere image quite unusual, combining elements technology and architecture an unexpected
Figure 6: Idefics2-chatty describes an Al-generated image.
A.4 Limitations
Our research highlights significant advancements in VLMs, yet several limitations persist.
25

---

Prompt
Idefics2 output
What happens to fish pelicans increase?
Teqpandal
Acarth
pelicans increase, they may consumne more fish, which could lead to decrease the population or imbalance the ecosystem. This could potentially affect other that rely fish food, such as seals, dolphins, and humans who fish for consumption.
Comppe
Figure 7: Idefics2-chatty answers a question on a scientific diagram.
Notably, current VLMs, including Idefics2, are prone to hallucinations, often generating content by describing nonexistent elements in images fabricating facts. This issue underscores the challenges in ensuring the accuracy and reliability of model outputs.
Furthermore, the development of these models is slowed down by the lack of openly available, large-scale pre-trained vision encoders and comprehensive human-created instruction datasets with long answers for fine-tuning. Such resources are crucial for advancing VLMs, especially those optimized for conversational applications.
Additionally, discussed in Section outputs that are non-inclusive or disrespectful. This vulnerability highlights the importance of continuing improve the robustness and ethical alignment of VLMs to prevent misuse and ensure they contribute positively in diverse applications.
models can be manipulated "jailbroken" to produce
A.5 Red-teaming
In the context of red-teaming exercise, our objective evaluate the propensity of the model to generate inaccurate, biased, offensive responses. We evaluate more specifically the chat-optimized checkpoin
While the model typically refrains from responding to offensive inputs, we observe that through repeated trials or guided interactions, tends hastily form judgments in situations necessitating nuanced contextual understanding, often perpetuating harmful stereotypes. Noteworthy instances include:
Speculating passing judgments, perpetuating historical disparities individuals' professions, social status, or insurance eligibility based solely on visual cues (e.g., age, attire, gender, facial expressions).
Generating content that promotes online harassment offensive memes reinforcing harmful associations from a portrait, or from a benign image.
Assuming emotional states or mental conditions based outward appearances.
Evaluating individuals' attractiveness solely based their visual appearance.
Additionally, we identify behaviors that increase security risks that already exist:
Successfully solving CAPTCHAs featuring distorted text within images.
Developing phishing schemes from screenshots of legitimate websites to deceive users into divulging their credentials.
26

---

Crafting step-by-step guides on constructing small- explosives using readily available chemicals from common supermarkets or manipulating firearms do maximum damage.
It's important to note that these security concerns are currently limited by the model's occasional inability to accurately read text within images.
We emphasize that the model would often encourage the user to exercise caution about the model's generation flag how problematic the initial query can be in the first place. For instance, when insistently prompted to write a racist comment, the model would answer that query before pointing "This type of stereotyping and dehumanization has been used throughout history to justify discrimination and oppression against people of color. By making light of such a serious issue, this meme perpetuates harmful stereotypes and contributes the ongoing struggle for racial equality and social justice.".
However, certain formulations circumvent (i.e. "jailbreak") these cautionary prompts, empha- sizing the need for critical thinking and discretion when engaging with the model's outputs. While jail-breaking text LLMs is an active research area, jail-breaking vision-language models have re- cently emerged as a new challenge as lision-language models become more capable and prominent (Shayegami al. 2024). The addition of the vision modality only introduces new avenues for injecting malicious prompts but also raises questions about the interaction between vision and language vulnerabilities.
A.6 Licenses
A.6.1
License for the models
The model is built on top of two pre-trained models: SigLIP-S0400M (Zhai et al. 2023) and Mistral- 7B-v0.1 (Jiang et al. 2023). Both were released under the Apache 2.0 license, and we release the Idefics2 checkpoints under the same license.
A.6.2 License for the Cauldron
Each of the publicly available sub-datasets present in the Cauldron are governed by specific licensing conditions. Therefore, when making use of them, you must take into consideration each of the licenses governing each dataset. To the extent we have any rights the prompts, these are licensed under CC-BY-4.0.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We conduct extensive experiments around pre-trained models, architecture choice, data and training methods in Section 3 and we build and present the results of our model in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations of our work in Section A.4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
28

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We accurately describe our model architecture, the setups of our experiments, including the hyperparameters, and the data used for the training, in Section We release the model and our training data open-source resources, making the evaluations reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
29

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We train our model on open datasets only. We also release the dataset we built during this work as open-source resource. Our model is integrated in the library Transformers (Wolf et al. 2020). The code to use the model is therefore available online.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
wwwp://wweeeeeeeeee Coaeee//dddeSbbiiii SoroodeeumisssioPPPe.. wwriiisssiiiiiyiyioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify the splits of the evaluations in Table and Table We describe the hyperparameters, the optimizer, number of epochs and details throughout the paper, and particularly in section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We present information about the statistical significance of the experiments in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
30

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information on the computer resources in Section for each ablation and in Section 4.1 for the training of the final model.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/neurips.. cc / http:/ppp....ppuuu.t/tccccccccie
Answer: [Yes]
Justification:
We have reviewed and applied the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: By releasing our model, we identify behaviors that increase security risks (although they already exist), and we discuss them in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
31

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: In Section A.5 we discuss the potential risks of our model. We release the demo of our model with system prompt teaching the model be respectful, polite and inclusive. The dataset we release is combination and modification of selected existing datasets which do not contain unsafe images.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In the references section, we acknowledge all prior work we use in this paper. Moreover, the licenses of our model and dataset present in Section A.6
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
32

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pa se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release model and dataset with model card and dataset card explaining how to use these assets. Moreover, we provide additional resources, like a Google Colab to fine-tune our model, or quantized versions of our model, as helpful tools for the community.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moddl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
33

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "IYSkbJ117g", "reviewer": "Reviewer_Ww8w", "summary": "This work studies a question: what matters when building vision-language models? To this end, this work provides analysis from the following aspects, 1) Are all pre-trained backbones equivalent for VLMs? 2) How does the fully autoregressive architecture compare to the cross-attention architecture? 3) Where are the efficiency gains? (Number of visual tokens and Preserving the original aspect ratio and image resolution) 4) How can one trade compute for performance? Based on these observations, this work further proposes a vision-language foundation model.", "questions": "- I understand YYYYYY and XXXXX are set for anonymization. But they indeed affect the reading. [Just a feedback]\n\n- Please discuss the observations/findings in the context of existing works. Moreover, another work which is released after NeurIPS would be useful to look at [d] \n\n    [d] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs\n\n- Please discuss or compare with methods that also aim to improve VLMs.", "limitations": "This work provides a discussion of the potential limitations in Section A.  I agree with the common issue of hallucinations. Could you provide some suggestions on how to avoid them? Second, I do not quite get \"lack of openly available\". Are the datasets used in this work not publicly available? [I did not force the authors to release the data but a clarification would be helpful]. Lastly, it would be better if the authors could provide some advice on handling \"jailbreak\" cases.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- First of all, I like the idea of analyzing the impact of each module on the VLMs (e.g., LLaVA). This work considers several aspects, such as language model, vision encoder, and input resolution.  \n\n- Second, introducing a VLM model and giving the specific training recipe. I think the three main stages are straightforward, including multi-stage pre-training, Instruction fine-tuning, and optimizing for chat scenarios. The last one is not used in other VLMs. I think it is useful. Please comment on does this strategy can help other existing methods such as LLaVA?", "weaknesses": "- Some of the findings are already studies in previous works. For example, the impact of a vision encoder has been studied in [a,b,c]. The effect of the language model is studied in [a,d]. The impact of input resolution [a,b,d]. However, I do not see a clear comparison and discussion with these works. Namely, although I think the analysis is interesting, it is not new and does not provide any new insights.\n\n   [a] Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models\n\n   [b] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs\n\n   [c] BRAVE : Broadening the visual encoding of vision-language models\n\n   [d] VILA: On Pre-training for Visual Language Models\n\n- Regarding the improvement or training of a new VLM model. While the training recipe is provided, it is crucial to compare it with other methods that also aim to improve VLMs. For example, [b,c,d] all improve the VLMs. I would suggest the authors give a direct comparison or a clear discussion on how the proposed method contributes and why it is important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your remarks that we’ll address one by one, unfortunately briefly and without citing your questions not to exceed the max length.\n\n1) Optimization for chat scenarios depends on training data.\\\nWe used YYYYYY, a compilation of 50 high-quality, mostly human-annotated datasets from literature.\\\nThese datasets' short Q/A pairs can lead to brief model responses.\\\nTo address this, we fine-tuned first on YYYYYY for improved performance (reasoning, OCR, …), then briefly fine-tuned on long conversations to adapt output length.\\\nLLaVA, trained directly on long conversations, doesn’t need this step.\\\nWe avoided using current conversational datasets due to their synthetic nature (often ChatGPT-generated) and simplistic, not diverse questions. YYYYYY's creation and open-source release aims to improve VLMs by leveraging diverse, accurate datasets from literature.\n\n2) We acknowledge the rapid evolution of multimodal vision-language models, with some overlap in findings expected given typical 6-12 month project timelines.\n\nRegarding the very recent papers you mentioned:\\\n-[c] BRAVE (April 10): Considered concurrent work per NeurIPS guidelines, given its date.\\\n-[a] Prismatic VLMs: We’ve cited it extensively. While there are similarities, we've expanded upon their findings. For instance, while they find that unfreezing the vision encoder degrades significantly the performance, we demonstrated that a LoRA strategy can achieve stable training and improved performance when adding trainable weights to the vision encoder, as mentioned just before Finding 2 in our paper. We also highlight additional differences in section 4.1.\\\n-[d] VILA: Also cited, VILA's scope on image resolution was more limited (224 to 336 pixels), while we extend it to 980 pixels and focused more on image splitting strategies rather than comparing the input resolutions for a fixed number of tokens.\\\n-[b] \"Eyes Wide Shut?\": While this paper analyzed the impact of pre-trained vision encoders, it represents only a very small portion of our broader study.\n\nWhile we appreciate the similarities you've noted (from our contributions from section 3.1), we believe our work offers several unique and significant contributions, particularly in sections 3.2, 3.3, 3.4, and 4:\\\n**Most important part for this answer**, please read our general message \"Author Rebuttal\" to all reviewers for a list of our unique contributions, as we cannot put them here due to length constraints.\n\nThese contributions offer new insights to the field beyond the points initially highlighted in your review, that can be now seen as complementary points in our broader study.\\\nWe've put effort to be comprehensive in our literature review and comparisons against other works throughout the paper, as shown by our extensive bibliography.\n\nWe hope this clarification helps to illustrate our contributions. We will do our best to highlight even more these differences in our final version.\n\n3) We agree that [b,c,d] all contribute to advancing VLMs in various ways. Our work, while complementary to these efforts, introduces several novel approaches and improvements that we believe are unique and significant (cf previous answer).\\\nWe have provided comparisons with other models throughout our paper. For instance, a key aspect in our strategy is efficiency at inference. We have managed to do so with three points:\\\n-Using an efficient pooling strategy with a very small number of visual tokens. This is compared against MM1, SPHINX-2K, DeepSeek-VL, DePALM (some of the strongest VLMs at that time, with better performance than VILA or Prismatic VLMs).\\\n-Using the image aspect ratio preserving strategy. To our knowledge, there is no other VLM using this strategy, so it could be compared against all the other models.\\\n-Fine-tuning with variable visual token counts (with/without image splitting). Unlike models we compare to like Llava, MM1, and Monkey that fine-tune only with image splitting, our approach allows users to choose token count at inference. This flexibility wasn't the norm before.\n\n4) Our findings remain relevant in the context of recent works. The field has shifted towards unfreezing language models when feasible or using LoRA for improved stability, aligning with our proposed methods.\\\nMost recent works also use image splitting with variable sub-image counts during training, enabling compute-performance trade-offs at inference (section 3.4).\\\nOur dataset YYYYYY demonstrated the potential of leveraging diverse existing academic datasets for instruction fine-tuning, previously implemented at a smaller scale with less diversity.\\\nThe dataset Cambrian-10M is very inspired by our work, containing most of the public academic datasets we have selected, and the images of the tables we have rendered (we created these table images from datasets that originally lacked visual representations, employing various styles to enhance diversity).\\ \nNotably, our XXXXXX-8B model outperforms Cambrian-1-8B on challenging benchmarks like MMMU and MathVista, while being significantly faster at inference due to fewer visual tokens (64 vs. 576).\n\n5) To mitigate hallucinations, we suggest: (1) improving image-text alignment in data, (2) using DPO to reduce hallucination probabilities, and (3) incorporating negative prompt datasets.\\\nWe've released all datasets used. Regarding \"lack of openly available\", prior works often used LLMs to create synthetic Q/A pairs, limiting diversity and encouraging hallucinations. Our approach involved compiling and processing/modifying diverse, high-quality publicly available datasets.\\\nRLHF methods could enhance VLM robustness against jailbreaking.\n\nWe hope that our responses have provided a clearer picture of our research's unique aspects and its position within the current landscape of vision-language models.\\\nWe have tried to show why our work could be impactful and how it's different from what has been done before.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F9LKv5VxgO", "reviewer": "Reviewer_Gtma", "summary": "The compares different methods and strategies involved in training VLMs\n- impact on inference efficiency by model architecture (fusion module: cross-attention versus autoregressive) & on training stability by multimodal training procedure\n- compare different design choices in a controlled environment and extract experimental findings \n- Build a new  8B model XXXXXX and a new instruction training dataset YYYYYY that surpasses or closely-matches the performance of many large-scale models", "questions": "see weaknesses", "limitations": "Discussed in section A.4", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper has an interesting scientific-report like approach to find a good recipe for training VLMs which would be beneficial to the community\n- Using the derived findings, the proposed model XXXXXX performs higher scores than larger models\n- The authors also curate a new vision-language finetuning dataset YYYYYY. \n- The authors consider many important, yet overlooked design choices, which would help expedite research and development of VLMs", "weaknesses": "- While I fully appreciate the importance of the work in standardizing a recipe for training VLMs, I believe it lacks vigour to make such bold claims. For example, Finding 4 does contradict results in Table 9, TextVQA. \n- Some results seem a bit too overarching. For example, while the authors make a generalised finding “the quality of the language model backbone has a higher impact on the performance of the final VLM than the quality of the vision backbone” (Finding 1), they do mention that this might be because the vision encoders might not be well trained (L108), which seems conflicting.\n- It would be nice to have most/all the findings validated for XXXXXX. \n- Can the authors elaborate on how YYYYYY is different from existing VLM instruction tuning datasets? Is there any scope of test dataset leaks?\n- How do the authors ensure that the superior performance of YYYYYY is not driven by the dataset alone and only because of their findings?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you reviewer Gtma for your interesting remarks, we will try to cover them one by one.\n\n> While I fully appreciate the importance of the work in standardizing a recipe for training VLMs, I believe it lacks vigour to make such bold claims. For example, Finding 4 does contradict results in Table 9, TextVQA.\n\nYou raise a valid point.\\\nOur ablation results show that for most tasks, particularly non-OCR tasks, the number of tokens to encode the image is not critically important.\\\nThis is significant for applications like robotics, which do not necessarily require strong OCR capabilities but need efficient inference.\\\nThis is supported by Tables 9 and 15, which show the image splitting strategy's utility primarily for OCR tasks.\\\nThis finding aligns with other works published on Arxiv after our submission.\\\nWe agree that Finding 4 should be stated more cautiously. We have rephrased it to: “Reducing the number of visual tokens with learned pooling significantly improves compute efficiency at training and inference while improving performance on non-OCR downstream tasks.”\n\n> Some results seem a bit too overarching. For example, while the authors make a generalised finding “the quality of the language model backbone has a higher impact on the performance of the final VLM than the quality of the vision backbone” (Finding 1), they do mention that this might be because the vision encoders might not be well trained (L108), which seems conflicting.\n\nWe understand that Finding 1 might have been confusing.\\\nOur intent was to highlight that, given a fixed size for the vision encoder (around 400M for CLIP-H or SigLIP) and a fixed size for the language model (around 7B for Llama1 and Mistral), using the best available open-source models at the time, the most significant performance improvement was observed when upgrading the LLM.\\\nWe have clarified this by rephrasing the finding with: “Better pre-trained LLMs and vision encoders lead to better performance on multimodal tasks. However, with the best current models for both, the LLM has a higher impact.”\n\n> It would be nice to have most/all the findings validated for XXXXXX.\n\nAll the findings are validated for the model XXXXXX, except for Finding 1 which has been done for the cross-attention architecture. However, we have validated that it holds also for the fully-autoregressive architecture as XXXXXX. We provide a detailed explanation for this question specifically in the answer to reviewer FVDr.\n\n> Can the authors elaborate on how YYYYYY is different from existing VLM instruction tuning datasets? Is there any scope of test dataset leaks?\n\nAt the time of submission, VLM instruction tuning datasets were primarily of two types: synthetically generated Q/A pairs, mostly using ChatGPT (like Llava), and compilations of existing training sets from different benchmarks (like InstructBLIP).\\\nWe adopted the second approach, conducting an extensive literature review to compile a mixture of the 50 highest quality datasets we found.\\\nTo our knowledge, this scale and diversity had not been done before, with many interesting datasets previously missed.\\\nWe transformed all datasets into a consistent format, created images for benchmarks without them (like tables), merged Q/A pairs on the same images to create multi-turn dialogues, prepared non-prompted datasets with prompts, and removed images present in test sets of benchmarks we evaluate on. We have done a contamination analysis by checking that images present in the splits of the benchmarks we evaluate on are not present in our dataset. We guarantee that the dataset YYYYYY is released for the community.\n\n> How do the authors ensure that the superior performance of YYYYYY is not driven by the dataset alone and only because of their findings?\n\nThe dataset YYYYYY certainly plays a crucial role in the final performance, but it is introduced during the fine-tuning stage. The findings are primarily derived before this stage, with the exception of Finding 6, which is independent of the data. Thus, the ablations in Section 3 informed better model and training method choices, while the dataset YYYYYY provided an additional, independent boost.\n\nThank you again for your review, and we will be happy for a further discussion if you have any additional questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4XHVFojcFw", "reviewer": "Reviewer_vo36", "summary": "This paper empirically investigates several essential design choices of vision-language models, such as language backbones, visual encoder backbones, and different architectures. Through extensive experiments, they evaluate different model architectures, training methods, and data utilization strategies. Based on their findings, the authors trained a foundational VLM with 8B parameters that shows state-of-the-art performance.", "questions": "**Availability and Openness**\n\n- Will the training code and datasets also be public for the community? Ensuring the availability of both the training code and datasets would greatly enhance the reproducibility of the study and allow the research community to build upon this work.\n\n**Real-World Applicability**\n\n- How well does the model handle real-world scenarios involving noisy or low-quality data, which are common outside controlled benchmark environments? It is crucial to understand the model's robustness and performance in less ideal, real-world conditions to evaluate its practical applicability.\n\n**Comparison with State-of-the-Art Models**\n\n- How does the model's performance compare with state-of-the-art commercial models such as GPT-4v, GPT-4o, and Gemini? Understanding the discrepancies and similarities in performance between this model and leading commercial models would provide a clearer picture of its competitive standing.\n\n**Architectural and Training Insights**\n\n- What is the performance impact of using cross-attention while unfreezing all parameters?\n- Can a column for the number of optimized parameters be added in Table 3 to highlight the differences? Including this information would help to better illustrate the comparative efficiency and complexity of the models.\n- Does the final model use both LoRA and DoRA? As discussed in the exploration, a full autoregressive structure is better when equipped with LoRA finetuning on the unimodal backbone. Also, the paper mentioned that DoRA is used for instruction finetuning. So, does the final model use both LoRA and DoRA for different purposes during training?\n\n**Model Design and Inference Capacity**\n\n- How many images can the model support during inference?\n\nMinor Issues: Missing citation detail in line 22", "limitations": "Generally, this is a good paper with several noticeable limitations:\n\n- The training resources are quite large and may not be accessible to many researchers, limiting the reproducibility of the results.\n- The evaluation benchmark is limited. More results on commercial SOTA VLMs like GPT-4v, and on more tasks such as reasoning, hallucination, and in-context learning, would be useful to fully assess the model's performance.\n- The red-teaming procedure could be more thorough to prevent malicious use of the proposed model.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "One of the key strengths of this paper is its focus on an important and timely topic. By addressing the critical decisions in VLM design and highlighting areas where improvements can be made, this research contributes to a deeper understanding of how to build more effective and efficient models.\n\nThe analysis and conclusions presented in the paper are particularly insightful and have the potential to significantly benefit the AI research community. By identifying key factors that influence VLM performance, such as the choice of model architecture and training methods, the authors provide clear and actionable recommendations for building better models. \n\nBesides, the availability of the trained models and public code is a notable strength of this work. Additionally, the paper is well-written and well-structured, making it easy to follow. Overall, this is a good paper, and I recommend accepting it.", "weaknesses": "The baseline models compared in the study are primarily open-source models. It would be beneficial to include and discuss state-of-the-art (SOTA) commercial models to provide a more comprehensive evaluation of the model's performance. \n\nAdditionally, incorporating performance reports of more open-source models, as listed on platforms like [MMBench](https://mmbench.opencompass.org.cn/leaderboard), in Tables 8 and 9 could better position the model's ability within the current research community. It is noticed that several open-source models of similar size achieve better performance. For instance, models like [InternLM-XComposer2-VL](https://github.com/InternLM/InternLM-XComposer) demonstrate superior performance despite having a comparable size, indicating potential areas for improvement in the proposed model.\n\nThe evaluation of the model's in-context learning ability is relatively limited. It would be useful to investigate whether the model's performance improves with an increased number of demonstrations, which could provide further insights into its capabilities and limitations in real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. We appreciate your feedback and will address each point individually. We unfortunately cannot cite your questions without exceeding the maximum length.\n\n1) It is true that our comparisons primarily involve open-source models. This is because state-of-the-art commercial models tend to be significantly larger, making direct comparisons challenging. However, we did include MM1, which is not an open-source model. Additionally, in Table 15 of the Appendix, we provide comparisons with Gemini and Claude 3. While our model approaches their performance on benchmarks like MathVista and TextVQA, it is clear that larger scales are needed to contain extensive factual knowledge in the model’s weights, and perform well on benchmarks like MMMU.\n\n2) You are raising a good point that this is a very active field, with many concurrent works, and there are surely potential areas for improvement in the proposed model. We compared ourselves to the best models we were aware of at the time of the submission. Besides InternLM-XComposer2-VL known after, we are not aware of any other model having potentially better scores for the same size at the time of the submission.\\\nNow, if we had to justify our work against InternLM, we would say that the scope of the study is different: when building our model, one major point was the efficiency at inference time. This involved using a perceiver resampler to reduce the number of visual tokens. While this approach works great to obtain even better performance on most tasks, a high number of tokens is still needed for OCR heavy tasks. We also do not have a lower score on the most tracked benchmark MMMU. Second, we only use open datasets, while they use proprietary data. Finally, we focused on learning from ablations for the researchers/community rather than just giving a model.\n\n3) We evaluated our base model with 8 in-context examples, as we wanted to compare our model to MM1-base-7B, the best base model in its class size at that time, which was only evaluated using 0-shot (which doesn’t really give information since the model cannot know in which format it should format its answer) and 8-shot.\\\nWe did see an expected improvement when going from 4-shot to 8-shot by 1-2 points.\\\nWhen evaluating the base model, we are very sensitive to the template expected by the benchmark, which was never seen during training. For example, answering “yes it is” while the growth truth is “yes” would be counted as false. Adding more in-context examples boosts the scores, but there is no evidence at first that it’s improving the model’s ability to solve the task. Potentially, it could only help it to better follow the template expected by the benchmark. A solution to discriminate that would be to use a LLM as judge approach, but it would start to be a bit complex and out of the scope of this current study, and also not comparable with any other work, so this is also a reason why we didn’t push the in-context learning evaluations.\n\n4) We guaranty that all the datasets used for the training are publicly available. Moreover, we are working on cleaning our codebase and documenting it before opening it.\n\n5) We agree on the importance of testing under real-world conditions. Our model has been publicly demoed and tested by many users, although we cannot share details here without compromising anonymity. Additionally, our model was tested by many users against many other models with real-world images and prompts, and obtained a strong ELO score.\n\n6) We provide in Table 15 in Appendix a comparison with the Gemini and Claude 3 series of models.\n\n7) For both the cross-attention and fully autoregressive architectures, unfreezing all parameters during pretraining led to unstable training and eventual loss divergence, motivating Finding 3 in our paper.\n\n8) Thank you for this suggestion. We have added the number of trainable and total parameters for each model in Table 3.\n\n9) The entire pretraining and ablations were done with LoRA. After DoRA was published, we tested it as a direct replacement for LoRA and found similar or slightly better performance. We decided to use DoRA for fine-tuning as it did not hurt performance, did not add more parameters, and was proven to be efficient in various cases by the community.\n\n10) The model can handle an arbitrary number of images as input, with the maximum number limited by the sequence length it was trained with. In our case, the maximum sequence length was 2048, allowing for just over 30 images. Fine-tuning with a longer sequence length could enable inference with more images, such as for videos.\n\n11) We have corrected this issue.\n\n12) Besides the comparison to commercial VLMs mentioned earlier, we acknowledge that the model is not evaluated on a large number of benchmarks (6 in Table 15). This is due to the challenge of finding benchmarks that effectively distinguish VLM performance.\\\nSome open-ended benchmarks are unreliable as good scores often indicate adherence to expected answer templates rather than true capability. For example, we obtained 81.2 on VQAv2 while Gemini 1.5 Pro scored 73.2. It doesn’t mean that our model is better than Gemini 1.5 Pro. We kept benchmarks like MMMU for its comprehensive categories, MathVista for reasoning and math abilities, TextVQA and DocVQA for OCR and document understanding, and MMBench for general VQA evaluation.\n\nThank you again for your detailed comments. We welcome further discussion if you have any additional questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NAvs4VCWmh", "reviewer": "Reviewer_FVDr", "summary": "This work conducts comprehensive experiments for re-examing common choices in the VLM area, such as unimodal backbone, connector, number of visual tokens, data, etc. Based on their experiments, they observe several findings important to the VLM research community. Finally, they rely on their key findings to collect corresponding training datasets and train a powerful VLM.", "questions": "See weaknesses above.", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The writing of this work is clear and easy to follow, and the experiments are thoroughly conducted.  I have to say, I appreciate the thorough experiments and clear reasoning presented in this work. The findings obtained during the experiments can also contribute to the advancement of the VLM community.", "weaknesses": "Since the authors ultimately chose a fully autoregressive training approach, why did they still use cross-attention training for the ablation study in Section 3.1 to support finding 1? Is it because finding 1 does not hold under the autoregressive training approach?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you FVDr for your review. We appreciate your positive feedback regarding the strengths of our work.\n\nRegarding the point mentioned about section 3.1, we can clarify our approach:\n\nWe wanted to start by showing some results with the cross-attention architecture that was the reference approach at that time. This is why the ablations in Section 3.1 were conducted under the cross-attention setup. However, we have verified that the two most significant findings (replacing Llama1 with Mistral and CLIP with SigLIP) also hold under the fully autoregressive approach. Given that these results were consistent with those obtained from the cross-attention architecture early in training, we did not extend these additional training to full completion to save computational resources. As a result, these ablations were not included in the final paper.\n\nWe will be pleased to add these results with comments in the final paper for clarity and readability. We also ensure that our conclusions are in line with recent competing work in the literature.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers, thank you for your detailed remarks. We have commented on each of your point individually.\
Before that, we would like to highlight a summary of our biggest contributions with this work.

- We demonstrate the effectiveness of LoRA training during pre-training for stable training and improved performance (to our knowledge, it wasn’t done before).
- We provide the first comprehensive comparison between cross-attention and fully autoregressive architectures (to our knowledge, it wasn’t done before).
- We show that a learned pooling strategy can significantly improve efficiency at inference while maintaining or improving performance, especially for non-OCR tasks – a finding with important implications for applications like robotics where performing fast inferences is needed (to our knowledge, it wasn’t done with a number of visual tokens as low as 64 before).
- We use the image aspect ratio preserving strategy for arbitrary resolution inputs, improving inference efficiency when lower image resolutions are passed as inputs (to our knowledge, it is not done in any current VLMs).
- We demonstrate that increased token counts for image encoding are primarily beneficial for OCR tasks (to our knowledge, it wasn’t done before).
- We contribute a large-scale, open-source instruction dataset compiled from existing literature (to our knowledge, it wasn’t done at this scale and with this diversity before).
- We provide the community with a strong and efficient 8B open-source VLM (state-of-the-art at the time of the submission).

Other complementary contributions, less “unique”, include:
- A study of the impact of the pre-trained language model and vision encoder on the final VLM.
- The gain obtained by using fully synthetic captions for image-text pairs during the pre-training, which are less noisy than the original alt-texts present on the internet, usually of very poor quality.
- The introduction of the task of text transcription of PDFs directly during the pre-training to increase OCR performance (to our knowledge, it wasn’t done before).

Meta-review: Reviewers and I applaud the in-depth and module-wise analysis of VLM as presented in this draft. Some concrete suggestions made by fellow reviwers and they would improve the draft. The topic is of great interest to the NeurIPS audience. Accept recommended.
Decision: Accept (poster)