POINTS: Improving Your Vision-language Model with Affordable Strategies

## Abstract

Abstract In recent years, vision-language models have achieved significant advancements, excelling in tasks once deemed challenging, such as optical character recognition and geometric problem-solving.
Despite these impressive achievements, several critical issues remain unaddressed:
1) Proprietary models rarely disclose detailed information about their architectures. In contrast, while open-source models provide visibility into their training strategies, detailed ablations of these strategies are highly anticipated.
2) Pre-training data is currently under-explored in open-source works, with most efforts empirically adding datasets from diverse sources, making the entire process elusive and cumbersome.
3) During the fine-tuning stage, the focus is often on adding and ablating more datasets, which frequently leads to diminishing returns.
Therefore, refining data schemes is essential for further enhancing model performance.
To address these issues, we propose the following contributions in this paper:
1) We trained a robust baseline model, leveraging the latest technological advancements in vision-language models.
Building upon existing advancements, we introduced effective improvements and conducted comprehensive ablation and validation for each technique incorporated into this strong baseline.
2) Inspired by recent work on large language models, we propose filtering pre-training data using perplexity, selecting the data with the lowest perplexity as the training set.
This approach allowed us to train on a curated 1M dataset, resulting in highly competitive performance.
3) During the visual instruction tuning stage, we experimented with model soup on different datasets when further introducing more datasets into the training set brought marginal improvements.
Integrating these innovations, we obtained a model with 9B parameters, performing competitively with a series of existing state-of-the-art models.
Additionally, these strategies we propose are efficient and relatively lightweight, allowing the community to adopt them easily for their models.

## 1 Introduction

Advancements in large language models (LLMs; Chowdhery et al. , Jiang et al. , OpenAI , Yang et al. , Dubey et al. ) have significantly enhanced the capabilities of vision-language large models (Fu et al. , Liu et al. , OpenAI , Dong et al. , Zhu et al. ), enabling more sophisticated analyses of textual and visual information. Prominent closed-source model paradigms such as GPT-4 (OpenAI, ), Gemini Pro 1.5 (Fu et al., ), and Claude 3 (Anthropic, ) have achieved remarkable success in expanding LLMs into the realm of vision-language models. Concurrently, open-source vision-language large models are also advancing rapidly, with numerous notable contributions emerging in the field (Liu et al., ; Chen et al., ).

Historically, LLaVA (Liu et al., ) has served as a common baseline. However, recent advancements have rendered its performance suboptimal. Thus, there is a need to establish a stronger baseline for further exploration. In this work, we enhance the vanilla LLaVA architecture by refining the pre-training dataset. Inspired by CapFusion (Yu et al., ), we merge the original captions with world knowledge and generated captions that exhibit good grammatical structure. For visual instruction tuning datasets, we introduce Individual Select (Liu et al., ) to curate effective instruction tuning datasets. Regarding model architecture, we first incorporate Dynamic High Resolution to help the model capture fine-grained details. To address image distortion issues inherent in Dynamic High Resolution, we propose a novel image splitting strategy called Consistent Aspect Ratio Dynamic High Resolution (CATTY), which maintains a consistent image ratio. Additionally, inspired by Vary (Wei et al., ), we merge features from a vision encoder trained separately with text-rich data with those from the original vision encoder, significantly boosting the model’s OCR capabilities. Unlike most existing works (Li et al., ; Chen et al., ), we extensively ablate each newly introduced component in the strong baseline to verify their individual benefits.

Recent works seldom explore the optimization of pre-training datasets. Most studies (Chen et al., ; Yao et al., ; Bai et al., ) tend to empirically combine samples from various large-scale datasets (Schuhmann et al., ; Byeon et al., ), often leading to inefficient and computationally expensive pre-training processes. In the domain of large language models, some research leverages perplexity to filter pre-training datasets. Inspired by this approach, we filter our pre-training dataset by selecting the top samples with the lowest perplexity values. This filtering process yields a subset of 1 million data samples, on which we subsequently pre-train our model. Experimental results demonstrate that the model trained on this filtered subset outperforms a model trained on a dataset five times larger.

In the visual instruction tuning stage, most existing works (Liu et al., ; Li et al., ; Chen et al., ) focus on collecting large quantities of datasets and performing ablation studies to select the most effective ones. However, this approach often reaches a plateau, where introducing additional datasets yields only marginal or even degraded performance. Previous research on model soup has demonstrated the benefits of merging weights from different models fine-tuned with various hyper-parameters. In this work, we propose using model soup to merge weights from models fine-tuned with different datasets to further improve performance when dataset selection no longer brings significant improvement. Compared to conducting model soup on models fine-tuned with different hyper-parameters, e.g. learning rate, the improvement with model soup on models fine-tuned with different datasets is much more prominent. Following this line of work, we further experiment with different model soup strategies and find that greedy model soup is the most effective.

By integrating the aforementioned innovations, we have developed a model called POINTS. Our contributions are threefold:

- $\bullet$
We propose a strong baseline that integrates the latest advancements in vision-language models and thoroughly verify the effectiveness of each component.
- $\bullet$
We introduce the use of perplexity to filter the pre-training dataset and conduct a detailed investigation of data distribution across different perplexity intervals.
- $\bullet$
We employ model soup to merge models fine-tuned with different datasets, thereby enhancing model performance when further dataset selection yields only marginal improvements.

## 2 Related Works

#### Multimodal Large Language Models

The rapid advancement of large language models (LLMs; Dubey et al. , Team et al. , Achiam et al. , Yang et al. , Su et al. ) has laid the groundwork for the emergence of multimodal large language models (MLLMs; Li et al. , Liu et al. , Liu et al. , Bai et al. , Qiao et al. ), which aim to integrate visual understanding with language reasoning and multimodal perception and comprehension. Prominent models such as GPT-4v (Achiam et al., ) and Gemini-1.5-Pro (Team et al., ), developed by major corporations, have spearheaded the MLLM era, utilizing proprietary training data and undisclosed training methodologies. Meanwhile, open-source models have been striving to keep pace. For instance, LLaVA-Next (Liu et al., ) and InternVL-1.5 (Chen et al., ) introduce dynamic high-resolution techniques by dividing a large image into multiple smaller segments with ratio-inconsistent resizing. MiniCPM-V (Yao et al., ) employs a specialized vision encoder to generate non-square image patches. Additionally, models like Vary (Wei et al., ), SPHINX (Lin et al., ), Cambrian-1 (Tong et al., ), and Mini-Gemini (Li et al., ) propose dual vision encoders to enhance visual capabilities. Furthermore, the significant progress in multimodal model evaluation (Liu et al., ; Chen et al., ; Fang et al., ) has also contributed to the rapid improvement of large vision-language models. In this work, we introduce POINT, a model trained exclusively with fully open-source datasets during both the pre-training and supervised fine-tuning (SFT) stages, demonstrating promising results on extensive benchmarks.

#### Visual Instruction Tuning

The selection of training data for multimodal models is of paramount importance (Laurençon et al., ; Tong et al., ), and most improvements in existing works stem from detailed ablation of instruction tuning datasets (Li et al., ; Liu et al., ; Chen et al., ). The commonly used approach to select the most effective datasets involves iteratively adding each dataset to the pool; if it brings improvement, we keep it, otherwise, we drop it. However, this approach may eventually plateau, as further additions might only yield marginal improvements. Previous workHe et al. () has shown the benefits of weight merging, but their experimental results are relatively preliminary. To further enhance performance, we systematically propose employing model soup (Wortsman et al., ) on different models fine-tuned with various visual instruction tuning datasets. This method involves merging the model weights after visual instruction tuning on diverse datasets, resulting in notable performance improvements.

## 3 Methods

This section is divided into three parts: i) In subsection 3.1, we integrate various techniques from previous methods (Liu et al., ; Lin et al., ; Wei et al., ; Liu et al., ; Chen et al., ) to create a strong baseline for further experiments. Additionally, we propose a novel dynamic resolution splitting method, termed Consistent Aspect Ratio Dynamic High Resolution (CATTY for short), to mitigate the issue of image distortion. ii) In subsection 3.2, we propose using perplexity to filter the pre-training dataset. iii) Finally, in subsection 3.3, we incorporate the concept of model soup (Wortsman et al., ) into the instruction tuning stage. We find that this straightforward approach can significantly improve the model’s performance, especially when further data selection only brings marginal or even degraded performance.

### 3.1 A Strong Baseline

In this section, we integrate the recent advancements from existing works to create a strong baseline, containing Dynamic High Resolution from InternVL1.5(Chen et al., ), CapFusion from (Yu et al., ), Dual Vision Encoder from Vary(Wei et al., ) and SPHINX(Lin et al., ), Individual Select from (Liu et al., ). Following LLaVA(Liu et al., ), POINTS mainly contains three parts: vision encoder, projector and the large language model. By integrating all these practices from previous works, we obtain the model structure and pipeline in Figure 1.

Figure: Figure 1: The architecture of POINTS. For each module (e.g. OCR ViT, General ViT, MLP Projector, and Large language model), the label to the left of the dash line indicates the status during pre-training, while the label to the right indicates the status during the instruction tuning stage.
Refer to caption: x1.png

#### Dynamic High Resolution

It has been verified that feeding high-resolution images to vision-language models is beneficial for capturing fine-grained details and reducing hallucinations (Liu et al., ). To enable vision encoder with fixed input resolutions to accommodate dynamic image resolutions, Dynamic High Resolution in LLaVA-Next (Liu et al., ) and InternVL-1.5 (Chen et al., ) splits high-resolution images into several tiles of the same resolution, which the original vision encoder can process. The concrete steps are as follows: i) First, the maximum number of tiles an image can be split into is predefined (set to 8 in our experiments). ii) Based on the maximum number of tiles, a table is created containing information about the target image before splitting. The key of the table is the aspect ratio, and the value is the width and height of the target image, which can be evenly divided by the resolution of the vision encoder. iii) For each image, the target resolution is fetched from the pre-computed table according to the similarity between aspect ratios. The current image is then resized to the target resolution and split into several tiles of the same resolution.

#### Consistent Aspect Ratio Dynamic High Resolution (CATTY)

Before splitting the image, Dynamic High Resolution in InternVL-1.5 (Chen et al., ) resizes the image to the target resolution. However, this resizing is not proportional to the image’s original aspect ratio, which can cause distortion. This issue has been discussed in previous articles(Yao et al., ). Therefore, we propose a splitting method that maintains the image’s aspect ratio, named Consistent Aspect Ratio Dynamic High Resolution (see Figure 2). The first two steps in CATTY are the same as those in InternVL-1.5, and the last step works as follows: Given an image with height $\mathrm{H}$ and width $\mathrm{W}$, we obtain the height and width of the referenced image from the pre-computed table, denoted as $\mathrm{H}^{\text{r}}$ and $\mathrm{W}^{\text{r}}$, respectively. Then, we resize the image to the target size ($\mathrm{H}^{\text{t}}\times\mathrm{W}^{\text{t}}$) by:

$$ ratio $\displaystyle=\text{min}(\mathrm{H},\mathrm{W})/\text{min}(\mathrm{H}^{\text{r }},\mathrm{W}^{\text{r}})$ (1) $\displaystyle\mathrm{H}^{\text{t}}$ $\displaystyle=\text{ratio}\times\mathrm{H}$ $\displaystyle\mathrm{W}^{\text{r}}$ $\displaystyle=\text{ratio}\times\mathrm{W}$ $$

Given the input resolution of a vision encoder, $\mathrm{H}^{\text{v}}\times\mathrm{W}^{\text{v}}$, the target image should be divided into $\frac{\mathrm{H}^{\text{r}}}{\mathrm{H}^{\text{v}}}\times\frac{\mathrm{W}^{
\text{r}}}{\mathrm{W}^{\text{v}}}$ tiles. Next, we split the target image, $\mathrm{H}^{\text{t}}\times\mathrm{W}^{\text{t}}$, using a sliding window with strides $(\mathrm{S}^{\text{h}},\mathrm{S}^{\text{w}})$ across the height and width, respectively. The strides $(\mathrm{S}^{\text{h}},\mathrm{S}^{\text{w}})$ are computed as follows:

$$ $\displaystyle\mathrm{S}^{\text{h}}$ $\displaystyle=(\mathrm{H}^{\text{t}}-\mathrm{H}^{\text{v}})/(\mathrm{H}^{\text {r}}/\mathrm{H}^{\text{v}}-1)$ (2) $\displaystyle\mathrm{S}^{\text{w}}$ $\displaystyle=(\mathrm{W}^{\text{t}}-\mathrm{W}^{\text{v}})/(\mathrm{W}^{\text {r}}/\mathrm{W}^{\text{v}}-1)$ $$

Figure: Figure 2: Comparison between dynamic high resolution in InternVL-1.5 and Consistent Aspect Ratio Dynamic High Resolution (CATTY) proposed by us.
Refer to caption: x2.png

In Equation 2, $\mathrm{S}^{\text{h}}$ is set to 0 if $\mathrm{H}^{\text{r}}/\mathrm{H}^{\text{v}}=1$, and similarly for $\mathrm{S}^{\text{w}}$. This approach allows us to divide a high-resolution image into several tiles without introducing any distortion. There is one exception: if the aspect ratio of the original image is larger than 8, we resize it to an aspect ratio of 1:8 by default. Alongside the tiles obtained using CATTY, we also include a thumbnail of the global view of the image to capture the overall context. This thumbnail is resized to match the input resolution of the vision encoder. Before feeding the features output by the vision encoder into the large language model, we employ the pixel shuffle technique with a down-sampling factor of 0.25, as described in InternLM-XComposer2-4KHD (Dong et al., ), to reduce the sequence length of the image features for improved efficiency.

#### CapFusion

The original captions in existing pre-training datasets are often noisy and structurally flawed, making them sub-optimal for model training. To address this, synthetic captions, such as those in LAION-COCO and BLIP-LAION (Li et al., ), generated by image captioning models, have been proposed. However, the simplistic syntactic and semantic structures in synthetic captions may contribute to issues like Scalability Deficiency and World Knowledge Loss (Yu et al., ). CapFusion strikes a balance between these two types of captions by utilizing a large language model to organically integrate raw and synthetic captions. This approach extracts real-world knowledge from the structurally flawed raw captions while merging it with the structured but syntactically simplified synthetic captions. Following the CapFusion methodology, we use InternLM-XComposer2 (Dong et al., ) to generate captions for images and InternLM2 (Cai et al., ) to integrate the original raw and synthetic captions. The prompts to generate image captions and merge captions are in the Appendix.

#### Dual Vision Encoder

Several previous works, such as SPHINX (Lin et al., ) and Cambrian-1 (Tong et al., ), have demonstrated that different vision encoders exhibit distinct advantages across various domains. Combining features from multiple encoders can lead to improved and more robust performance. Unlike the perception and reasoning required for natural images, text-intensive images demand different capabilities from vision-language models (Wei et al., ). To enhance optical character recognition (OCR) capabilities, we train a separate vision encoder, referred to as the OCR ViT, to extract textual features from images, following the methodology of Vary (Wei et al., ). Unlike Vary, we do not construct training samples, such as charts, ourselves; instead, we utilize OCR results (extracted using PaddleOCR in our case) for pre-training. Additionally, we include natural captions in the pre-training dataset for the OCR vision encoder. More details about the composition of the pre-training datasets for the OCR vision encoder will be discussed in the following section. We merge the features from the general vision encoder (General ViT) and the OCR vision encoder using a weighted average before feeding them into the large language model.

#### Individual Select

Individual Select, as proposed by Liu et al. (), aims to identify the most effective instruction tuning datasets. Building on this approach, we adopt the dataset composition from Liu et al. () as our candidate pool and incorporate additional datasets used in DeepSeek-VL (Lu et al., ), Cambrian-1 (Tong et al., ), and Cauldron (Laurençon et al., ). Ultimately, we integrate 16 more datasets into those identified by Liu et al. () (further details are provided in Appendix). To enhance the diversity of prompts, given the homogeneity in the style of prompts within academic datasets, we employ GPT-4o to generate question-answer pairs in line with previous works (Lu et al., ; Chen et al., ) (the prompt to generate question-answer pairs will be provided in the Appendix). The images for these pairs are randomly selected from LAION-5B (Schuhmann et al., ). We refer to the final composition of visual instruction tuning datasets as the Base Set.

### 3.2 Pre-train Data Selection

In the context of large language models, perplexity has long been employed as a metric to assess the quality of pre-trained datasets (Albalak et al., ; Marion et al., ). Inspired by this approach, we utilize an off-the-shelf vision-language model, $P$—either the model obtained through the steps outlined in subsection 3.1 or an open-sourced VLM—to further filter out low-quality pre-trained datasets obtained via Capfusion, as described above. For each item, $s$, in the pre-trained dataset mentioned in subsection 3.1, we compute the perplexity for all text tokens using the following formula:

$$ $\displaystyle\mathrm{Perplexity}(s)$ $\displaystyle=\mathrm{exp}(-\frac{1}{N}\sum_{i=1}^{N}\mathrm{log}P(w_{i}|w_{1} ,w_{2},...,w_{i-1}))$ (3) $$

Let $\{w_{1},\ldots,w_{N}\}$ represent the text token sequence for $s$. We sort all these items in ascending order and select the first $20\%$ for the pre-training stage. Upon closer examination of the first and last $20\%$ of items, we observe that the distinguishing factor is not the quality of the data, which contrasts with observations in large language models. The last $20\%$ of items often contain obscure world knowledge, such as game version numbers and computer factory serial numbers. This type of world knowledge is extremely rare and contains very little information, making it less beneficial for the model’s learning. In the Appendix, we provide some examples randomly sampled from the first and last $20\%$ of items.

### 3.3 Instruction Data Selection with Model Soup

Visual instruction tuning data is crucial for the superior performance of existing vision-language models (Chen et al., ; Dong et al., ; Liu et al., ). However, most existing works focus on selecting more effective datasets by iterative ablation. In many cases, this approach reaches a plateau, where further data selection can only bring marginal improvements or even degrade performance. In this section, we systematically introduce the benefits of using model soup to integrate the advantages of models fine-tuned with different instruction tuning datasets after data selection meets a bottleneck. The philosophy behind model soup is as follows: given a pre-trained model, fine-tuning the model with different hyper-parameters,
$h_{1},\ldots,h_{k}$, results in several fine-tuned models converging to different local optima, denoted as $f(\theta_{1},h_{1}),\ldots,f(\theta_{k},h_{k})$. These hyper-parameters include learning rate, data augmentation, initialization seed, etc. By interpolating the weights of these fine-tuned models, we can always obtain a stronger model, $f(\theta_{s},h_{s})$. Given the pre-trained model obtained through the methods discussed above, a base instruction tuning dataset $D$, and a series of visual instruction tuning datasets $d_{1},\ldots,d_{k}$ to be selected, we can obtain a stronger model using the following steps:

- $\bullet$
For each dataset $d_{i}\in\{d_{1},...,d_{k}\}$, we add it to the base instruction tuning dataset, $D$, to obtain an augmented dataset, $D^{\ast}_{i}$.
- $\bullet$
We train $k$ models using each augmented from $\{D^{\ast}_{1},...,D^{\ast}_{k}\}$ concurrently, and obtain $\{f(D^{\ast}_{1};\theta_{1}),...,f(D^{\ast}_{k};\theta_{k})\}$.
- $\bullet$
We select $p$ models from $\{f(D^{\ast}_{1};\theta_{1}),...,f(D^{\ast}_{k};\theta_{k})\}$, and merge the weights from all these selected models to obtain a stronger model.

For the third step above, we choose several methods to select the best composition of fine-tuned models to obtain a final model with superior performance, namely, Maximum Soup, Average Soup, and Greedy Soup.

#### Maximum Soup

Given an evaluation score, $\mathrm{Acc}$, we can obtain a strong model, $f(\theta_{s})$, using the following formula:

$$ $\displaystyle\{\theta_{i}\}_{\mathrm{len}(\{\theta_{i}\})=p}$ $\displaystyle=\mathrm{Arg}_{(\theta_{i})}(\mathrm{Top}_{p}(\{\mathrm{Acc}(f(D^ {\ast}_{1};\theta_{1})),...,\mathrm{Acc}(f(D^{\ast}_{k};\theta_{k})\})))$ (4) $\displaystyle f(\theta_{s})$ $\displaystyle=f(\frac{1}{p}\sum_{i=1}^{p}\theta_{i})$ $$

#### Average Soup

By taking the average of weights from all fine-tuned models, we can obtain a stronger model, $f(\theta_{s})$:

$$ $\displaystyle f(\theta_{s})$ $\displaystyle=f(\frac{1}{k}\sum_{i=1}^{k}\theta_{i})$ (5) $$

#### Greedy Soup

We start by sorting the fine-tuned models in descending order based on their evaluation scores. Next, we iterate through these sorted models. For each model, we compute the average of its weights with those of all models currently in the model pool. If the evaluation score improves, the model is added to the pool. Finally, we average the weights of all models in the pool to obtain a stronger model, denoted as $f(\theta_{s})$. The table below outlines the detailed pipeline of Greedy Soup.

Figure: Algorithm 1 Greedy Soup for Visual Instruction Tuning Datasets

## 4 Experiments

This section is divided into five subsections: (i) evaluation setup, (ii) pre-training and instruction-tuning datasets used to train the strong baseline (iii) details about the training setup for the OCR ViT pre-training, the vision-language pre-training, and the visual instruction tuning stages, (iv) ablation studies and analyses of each component used to build our final model, and (v) comparison with other works on extensive benchmarks.

### 4.1 Evaluation Setup

Before embarking on our exploration, we sought a robust evaluation metric to comprehensively assess the various capabilities of our model. This is where OpenCompass (Contributors, ) proves helpful. OpenCompass proposes eight benchmarks to balance the evaluation of a model from different perspectives. These benchmarks include MMBench (Liu et al., ) and MMStar (Chen et al., ) for diagnosing general abilities, MMMU (Yue et al., ) for testing STEM-related abilities, HallusionBench (Liu et al., ) for model hallucination, MathVista (Lu et al., ) for math-related abilities, AI2D (Kembhavi et al., ) for chart-related abilities, OCRBench (Liu et al., ) for OCR capabilities, and MMVet (Yu et al., ) for subjective evaluation. By averaging the metrics from these benchmarks, OpenCompass derives a score that represents the comprehensive ability of a model. Additionally, it offers a useful tool, VLMEvalKit (Duan et al., ), for one-click evaluation. Therefore, unless otherwise specified, we will use these eight benchmarks for our ablation study, with the exception of MMBench, for which we will use the dev-en split.

### 4.2 Data Setup

#### Pre-train Dataset

To train the OCR ViT, we randomly selected 20 million data points from LAION-5B-en (Schuhmann et al., ), LAION-5B-cn (Schuhmann et al., ), WuKong (Gu et al., ), and Zero (Gu et al., ). We then used PaddleOCR to extract text from the images, replacing the original captions to form new image-caption pairs for pre-training. Following Vary (Wei et al., ), we also included 10 million original data samples from LAION-5B, where the captions are the original ones crawled from the Internet. However, we did not adopt the cumbersome pipeline of constructing a new dataset for OCR enhancement, such as crawling PDF files and converting them to images for training (Bai et al., ), as we found our existing pipeline already performs well on OCR-related tasks. For the vision-language pre-training in constructing the strong baseline, we used CapFusion to construct 20 million data points (note that these data do not overlap with those used in OCR ViT pre-training) from LAION-5B. From this set, we selected 5 million data points, as we found this setting works best, similar to the observation in Liu et al. (). Based on the 5 million data, we further selected a 1 million dataset for the final vision-language alignment by choosing the top 20% of data with the lowest perplexity value.

#### Visual Instruction Tuning Dataset

Based on the datasets identified by Liu et al. (), we further employ Individual Select to choose additional datasets from those proposed in (Lu et al., ), (Tong et al., ), and (Laurençon et al., ). The final composition of datasets, referred to as the Base Set, used to construct the robust baseline is presented in Appendix.

### 4.3 Training Setup

#### Pre-training Setup for OCR ViT

The pre-training framework follows the standard LLaVA-style architecture (Liu et al., ), comprising a vision encoder, a two-layer MLP, and a large language model. The vision encoder is initialized from OpenAI’s CLIP-ViT-Large-336, while the large language model is initialized from Yi-1.5-9B-Chat (Young et al., ). Throughout the pre-training stage, the large language model remains frozen, whereas the vision encoder and MLP are trainable. The learning rates for the vision encoder and MLP are set to $2\times 10^{-4}$ and $2\times 10^{-5}$, respectively, with a warm-up schedule during the first 3% of steps, followed by a cosine decay schedule for the remaining steps.

#### Setup for the Vision-language Pre-training Stage

The General ViT, depicted in Figure 1, is initialized from OpenAI’s CLIP-ViT-Large-336, while the OCR ViT is derived from the preceding stage. For the General ViT, only the last three layers are trainable, as this configuration yielded the best results in our experiments. The OCR ViT remains frozen throughout this stage, consistent with the settings used in Vary(Wei et al., ). Features from the penultimate layer of both the General and OCR ViT are selected and fed into the projector. The projector itself is a two-layer MLP, which remains tunable during the pre-training stage. The learning rates for the General ViT and the MLP are set to $2\times 10^{-4}$ and $2\times 10^{-5}$, respectively. A warm-up schedule is applied during the first 3% of steps, followed by a cosine decay schedule for the remaining steps.

#### Setup for the Visual Instruction Tuning Stage

Both the General ViT and OCR ViT remain frozen throughout the entire stage. The learning rates for the projector and the large language model are both set to $2\times 10^{-5}$. A warm-up schedule is applied during the first 3% of steps, followed by a cosine decay schedule for the remaining steps.

### 4.4 Ablation Study and Analysis

Figure: Figure 3: The superiority of Model Soup. When adding additional instruction tuning datasets no longer yields benefits (Individual Select), Model Soup can significantly enhance performance.
Refer to caption: x3.png

#### Each Component to Build the Strong Baseline

As shown in Table 1, each component introduced in subsection 3.1 contributes to steady improvements. These enhancements are significant; for instance, after introducing Dynamic High Resolution to split the input image, we observe substantial improvements in OCR-related tasks, such as OCRBench, with performance increasing from 49.6% to 55.6%. Additionally, the use of high-resolution images with Dynamic High Resolution helps reduce hallucination, primarily due to the increased detail in the high-resolution images. Furthermore, replacing the original Dynamic High Resolution with CATTY results in notable improvements across various benchmarks, with OCR-related benchmarks showing greater gains than others. This is likely because image distortion has a more pronounced negative impact on text within images. Compared to general visual feature extraction, the ability to extract text features from images is limited for CLIP-ViT(Radford et al., ), as it was trained on a large quantity of general image-text pairs. Consequently, we observe substantial improvements on OCRBench after integrating features from an additional ViT, post-trained on text-rich images. Among the 5 strategies, incorporating more visual instruction tuning datasets by Individual Select yields the most significant improvements. This observation aligns with existing works(Chen et al., ; Li et al., ; Tong et al., ), underscoring the importance of selecting effective datasets during the visual instruction tuning stage.

**Table 1: Ablation about each component to build the strong baseline. CF: CapFusion(Yu et al., ), DHR: Dynamic high resolution(Chen et al., ), CATTY: Consistent aspect ratio dynamic high resolution proposed by us, DEV: Dual vision encoder(Wei et al., ), IS: Individual select(Liu et al., ). MMB: the dev-en split of MMBench(Liu et al., ), MV: MathVista(Lu et al., ), HB: HallusionBench(Liu et al., ), OCR: OCRBench(Liu et al., ), Overall: the average of scores on the first 8 benchmarks.**
| CF | DHR | CATTY | DVE | IS | MMB | MV | HB | OCR | AI2D | MMVet | MMStar | MMMU | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  |  | 72.1 | 43.8 | 35.7 | 48.9 | 73.2 | 40.0 | 51.5 | 36.2 | 50.2 |
| ✓ |  |  |  |  | 74.8 | 44.8 | 35.5 | 49.6 | 74.5 | 41.2 | 51.8 | 36.4 | 51.1 |
| ✓ | ✓ |  |  |  | 75.1 | 45.1 | 38.0 | 55.6 | 75.0 | 42.2 | 52.6 | 37.5 | 52.6 |
| ✓ |  | ✓ |  |  | 75.9 | 45.7 | 39.1 | 56.9 | 75.8 | 43.1 | 52.4 | 37.9 | 53.4 |
| ✓ |  | ✓ | ✓ |  | 77.3 | 49.2 | 42.3 | 60.3 | 76.0 | 44.5 | 54.3 | 40.1 | 55.5 |
| ✓ |  | ✓ | ✓ | ✓ | 80.1 | 57.4 | 44.2 | 69.2 | 76.4 | 47.2 | 54.5 | 43.3 | 59.0 |

**Table 2: The first two rows compare the use of different data quantities during the pre-training stage. The third row represents a subset of the 5M dataset from the first row, filtered by perplexity.**
| #num | perplexity | Overall |
| --- | --- | --- |
| 5M |  | 59.0 |
| 20M |  | 58.8 |
| 1M | ✓ | 59.6 |

**Table 5: Comparison between different methods. MMB: the dev-en split of MMBench(Liu et al., ), MV: MathVista(Lu et al., ), HB: HallusionBench(Liu et al., ), OCR: OCRBench(Liu et al., ), SCI: ScienceQA(Lu et al., ), MME: MME(Yin et al., ), RWQ: RealWorldQA, Wild: LLaVA-Wild(Liu et al., ). Cambrian-34: Cambrian-34B(Tong et al., ), Ovis1.5-LLaMA3-8B: Ovis1.5(Lu et al., ), IXC-2.5: InternLM-XComposer-2.5(Zhang et al., ), OneVision: LLaVA-OneVision(Li et al., ), Idefics3-LLaMA3-8B: IDEFICS3 (Laurençon et al., ). The language model POINTS-9B uses is Yi-1.5-9B (Young et al., ). Results are obtained from the leaderboard of OpenCompass, except for MMBench. POINTS-7B uses is Qwen-2.5-7B (Team, ).**
| Methods | MMB | MV | HB | OCR | AI2D | MMVet | MMStar | MMMU | SCI | MME | RWQ | Wild |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Proprietary models |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4o-0513 | - | 61.3 | 55.0 | 73.6 | 84.6 | 69.1 | 63.9 | 69.2 | 90.7 | 2310.3 | 75.4 | 102.0 |
| Claude3.5-Sonnet | - | 61.6 | 49.9 | 78.8 | 80.2 | 66.0 | 62.2 | 65.9 | 88.9 | 1920.0 | 60.1 | 81.0 |
| Gemini-1.5-Pro | - | 57.7 | 45.6 | 75.4 | 79.1 | 64.0 | 59.1 | 60.6 | 85.7 | 2110.6 | 64.1 | 95.3 |
| Open-source models |  |  |  |  |  |  |  |  |  |  |  |  |
| Cambrian-34B | 81.4 | 50.3 | 41.6 | 59.1 | 79.5 | 53.2 | 54.2 | 50.4 | 85.6 | 2049.9 | 67.1 | 82.0 |
| Ovis1.5-LLaMA3-8B | - | 63.0 | 45.0 | 74.4 | 82.5 | 50.9 | 57.3 | 48.3 | 88.8 | 1948.5 | 64.2 | 79.9 |
| Idefics3-LLaMA3-8B | - | 58.4 | 43.7 | 55.0 | 76.5 | 41.7 | 55.0 | 46.6 | 91.3 | 1937.4 | 62.6 | 66.3 |
| InternVL2-8B | - | 58.3 | 45.0 | 79.4 | 83.6 | 54.3 | 61.5 | 51.2 | 97.1 | 2215.1 | 64.2 | 73.3 |
| IXC-2.5 | - | 63.7 | 43.1 | 68.6 | 81.6 | 49.3 | 59.9 | 42.9 | 96.6 | 2233.1 | 67.8 | 70.2 |
| OneVision | 80.8 | 62.3 | 31.6 | 62.2 | 82.4 | 51.9 | 61.9 | 47.9 | 95.4 | 1993.6 | 69.9 | 81.0 |
| Ours |  |  |  |  |  |  |  |  |  |  |  |  |
| POINTS-9B | 83.2 | 60.7 | 48.0 | 70.6 | 78.5 | 50.0 | 56.4 | 46.9 | 92.9 | 2017.8 | 65.9 | 69.3 |
| POINTS-7B | 83.2 | 63.1 | 46.0 | 72.0 | 80.9 | 52.3 | 61.0 | 49.4 | 94.8 | 2195.2 | 67.3 | 71.1 |

#### Pre-train Dataset

As shown in Table 2, scaling up the dataset size (constructed by CapFusion) from 5M to 20M results in downgraded performance, similar to the observations in Liu et al. (). Additionally, some works also achieve promising performance using relative small pre-training datasets instead of a huge number of datasets during the pre-training stage (Li et al., ; Liu et al., ). We believe the possible reasons are: i) The vision encoder of most existing vision-language models is initialized from a pre-trained model that has already been trained on a large quantity of image-text pairs. It is highly likely that most of the data used in the vision-language pre-training stage has already been seen by the vision encoder, thus bringing only marginal or even negative impact when scaling up the size of the vision-language pre-training dataset. ii) The pre-training datasets are quite homogeneous for existing large-scale web-crawled datasets, e.g., LAION-5B and COYO-700M (Byeon et al., ). We plot the distribution of the main entity for each image of a subset extracted from LAION-5B in the Appendix and find that this distribution is long-tailed and constrained to a few objects, e.g., person. Thus, indiscriminately pre-training the model on such datasets can only bring limited benefits. As shown in the third row, we can improve performance by pre-training the model on merely 1M data, coming from the top 20% of the 5M data in the first row, which has the lowest perplexity. This result shows that excessively exposing the model to obscure and scarce knowledge during the transition is detrimental to its learning. Furthermore, compared to fusing features from a separate OCR-enhanced vision encoder, introducing a large OCR dataset has two obvious drawbacks: i) During the pre-training stage, the model has to align both the general features and OCR-related features, which may result in conflicts (Wei et al., ). ii) Since the size of the dataset used in vision-language pre-training is relatively small, a large OCR dataset may overwhelm the learning process, which is not helpful for learning other kinds of knowledge. Thus, introducing features from another OCR ViT can yield superior performance in Table 4.

#### Improve the Performance with Model Soup on Different Datasets.

As described in previous sections, increasingly adding more instruction tuning datasets often reaches a plateau where further increasing the number of datasets yields minimal improvement. However, by incorporating model soup across different datasets, we observe substantial enhancements, as shown in Figure 3, with the overall score increasing from 59.0 to 61.8. We also compare the benefits of various model soup strategies. Among them, greedy soup achieves the best performance, outperforming maximum soup and average soup by 0.8 and 0.6 points, respectively. Unless otherwise specified, we will use greedy soup by default in subsequent experiments. Additionally, we include the results of conducting model soup over different hyperparameters, e.g. different learning rates. As shown, model soup over hyperparameters brings only marginal improvement. Furthermore, we verify in the Appendix that model soup consistently improves performance regardless of the Base Set used.

### 4.5 Comparison with Other Works

In addition to the 8 benchmarks used in the ablation studies above, we further include ScienceQA (Lu et al., ), MME (Yin et al., ), LLaVA-Wild (Liu et al., ), and ReadWorldQA to compare the performance of different models. The following table shows the performance of these models. As shown in Table 5, POINTS achieves performance comparable to existing state-of-the-art models of similar size and even surpasses models with much larger sizes, such as Cambrian-34B. Additionally, compared to the models listed in the table, POINTS uses a much smaller pre-training dataset (e.g., 1M), fewer visual instruction tuning datasets, and all the datasets we used are publicly available. This makes it more affordable for the community to adopt the strategies proposed in this paper. Furthermore, each aspect of POINTS is clearly presented and thoroughly analyzed, making the effectiveness of each strategy employed in our model evident.

## 5 Conclusion

Vision-language models have achieved significant progress in recent years. Following this trend (Chen et al., ; Li et al., ; Liu et al., ; Zhang et al., ; Tong et al., ), we first establish a strong baseline by integrating various advancements proposed in recent works (Liu et al., ; Yu et al., ; Wei et al., ; Liu et al., ) for further experiments. Additionally, we delve into the intricate details of these advancements and propose effective refinements, such as the Consistent Aspect Ratio Dynamic High Resolution. We also conduct extensive experiments to verify the effectiveness of each component in constructing the strong baseline. Secondly, we propose using perplexity to filter the pre-training dataset, retaining only the top 20% of data with the smallest perplexity values during the pre-training stage. This filtering method also brings significant improvements. Model Soup (Wortsman et al., ) has shown promising potential to further enhance performance by averaging the weights of fine-tuned models with different hyperparameters. However, we find that conducting model soup over different dataset settings can yield even more substantial improvements.

## Appendix A Visual Instruction Tuning Datasets

Table 6 shows the visual instruction tuning datasets used in this work.

**Table 6: Visual instruction tuning datasets to build the strong baseline and the those finally selected to conduct model soup (marked in red).**
| Category | Dataset |
| --- | --- |
| Conversation | LLaVAR(Zhang et al., ), inhouse GPT-4o data, Mini-Gemini(Li et al., ) |
| LVIS-Instruct4V(Wang et al., ) |  |
| Document | DocVQA(en)(Mathew et al., ) |
| Caption | ALLaVA(Chen et al., ), ShareGPT4V(Chen et al., ), LAION-GPT4V |
| General QA | VSR(Zhang et al., ), IConQA(Lu et al., ) |
| Science | AI2D(Kembhavi et al., ), TQA(Kim et al., ), ScienceQA(Lu et al., ) |
| Chart&Screen | DVQA(Kafle et al., ), POIE(Kuang et al., ), MapQA(Chang et al., ) |
| ScreenQA(Hsiao et al., ) |  |
| Mathematics | GeoQA+(Cao & Xiao, ), Geo3K(Lu et al., ), TabMWP(Lu et al., ) |
| CLEVR-Math(Lindström & Abraham, ), SuperCLEVER(Li et al., ), |  |
|  | MathV360K(Shi et al., ) |
| Knowledge | KVQA(Sanket Shah & Talukdar, ) |
| OCR | InfoVQA(Mathew et al., ), TextVQA(Singh et al., ) |
| ST-VQA(Biten et al., ), ICDAR2015, HME100K(Yuan et al., ) |  |
| Text-only | LIMA(Zhou et al., ), Alpaca-GPT4(Peng et al., ) |
| OpenHermes2.5(Teknium, ), MetaMathQA(Yu et al., ) |  |
| MathInstruct(Yue et al., ), orca-math-word-problems-200k(Mitra et al., ) |  |
| atlas-math-sets, Math |  |

## Appendix B Prompt for CapFusion

Figure: Figure 4: Prompt for image caption generation and captions merging.
Refer to caption: x4.png

Figure 4 shows the prompts to generate new image caption and merge the original caption and the generated caption.

## Appendix C Prompt for inhouse GPT-4o Dataset

Figure 5 shows the prompt to generate the inhouse GPT-4o data in Table 6.

Figure: Figure 5: Prompt to generate the inhouse GPT-4o dataset.
Refer to caption: x5.png

## Appendix D Model Soup with Different Base Set

To verify that model soup consistently improves performance regardless of the Base Set used, we randomly sampled 6 datasets from the Base Set in Table 6 to conduct model soup, while the remaining datasets were used as the new Base Set. As shown in Figure 6, model soup also brings significant improvements compared to individual selection, demonstrating the effectiveness and universality of model soup.

Figure: Figure 6: Model Soup brings consistent improvement regardless of what Base Set is used.
Refer to caption: x6.png

## Appendix E Homogeneity in Existing Pre-training Dataset

We randomly sampled 5 million images from LAION-5B and used POINTS to identify the main object in each image. We then plotted the distribution of the top six objects from the sampled data. As shown on the left side of Figure 7, these six objects account for more than 90% of the total data. The right side of Figure 7 illustrates the distribution of the top six objects after applying a simple balancing technique: if the count of a particular object exceeds the average count of the top six objects, we down-sample it to 60% of its total count. We re-trained the strong baseline model on both the original and balanced pre-training datasets. The overall score of the balanced version outperformed the original by 0.6. This is an initial investigation into the distribution of the pre-training dataset, and we plan to explore this direction further in our future research.

Figure: Figure 7: Top 6 objects from the subset we randomly sample from LAION-5B (left), and the distribution of the top 6 objects after simple balance (right).
Refer to caption: x7.png

## Appendix F Case Study of Pre-training Dataset

As discussed in Section 3.2, we filter the pre-training dataset using perplexity. Figure 8 shows samples randomly selected from the top 20% and bottom 20% of the data, which have the lowest perplexity values. As illustrated, the captions in the bottom 20% of the data are more likely to contain obscure world knowledge. While augmenting the model with more world knowledge during pre-training can help it generalize better in real-world scenarios, the relatively small scale of the vision-language model pre-training dataset makes this obscure world knowledge in the bottom 20% quite sparse (seldom appearing more than once during pre-training). Consequently, this world knowledge is more likely to be noise rather than informative content for pre-training. Additionally, we also use InternVL2 (Chen et al., ) to filter the pre-training dataset. The model pre-trained on the filtered subset achieves an overall score of 60.1, surpassing the model pre-trained on the original 5M dataset by 1.1 points.

Figure: Figure 8: The left are samples randomly selected from the top 20% and the right are samples randomly selected from the last 20%. These obscure world knowledge is marked in red.
Refer to caption: x8.png

## Appendix G Ablation about the Maximum Number of Tiles

We perform more fine-grained ablation studies about the maximum number of tiles used in CATTY, and Figure 9 shows the results.

Figure: Figure 9: Ablation study about the maximum split used in CATTY.
Refer to caption: x9.png

## Appendix H Ablation about the Feature Average from Vision Encoder

Before feeding features into the LLM, we compute the weighted average of features from both the general and OCR vision encoders. Figure 10 illustrates the model’s performance when different weights are assigned to the general vision encoder (note that the weights assigned to the general and OCR vision encoders sum to 1). In this ablation study, we adhere to the experimental settings described in the fifth row of Table 1 in the main paper.

Figure: Figure 10: Ablation about the weight assigned to the general vision encoder.
Refer to caption: x10.png

## Appendix I Training Cost of POINTS

We employ data parallelism (DP) (Li et al., ) to distribute the data and tensor parallelism (TP) (Shridhar et al., ) to partition the model across multiple GPUs. All our models are trained using $32\times\text{H800 80G GPUs}$. The pre-training stage is completed in 3 hours, while the visual instruction tuning stage takes 7 hours.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - Is the baseline in Table 4 trained only on a base instruction tuning dataset, or is it trained on a combination of a base instruction tuning dataset and a series of visual instruction tuning datasets? How was the baseline model trained?

- What specific off-the-shelf vision-language model was used to calculate perplexity in Table 2? In the experiments, did you observe any differences when using different VLMs?
soundness: 3
strengths: - Paper is clearly written.

- Introduced approaches are technically sound.
summary: This paper proposes a robust baseline vision-language model. The main contributions are threefold: (1) Framework improvement: the authors introduce Consistent Aspect Ratio Dynamic High Resolution for the vision encoder of the multimodal LLM; (2) Data filtering: the authors use perplexity to filter the pre-training dataset; (3) Model ensemble: this work employs model soup methods to merge models fine-tuned with different datasets to achieve better results.
weaknesses: - Novelty concern: The proposed strong baseline model primarily integrates multiple existing advancements in vision-language models, with limited novel technical contributions or findings. Specifically, the proposed CATTY encoding combines dynamic high-resolution encoding and sliding window techniques, both of which have been explored in previous works [1-3]. Additionally, model soup or model ensemble techniques have been shown to effectively boost performance. The novelty of the introduced maximum, average, and greedy soup methods for merging model results is limited.

- Incomplete experimental analysis: Experimental results comparing CATTY with DHR are shown in Table 1. However, the proposed CATTY encoding approach, which leverages a sliding window with overlapping, may also introduce extra computation costs, additional vision tokens, or repetitive information. This analysis is crucial for understanding the pros and cons of using CATTY for future work, yet these aspects are not examined.

- Confusing table and figure references: In the experiments section, some table and figure references are confusing. For instance, in Line 480, should this refer to Table 3? And in Line 484, should this be Table 4? This paper may require more attention to detail in the writing.

[1] Chen, Zhe, et al. "How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites." arXiv preprint arXiv:2404.16821 (2024).

[2] Lin, Ziyi, et al. "Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models." arXiv preprint arXiv:2311.07575 (2023).

[3] Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - While perplexity filtering effectively reduces the dataset size while preserving model performance, there is a potential risk that the model's generalization ability could regress due to reduced exposure to diverse data. Could you provide more insights or experimental evidence on the impact of this data reduction on the model's generalization capabilities?
- Regarding the Model Soup technique, is there a potential risk of overfitting to benchmarks due to the greedy soup method? Could you elaborate on why "MathV360K" showed no improvement while other datasets did?
- Have you observed any cases where CATTY underperforms compared to DHR? It would be useful to understand any trade-offs involved, particularly in terms of applicability to different image types.
- Typo in the table 4 "Greey Soup" -> "Greedy Soup"
soundness: 2
strengths: - The use of perplexity-based filtering of pre-training data is inspired by its success in large language models. This approach is a creative application of an existing concept in a new context, addressing data quality for vision-language models.
- The paper is well-structured, with clear divisions between each method's presentation and the experimental results validating those methods.
- The paper's focus on affordable strategies is practical for the broader community, especially for those without extensive computational resources. Achieving competitive performance using lightweight techniques is useful.
summary: The paper proposes affordable and efficient methods to improve vision-language models, focusing on three main parts: (1) building a robust baseline model with thorough validation, (2) using perplexity filtering to improve the quality of pre-training data, and (3) applying "model soup" to merge weights from different fine-tuned models. The final model, POINTS, achieves competitive performance with state-of-the-art models while remaining lightweight and accessible. The authors also introduce improvements like CATTY to reduce image distortion and enhance OCR capabilities, all while keeping computational costs manageable.
weaknesses: - Each of the proposed methods, such as perplexity filtering, CATTY, and model soup, are either existing methods or slight modifications of existing approaches. This limits the overall novelty of the contributions. More significant deviations or novel techniques could make the paper's contributions more impactful.
- While the authors claimed model soup has introduced improvements, the paper does not thoroughly explore its potential limitations or generalizability across different tasks or datasets. There are only limited details on the possible negative effects of merging model weights, particularly if the datasets have significant divergence in distribution. Also, I wonder if there can be potential overfitting to benchmark from the Greedy Soup method and why "MathV360K" has no improvement while others have improved.
- While perplexity filtering effectively reduces the dataset size while preserving model performance, there is a potential risk that the model's ability to generalize could regress due to a lack of exposure to a diverse range of data. The paper does not provide any experiments or discussion addressing this potential drawback. I think it is important to understand the gap in the understanding of the impact of data reduction on generalization performance.

---

confidence: 4
contribution: 4
initial_rating: 5
presentation: 3
questions: See Weaknesses. Most of my current concerns with the paper are around a lack of proper motivation and analyses of the proposed components.
soundness: 4
strengths: - The proposed components are indeed effective and target the limitation of existing vision-language models in great detail.
- Compared to finetuning dataset selection, the proposed model soup is a more sustainable design choice in terms of the return on performance with scale.
- Perplexity-based dataset selection technique is simple and could have a potentially broader impact on data-efficient pretraining of large foundation models.
summary: The paper proposes to improve the baseline LLaVA model in terms of the following three aspects: (a) to refine the pre-training dataset quality, the paper introduces CapFusion utilizing an LLM to integrate world knowledge from existing captions while simplifying their structures, (b) the model architecture is improved through the introduction of dynamic high resolution component with consistent aspect ratio, and (c) the visual instruction tuning is enhanced through the usage of model soup that merges weights from models fine-tuned on different learning rates.
weaknesses: - While the paper is overall well-written, it is hard to spot the motivation behind the introduced components now and then. For instance, the introduction states "... However, recent advancements have rendered its (LLaVa's) performance suboptimal. Thus, 
 there is a need to establish a stronger baseline for further exploration..." Without proper references or an explicit description of how LLaVa's performance has gone suboptimal, the current motivation reads as "We know the model x and we want to improve upon it." While the latter motivation is indeed genuine, this however opens up a plethora of search space where the model x can be improved. As such, I am reluctant about why the paper would then improve model x in only a few chosen directions.

- Continuing on the above note, a good part of the experiments are occupied with performance on OCR-related tasks. However, there is no such concrete discussion of why the paper chooses to highlight the OCR task, what limitations the current baselines have for the OCR task, and what exactly are the example scenarios where the proposed model performs better on OCR than the baseline LLAVA. 

- My major concern with the current experiments is a lack of discussion on the additional resource overhead brought about by the proposed components. For a practitioner to prefer the proposed POINTS over LLAVA, it would be helpful if the paper mentions: (a) the overhead in pretraining/finetuning in terms of computational resources such as GPU memory, training/inference times in terms of wall-clock/FLOPs as well as the data efficiency of the proposed method. On the lattermost note, I would like to see a discussion on how the POINT model performs compared to LLAVA if a downstream fine-tuning task has limited (few-shot) training examples.

- How does the proposed CATTY method scale with the image size in terms of the model inference time?

Minor suggestions: 

- Introduce the full name for OCR before the first usage of the acronym in the paper (line 057).
- Lines 75-78: citation required for ".. Compared to conducting model soup on models fine-tuned with different hyper-parameters, e.g. learning rate, the improvement with model soup on models fine-tuned with different datasets is much more prominent."

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: * Could the authors discuss in more detail the choice and comparisons of baselines ? I acknowledge that it is easy to get lost in the number of new VLMs coming out these days but since the authors propose an analysis of the training + modeling frameworks, i would expect a comparison table that could explain the choice (and exclusion) of VLMs instances based on a few criteria (datasets, size, LLM base, ViT base, presence of OCR data, dynamic high resolution processing etc..)
* In the ablation study Table 1, what does it mean to not have IS? Does it mean training on all the potential candidates additional datasets?
* Could the authors discuss and analyze the choice of the base LLM (Yi-1.5-9B-Chat l388) ? Specifically, why not choose a base LLM that is the same as the closest baseline (for instance InternVL 1.5 that also has a dynamic high resolution processing)? It would make it easier to disentangle the contributions that come from the dataset selection, the model soup and the base LLM compared to other baselines.

Overall, the authors presented their method in a very clear and structured way. The authors also perform a thorough ablation study. However some key choices (such as GPT-4O distilled dataset, a different base LLM compared to the other baselines) are not discussed or ablated which makes it difficult to disentangle the benefit of the overall approach. The lack of discussion around the baselines and their performance makes it challenging to evaluate the novelty and impact of the proposed contribution.
soundness: 2
strengths: * Well-written and easy to follow.
* Comprehensive set of VL benchmarks including hallucination and OCR benchmarks
* Table 2 shows the benefit of a perplexity-based selection of the base pretraining dataset, which is an interesting result in terms of compute efficiency.
* The authors manage to get competing performance in a wide set of VL benchmarks with a much smaller pretraining dataset (~1M)  than other methods.
* All the datasets and pretrained backbones used by the authors are publicly available which make it easily reproducible.
summary: The paper addresses the problem of model design choices and dataset selection in vision-language models learning. The authors propose a combination of existing methods to build a strong and sample-efficient baseline using a relatively small pretraining dataset of size ~1M. The main contribution is on the analysis of the dataset (individual select) and model selection (model soup) using and combining existing methods. The authors also propose a  novel image splitting strategy to tackle the well known image distortion problem in models that use a dynamic high resolution image processing.
weaknesses: * Ablation study is thorough but the comparison with other baselines is not discussed. Specifically, POINTS is outperformed in many of the benchmarks the authors chose by similarly sized open-source models, but the differences are not discussed.
* The authors also did not discuss the choice of the base LLM (l380 Yi-1.5-9B-Chat) compared to other baselines, which makes it difficult to disentangle the reasons for the difference in performance.
* The authors introduce a QA dataset distilled form GPT-4O and include it in their base set. It is possible that this knowledge distillation alone explains why adding more data is not efficient and why a ~1M dataset is enough. The authors only justify this addition by the need to have prompt diversity but do not ablate it.

**Minor Issues**
* Small typo l480: refer to table 4 instead of table 5
* Some tables are missing the hyperlink ref (l460 and l480 for instance)
* Typo l482 “increasing adding more”
* The authors could rephrase l058 ” Unlike most existing works (Li et al., 2024; Chen et al., 2024c), we extensively ablate each newly introduced component in the strong baseline to verify their individual benefits” and not point out the lack of ablations in other works, especially when those cited work actually reference a whole blogpost about ablations in LLaVA-Next (see [1]) . 
[1]  https://llava-vl.github.io/blog/2024-05-25-llava-next-ablations/

Meta-review: This paper discusses critical design choices when training LLaVA-like vision-language models. The authors explore three main components : the training data, image splitting, and model averaging. All three components seem to provide improvements. Reviewers have noted that the paper lacks details and could be improved in a variety of ways. While defining strong and simple baselines is of outstanding value to the community, the current form of this work is not good enough to warrant acceptance at ICLR.

Decision: Reject