Learning Interleaved Image-Text Comprehension in Vision-Language Large Models

## Abstract

Abstract The swift progress of Multi-modal Large Models (MLLMs) has showcased their impressive ability to tackle tasks blending vision and language.
Yet, most current models and benchmarks cater to scenarios with a narrow scope of visual and textual contexts.
These models often fall short when faced with complex comprehension tasks, which involve navigating through a plethora of irrelevant and potentially misleading information in both text and image forms.
To bridge this gap, we introduce a new, more demanding task known as Interleaved Image-Text Comprehension (IITC).
This task challenges models to discern and disregard superfluous elements in both images and text to accurately answer questions and to follow intricate instructions to pinpoint the relevant image.
In support of this task, we further craft a new VEGA dataset, tailored for the IITC task on scientific content, and devised a subtask, Image-Text Association (ITA), to refine image-text correlation skills.
Our evaluation of four leading closed-source models, as well as various open-source models using VEGA, underscores the rigorous nature of IITC.
Even the most advanced models, such as Gemini-1.5-pro and GPT4V, only achieved modest success.
By employing a multi-task, multi-scale post-training strategy, we have set a robust baseline for MLLMs on the IITC task, attaining an 85.8 % percent 85.8 85.8\% 85.8 % accuracy rate in image association and a 0.508 0.508 0.508 0.508 Rouge score. These results validate the effectiveness of our dataset in improving MLLMs capabilities for nuanced image-text comprehension. Project page: https://zhourax.github.io/VEGA/

## 1 Introduction

The swift advancement of Multi-modal Large Language Models (MLLMs) [, , , , ] has recently showcased their remarkable aptitude for tackling vision-language tasks.
These models have significantly improved in areas such as logical reasoning [, ], high-resolution image comprehension [], In-Context Learning (ICL) [, ], and Chain of Thought (COT) processing [], leading to notable achievements.
Despite these advancements, current applications typically engage MLLMs in the basic multi-modal comprehension task, which involves posing questions tied to a narrow scope of visual content.
This approach falls short when compared to human daily comprehension.

Human comprehension, particularly in document analysis, presents unique challenges not found in basic comprehension tasks such as Visual Question Answering (VQA) tasks []. As shown in Fig. 1, these challenges include: 1) Interleaved Text-Image Distraction: MLLMs are forced to find the correct context from irrelevant text-image mixtures.
2) Long-Form Content: The inputs consist of intertwined sequences of images and text, demanding comprehension over long multimodal context.
3) Image-Referenced Answers: The model is tasked with identifying and linking the relevant image and text based on specific instructions before crafting an appropriate response. Recent prominent open-source MLLMs, such as LLaVA 1.5 [], CogVLM [], and PaperOwl [], lack support for multiple image inputs and struggle with such intricate comprehension task. Multi-image, multi-turn models [] and video-based MLLMs [] face challenges in concurrently processing extended visual and textual contexts effectively. Additionally, benchmarks for evaluating such an intricate task are limited, obscuring our understanding of MLLMs’ practical comprehension abilities.

Figure: Figure 1: Comparison between existing VQA tasks and our IITC task. Left: The input for existing VQA tasks only incorporates a limited amount of image and text data, which is highly relevant to the question. Right: The input for the IITC task contains longer images and text information, which includes redundant and misleading data. The model needs to specify the reference image when providing an answer.
Refer to caption: x1.png

To bridge this gap, we introduce a novel multi-modal task, Interleaved Image-Text Comprehension (IITC), which involves locating the relevant text and image within a complex context given a question, providing an accurate answer, and outputting the corresponding image index. Meanwhile, we present the first benchmark MLLM to handle the above challenges within the IITC task.

Specifically, we observe that IITC is a complex task where simply increasing the number of Interleaved Image-Text QA samples is insufficient for accurate comprehension.
Therefore, we propose utilizing a multi-task learning strategy to enhance the model’s understanding capabilities: employing the Image-Text Association (ITA) auxiliary task to strengthen the model’s ability to accurately locate image-text paragraph correspondences.
In addition, we introduce a multi-scale training strategy, constructing the training data with progressively increasing context lengths and image numbers to enhance the model’s robustness against gradually increasing redundant image-text content.

Ultimately, we have developed a novel dataset, designated as VEGA. It is comprised of two subsets, one tailored for the IITC task and another for the ITA task. The longest interleaved image-text content in VEGA reaches up to 8 images and 8k tokens. We design the instruction of the IITC task to be a question about only one of the images, requiring the model to specify the image it refers to in its answer.
We assess the model’s interleaved image-text reading comprehension ability by both the correct rate of associated images, and the text quality of the answer by ROUGE[] and BLEU []. We have evaluated several state-of-the-art MLLMs on our dataset, validating the challenge of our tasks. Furthermore, we have fine-tuned the Qwen-VL-Chat model [] on the VEGA dataset to set a robust baseline for the IITC task.

In conclusion, this study makes several noteworthy contributions to the field:

- •
Identifying a Novel Task. We introduce a novel Interleaved Image-Text Comprehension (IITC) task. It evaluates and enhances MLLMs’ capabilities of following complex instructions and extracting key cues in long and interleaved image-text scenarios.
- •
Introducing the VEGA Dataset. We develop a new VEGA Dataset for the IITC task that enables a comprehensive understanding of scientific literature, whose multi-modal context reaches up to 8,000 tokens in length and contains up to 8 images.
- •
Benchmarking MLLMs in IITC. We utilize the VEGA dataset to appraise the IITC capabilities of current state-of-the-art models, including GPT4V, Gemini-1.5-pro, and Qwen-VL-Chat, experimentally validating the challenge of IITC.
- •
Enhancing MLLMs’ IITC Capabilities. We fine-tune the Qwen-VL-Chat model on the VEGA dataset with a multi-scale, multi-task training strategy and our experimental findings demonstrate that it achieves an image association accuracy rate of 85.8%. This represents a significant improvement and establishes a strong baseline for the IITC task.

## 2 Related Work

### 2.1 MLLMs

Recently, there has been a notable increase in the deployment of MLLMs aimed at tackling more complex tasks that involve multiple forms of media. These advanced models are equipped to understand and interpret not just text, but also visual content like images and videos []. For instance, BLIP-2 [] features the Q-Former, a key component that establishes a link between a static LLM and visual data, showing impressive results in VQA tasks. InstructBLIP [] is specifically fine-tuned using a variety of instruction-based datasets, which improves its understanding of visual scenes and dialogues. Multi-modal-CoT [] brings the concept of chain-of-thought [] into the multi-modal domain, demonstrating strong performance on the ScienceQA benchmark [].
LLaVA [] uses a linear approach and fine-tunes the entire LLM to enhance its effectiveness. The LLaVA-NeXT [], compared to LLaVA-1.5, quadruples the pixel resolution of input images, improves visual guidance for data blending, and offers enhanced visual reasoning and OCR capabilities.
The LLaMA-Adapter [] introduces a lightweight adapter module, enhancing the model’s capability to process visual information. NExT-GPT [] connects LLM with multimodal adapters and various diffusion decoders, enabling NExT-GPT to perceive inputs and generate outputs in any combination of text, images, video, and audio.

In this work, our focus is on the IITC task, so we concentrate only on models that support multi-image input, such as Qwen-VL [] and InternVL []. We use Qwen-VL-Chat as the base model for training to establish a baseline for the IITC task.

Figure: Figure 2: The task definition of IITC and ITA tasks. (a) The IITC task takes long interleaved image-text content as input and requires the model to specify the image it refers to in its response. (b) The ITA task takes shuffled images and text segments from different articles as input and requires the model to output the relationship between the text and the images. <Text *> and <Image *> represent a text segment and an image, respectively. They are both tokenized and fed into the model along with the task prompt and the question.
Refer to caption: x2.png

### 2.2 MLLM Benchmarks

The evolution of MLLMs, driven by multi-modal pretraining and instruction tuning, has outpaced traditional benchmarks like visual question answering and image captioning. New benchmarks have emerged to assess aspects such as OCR capabilities, adversarial robustness, and hallucination susceptibility. POPE [], HaELM HaELM [], LAMM [], LVLM-eHub [], and MM-Vet [] provide insights into these areas and give a holistic view of MLLM’s performance.

Vision-language benchmarks like Winoground [], RAVEN [] , OK-VQA [], and VCR [] aim to measure MLLMs’ nuanced reasoning abilities. TextVQA [], FigureQA [], ScienceQA [], and MathVista [] contribute to understanding MLLMs’ domain-specific reasoning. Comprehensive benchmarks like MME [], MMBench [], and SEED-Bench cover a variety of reasoning skills. However, the prevalent use of multiple-choice formats can lead models to rely on hints and overlook the reasoning process, focusing only on the final answer’s correctness.

## 3 Proposed Method

In this section, we introduce our VEGA dataset by first elaborating the task definition (3.1), followed by outlining the process of construction (3.2) and statistics (3.3). Finally, we present the training details of our baseline MLLM model (3.4).

### 3.1 Task Definition

Interleaved Image-Text Comprehension (IITC). Our VEGA dataset is meticulously designed to address the challenging IITC task, with the goal of advancing the real-world comprehension capabilities of MLLMs. As demonstrated in Fig. 1, an MLLM is presented with the complex task of discerning relevant text and images in response to a given question.
The model is expected to not only deliver an accurate answer but also to identify the specific image index related to the response. This requirement extends beyond the traditional boundaries of VQA tasks [].

To facilitate this process, we have developed a specialized prompt that directs MLLM to carry out the desired instruction. The complete input for MLLM is organized as shown in Fig. 2 (a). We utilize the notation “[Picture n]” within the text to establish a clear reference to the n-th image, whether it appears in various sections of an article or across different examples. This approach enables the model to learn a consistent and standardized format for image referencing.

In the IITC task, the crux of the challenge lies in the model’s proficiency in accurately linking the appropriate images with the corresponding textual information as dictated by the instructions. This precise pairing is crucial, for it is the foundation upon which a coherent answer is constructed. To quantitatively gauge the model’s aptitude for this association, we have crafted each question pertaining to a specific input image, necessitating that the model explicitly reference this image as part of its response.
The model’s prowess in image-text association is quantified by its success rate in identifying the correct image. Meanwhile, the textual response’s quality is evaluated using established linguistic metrics, namely the ROUGE [] and BLEU [] scores. Together, these measures provide a comprehensive assessment of the model’s performance on the IITC task.

Image-Text Association (ITA). It is the auxiliary task designed to sharpen MLLM’s precision in aligning text paragraphs with their relevant images. As illustrated in Fig. 2(b), ITA challenges the model with a shuffled array of inputs: $\left(P_{1},\ldots,P_{n},T_{1},\ldots,T_{n}\right)$. Each image, labeled as $P_{i}$ from $i=1$ to $n$, is linked to a unique article, while each text snippet $T_{i}$ provides context for its image $P_{i}$. The textual inputs are intentionally scrambled at the outset. ITA’s goal is for the model to accurately match and declare the correct pairings of each text segment $T_{i}$ with its image $P_{i}$, as demonstrated in Fig. 2(b).
Compared to the IITC task, ITA places greater emphasis on the model’s ability to associate images with text. It is anticipated that training with ITA will bolster this skill within the model, leading to improved performance on the IITC task.
For the ITA task evaluation, with a set of $n$ images, we count the number of correctly associated image-text pairs $m$, and use the ratio $m/n$ as the performance metric for the ITA task.

Figure: Figure 3: The construction process of the VEGA dataset. We use the SciGraphQA dataset as a foundation, to which we add more images and text to its context, modify questions and answers that lack clear image references, and incorporates references to meet the requirements of the IITC task.
Refer to caption: x3.png

### 3.2 Data Collection

We develop the VEGA dataset upon the foundation of SciGraphQA [].
SciGraphQA is a multi-turn question-answering dataset for scientific graphs, containing 295k high-quality, multi-turn data entries.
It collected part of the papers published on arXiv from 2010 to 2020, and extracted the pictures (denoted by $\{P_{i}\}^{N}_{i=1}$, $N$ is the total number of pictures), the captions ($\{C_{i}\}^{N}_{i=1}$) of the pictures, and the first paragraphs ($\{M_{i}\}^{N}_{i=1}$) mentioning the pictures from the papers.

As outlined in Fig. 3, our enhancement involves a meticulous process of questions filtering that pertain to the visual-textual context, constructing long visual and textual context, and formulating answers that directly reference images.
The VEGA dataset is structured into two subsets, each specifically curated to train models on the IITC and ITA tasks, respectively.

#### 3.2.1 Subset for IITC

Question Filtering.
In the original SciGraphQA dataset, some questions about images are quite general, such as “What are the implications of the results shown in the graph?” and “What does the x-axis and y-axis of the graph represent?” For more examples, please refer to the supplementary materials. However, when these questions are placed within the context of multi-image long articles, they encounter issues with ambiguous image references. Furthermore, the answers are heavily dependent on the image content, rather than the multi-modal context information, which is not aligned with the objectives of the IITC task. To address this, as illustrated in Fig. 3, we modify the training set by reducing the proportion of such data and incorporating explicit image references in the questions, thereby enhancing the model’s image-reading capabilities. For the test set, we filter out these questions and focus on evaluating the model’s ability to comprehend complex interactions between text and images based on specific queries. Through manual screening, we curate over 2,000 questions and selecte 700 high-quality questions to form the final IITC test set.

Context Construction.
We investigate two approaches for crafting extended multi-modal contexts. The first approach involves randomly merging images and their introductory paragraphs from various papers in the original SciGraphQA dataset into a long context designated as $<P^{k}_{i}M^{k}_{i}><P^{l}_{j}M^{l}_{j}>...<P^{m}_{n}M^{m}_{n}>$ where $k$, $l$ and $m$ denote different papers, and then posing questions on each image and its textual surroundings.
The second expands the text-image sequence from the original paper based on the paragraph $\{M_{i}\}^{N}_{i=1}$. The expanded sequence is denoted as $\{E_{i}\}^{N}_{i=1}$, which can maintain the structure of the paper and capture the inherent multi-modal context.

For the IITC task’s test set, we utilize the latter approach. During the training set construction, we assess models trained with both approaches, noting superior performance with the second, as mentioned in Sec. 4.3.
Beyond the advantage of consistency in data construction, we identify two key reasons for this outcome. First, the disparity in image-text content across different papers is significant, and the first approach does not effectively support the model’s ability to discern between relevant and distracting multi-modal information, thereby hindering its capability to accurately match questions with the correct image-text responses. Second, the brevity of the first mention paragraph $M_{i}$ often results in an insufficient textual context, impeding question comprehension and response.
Consequently, we select the second one for assembling the multi-modal long context for IITC.

We elaborate on the second approach of context construction.
Illustrated in Fig. 3,
we retrieve the paper’s Tex files from Arxiv based on the ‘id’ field from the original SciGraphQA dataset. We then match the ’caption’ field with TeX commands ‘\caption{.}’, ‘\label{.}’, and ‘\ref{.}’ to pinpoint the paper’s paragraph that first mentions the picture. Next, we expand this paragraph’s context by sequentially incorporating lines from the adjacent text. We intersperse 2 to 8 random pictures from the document into this text until achieving a specific context length.
To evaluate the model’s efficacy across varying token lengths, we craft two dataset versions capped at 4k and 8k tokens, ensuring a balanced token count distribution as depicted in Fig. 4.
To accurately represent the text-image spatial relationship, we maintain the sequence and placement of the inserted images as they appear in the TeX files. Lastly, we substitute the ’\ref{.}’ markers related to the embedded images with the notation "[Picture n]", clarifying the link between these references and their corresponding images.

Answer Modification.
IITC requires the model to specify the image it refers to in its response. To meet this requirement, we modify the answers in the original dataset. We replace the subjects in the original answers, such as ‘graph’ and ‘figure’, with the notation [Picture $i$](Figure $j$), where $i$ denotes the sequence number of the input image, and $j$ signifies the image’s index within the original document. Alternatively, we add ‘For more information, please refer to [Picture $i$](Figure $j$).’

Figure: Figure 4: The distribution of the number of images and tokens in the IITC subset of the VEGA dataset. The number of tokens for each image is 256.
Refer to caption: x4.png

#### 3.2.2 Subset for ITA

We craft the Image-Text Association (ITA) task to bolster the model’s proficiency in correlating text with images and to fortify its accuracy in image indexing responses.
For each image, we generate a textual context $T_{i}=E_{i}$ akin to the IITC task, extending the text around the first mention paragraph $M_{i}$, capped at 2,000 characters. Note that in the ITA task, the expanded context paragraphs $E_{i}$ does not include images outside of $P_{i}$.
We streamline the ITA task by pairing images with text from disparate articles, randomizing them, and prompting the model to reorder these pairs.
Despite the task’s complexity, leading-edge methods like Qwen-MAX [] and Gemini-1.0-pro [] exhibit only moderate success, particularly as the volume of pairs escalates, as evidenced in Table 3. At the same time, we find that direct training with expanded image-text pairs is insufficient for the model to master text-image associations. To mitigate the learning challenge, we implement a multi-scale training strategy in the ITA task. We design three textual scales for the training set, corresponding to the image caption $C_{i}$, the first mention paragraph $M_{i}$, and the expanded context paragraphs $E_{i}$. We also design two image quantity scales, with sets of three and five images. Experiments have shown that this multi-scale training set construction method can make the model more adaptable to the ITA task. In testing, we evaluate the model using solely the extended pairs, independent of other scaled data.

Figure: Table 1: The composition of the VEGA Dataset.
[Uncaptioned image]: x5.png

### 3.3 Dataset Statistics

The VEGA dataset is an extensive repository featuring 50k scientific literature entries, over 200k question-and-answer pairs, and a rich trove of 400k images. It includes the IITC subset, which is segmented into two categories based on token length: one supports up to 4,000 tokens, while the other extends to 8,000 tokens. Here, images are equated to 256 tokens each. Both categories offer roughly 200k training instances and approximately 700 meticulously curated, high-caliber test samples.
The ITA task is categorized into six divisions, with two dedicated to image quantity and three to text length. Table. 3.2.2 presents the train and test data statistics for the VEGA dataset, while Fig. 4 details the distribution of image numbers and token counts within the IITC subset, providing insights into the visual and textual context lengths.

### 3.4 Baseline Model

To further validate our dataset, we fine-tune the Qwen-VL-Chat [] model at two distinct maximum token lengths, 4k and 8k, training a dedicated model for each configuration, denoted as VEGA-Base-4k and VEGA-Base-8k. Given the IITC task’s complexity, we adopt a multi-task, multi-scale training regimen. We simultaneously train on the IITC and ITA tasks, categorizing them into varying difficulty levels based on image count and token length, as detailed in Fig. 4. This method aims to simplify the learning curve, bolster the model’s grasp of image-text correlations, and amplify its capacity to distill essential information. Our training utilizes roughly 340K data points. For a detailed data breakdown, please see our supplementary materials.

In line with the Qwen-VL-Chat fine-tuning protocol, we freeze the visual encoder while honing the language model and adapter module. We implement a cosine learning rate schedule starting at 1 × e-5 and leverage DeepSpeed Zero-3 for distributed training.
The VEGA-Base-4k utilizes a batch size of 8, taking up 31 A100 GPU days, whereas the VEGA-Base-8k, also running with a batch size of 8, totaled 48 A100 GPU days.

**Table 4: Analysis on the effectiveness of multi-task and multi-scale training strategies. Multi-task denotes jointly training on the IITC and ITA tasks, otherwise training solely on the IITC task. Multi-scale implies employing three textual scales during the ITA task training, otherwise using only the Extent Context as a single scale.**
| Multi-Task | Multi-Scale | IITC 4k |  | IITC 8k |  | ITA |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Rouge-L | BLEU | Acc |  | Rouge-L | BLEU | Acc |  | 3 pic | 5 pic |  |  |
| w/o | - | 0.501 | 0.247 | 0.838 |  | 0.478 | 0.220 | 0.766 |  | - | - |
| w | w/o | 0.502 | 0.247 | 0.833 |  | 0.485 | 0.228 | 0.779 |  | 0.155 | 0.004 |
| w | w | 0.508 | 0.252 | 0.858 |  | 0.488 | 0.228 | 0.789 |  | 0.998 | 0.992 |

**Table 5: Performance comparison of the base model on SciGraphQA and IITC 8k Task with different training sets. VEGA^* denotes the first context construction method for the IITC task that integrates images and their initial mentioning paragraphs from several papers, as detailed in Section 3.2.1.**
| Base model | Training Set | SciGraphQA |  | IITC 8k |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Rouge-L | BLEU |  | Rouge-L | BLEU | Acc |  |  |
| Qwen-VL-Chat | SciGraphQA | 0.538 | 0.290 |  | 0.406 | 0.153 | 0.000 |
| Qwen-VL-Chat | VEGA | 0.522 | 0.266 |  | 0.473 | 0.223 | 0.775 |
| Qwen-VL-Chat | VEGA^* | 0.507 | 0.241 |  | 0.415 | 0.160 | 0.521 |

## 4 Experiments

### 4.1 Experimental Setup

We evaluate the performance of the current state-of-the-art MLLMs on the IITC and ITA tasks. Since most advanced open-source MLLMs do not support multi-image inputs or have not been trained on multi-image data, we only test two open-source MLLMs: 1) InternVL [] and 2) Qwen-VL-Chat [], as well as four advanced proprietary models: 1) Gemini-1.5-pro [], 2) Gemini-1.0-pro [], 3) GPT4V [], and 4) Qwen-Max []. Finally, we evaluate the performance of our baseline models VEGA-Base-4k and VEGA-Base-8k. During the evaluating of proprietary models, a small fraction of issues arise where the models are unable to provide answers. These instances are excluded from the scoring statistics. When evaluating Qwen-Max, the API call is limited to a maximum token input of 6000, leading to the exclusion of some test data, considering the cost of evaluating, we standardized the input images by resizing them to $448\times 448$ pixels.

### 4.2 Main Results

Table 3 illustrates the evaluation results of the mainstream state-of-the-art MLLMs on the VEGA dataset, highlighting the considerable challenge posed by the IITC and ITA tasks. InternVL and Qwen-VL-Chat exhibit poor performance on both tasks, which can be attributed to these open-source models’ limited capability to follow instructions. Consequently, they struggle significantly with complex contextual inputs and directives inherent in these tasks.
Among the proprietary models, GPT4V achieves the highest image-relation accuracy on the IITC task. However, error analysis indicates that the primary issues include interference from similar images and instability in following instructions. Gemini-1.5-pro demonstrates exceptional accuracy in the ITA task, underscoring its robust image-text comprehension capabilities. In terms of text generation quality, Gemini-1.0-pro scores higher on Rouge-L and BLEU metrics compared to Gemini-1.5-pro and GPT4V. This discrepancy is due to two main factors: 1) Gemini-1.5-pro and GPT4V tend to summarize using their own language rather than directly incorporating descriptions from the original text, and 2) GPT4V’s outputs are often longer than the Ground Truth, disadvantaging it in automated evaluations. More qualitative evaluations of these proprietary models are given in the supplementary materials.
Our model, trained using Qwen-VL-Chat 7B on the VEGA dataset, achieves state-of-the-art results across all metrics in both tasks. This demonstrates the effectiveness of our dataset in enhancing the model’s capability to process interleaved image-text inputs.

### 4.3 Ablation Study

Multi-task and Multi-scale Learning. We investigate the impact of multi-task and multi-scale training strategies on model training. As shown in Table 4, our base model is Qwen-VL-Chat, where “Multi-Task” represents joint training on both the IITC and ITA tasks, while “without Multi-Task” indicates training solely on the IITC task. “Multi-Scale” denotes the use of a multi-scale training strategy in the ITA task, otherwise training is only conducted on the extended context scale of the ITA task. The experimental results show that the model employing both Multi-Scale and Multi-Task strategies improves the accuracy of image-text association by 2.3% compared to the model without these strategies, confirming the effectiveness of the multi-scale and multi-task training strategies. We also find that without multi-scale training, the model shows almost no performance improvement on the ITA task, with accuracy comparable to random guessing.
Observing the model’s failure case, it organizes the pictures and text chaotically. Such failure originates from two primary causes: First, the substantial volume of both text and images adds complexity to the task, challenging the model’s ability to grasp the connections between textual and visual contexts. This complexity hinders the model’s compliance with the directives for producing the correct output. Second, the complexity of ITA is attributed to the necessity of navigating multiple visual-textual relationships rather than identifying a single image in the answer. By adopting a multi-scale approach that streamlines the learning process during training, we have improved the model’s compliance with instructions, leading to a significant enhancement in its performance on the ITA task. Additionally, we evaluate the impact of different lengths of visual context on the model performance in Fig. 3.2.2.

Comparison of SciGraphQA and VEGA. We trained Qwen-VL-Chat using the SciGraphQA [] and VEGA datasets and tested the models’ performance on these two datasets. As shown in Table 5, the model trained with the VEGA dataset performed better on both tasks, while the model trained with SciGraphQA struggled with the IITC task. These results suggest that training with the VEGA dataset not only enhanced the model’s ability to handle long interleaved image-text inputs but also maintained its capability to process traditional VQA input patterns.

Comparison of Context Construction Methods. We assessed the impact of two context construction techniques on model performance for the IITC task. VEGA^* represents the first method that builds multi-modal context for the IITC task by integrating images and their initial mention paragraphs from several papers, as detailed in Section 3.2.1.
As shown in Table 5, the data reveals that VEGA^* underperforms on both SciGraphQA and IITC tasks compared to our baseline model, underscoring our method’s superiority. This inferior performance may stem from the fact that combining text and image context from the same paper could introduce extraneous material, inadvertently sharpening the model’s proficiency in discerning relevant information from noise during training.

## 5 Conclusion

In this study, we present the VEGA dataset, tailored to boost MLLMs’ Interleaved Image-Text Comprehension (IITC) in real-world applications. We employ the VEGA test set to evaluate leading MLLMs, exposing the stringent requirements of IITC tasks, particularly in multi-modal comprehension with a focus on instruction adherence. Our cutting-edge approach integrates multi-task and multi-scale learning, utilizing VEGA within the Qwen-VL-Chat framework to refine MLLMs’ ability to interleave image-text comprehension and accurately generate image indexes following instructions. This approach outstrips premier proprietary models like Gemini 1.5 Pro and GPT4V, proving VEGA’s potential to substantially improve IITC performance. Future enhancements to the dataset will involve embedding subtler instructions, increased image-text interactivity in answers, a wider array of data sources, and more cutting-edge evaluation methods for the IITC task.

## Appendix A Training Data

Data Composition During the training of the VEGA-Base model, we utilized the entire IITC subset of the VEGA dataset. As for the ITA subset, we incorporated all of the Expanded Context data, as well as half of the data from both the Captions and First Mentions. The training data distribution is visualized in Figure 5.

Figure: Figure 5: The composition of data for model training.
Refer to caption: x6.png

## Appendix B Question Filtering

In the construction of the VEGA dataset, we filtered out the questions

## Appendix C Case Study

This section presents the performance of VEGA-Base-4k, Gemini-1.5-pro, and GPT4V on the IITC 4k task. Fig. 6 and 7 illustrates cases where all three models failed, highlighting the challenging and complex nature of the IITC task. The models were misled by deceptive textual content, resulting in incorrect answers. Fig. 8 and 9 shows instances where all three models provided the correct response. Fig. 10, 11, 12 and 13 depict scenarios where VEGA-Base-4K answered correctly, while Gemini-1.5-pro and GPT4V did not. VEGA-Base-4k successfully navigated through distractions from similar images and irrelevant text, accurately interpreting the information in the images to arrive at the correct answers, demonstrating its robust capability to process interleaved image-text inputs.

Figure: Figure 6: The context for Case 1
Refer to caption: x7.png

Figure: Figure 7: Models’ answer and analysis for Case 1. The pictures referred in the Ground Truth are marked in blue, the incorrect images referred to in the model’s answer are marked in red, and the correct images are marked in green.
Refer to caption: x8.png

Figure: Figure 8: The context for Case 2
Refer to caption: x9.png

Figure: Figure 9: Models’ answer and analysis for Case 2. The pictures referred in the Ground Truth are marked in blue, the incorrect images referred to in the model’s answer are marked in red, and the correct images are marked in green.
Refer to caption: x10.png

Figure: Figure 10: The context for Case 3
Refer to caption: x11.png

Figure: Figure 11: Models’ answer and analysis for Case 3. The pictures referred in the Ground Truth are marked in blue, the incorrect images referred to in the model’s answer are marked in red, and the correct images are marked in green.
Refer to caption: x12.png

Figure: Figure 12: The context for Case 4
Refer to caption: x13.png

Figure: Figure 13: Models’ answer and analysis for Case 4. The pictures referred in the Ground Truth are marked in blue, the incorrect images referred to in the model’s answer are marked in red, and the correct images are marked in green.
Refer to caption: x14.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Please see the weaknesses section. Most of the weaknesses are minor. The weakness I care about and the one where an author response would cause me to increase my rating is "analyses". What analyses have the authors done on how properties of the dataset (e.g. number of images, context length, etc) affect the performance of MLLMs? Also, can the authors discuss why there are multiple evaluation metrics and what each of them signify?
soundness: 3
strengths: - There are no resources (as far as I know of) to study and characterize _long-context_ interleaved image-text comprehension or empirical studies of this comprehension ability. This is one of the first.
- The experimental evaluation is broad, covering multiple vision language models including a mix of closed source LLMs and open source LLMs.
- I especially appreciate the honest of the authors in showing that training on Vega _does not_ in general improve performance on other VQA tasks.
summary: Long-context interleaved image-text comprehension is an important skill required to read scientific papers, for example. Humans can do this well, but characterization of this ability is understudied. This paper introduces a multimodal task called Interleaved-Image-Text Comprehension which requires answering a question using interleaved image and text and grounding the answer in one of the in-context images. The authors develop a dataset called VEGA to evaluate models on this task and auxilliary simpler task called image-text association. They evaluate multiple frontier LLMs and some open LLMs on the task, and develop a training strategy which they show improves performance on this task.
weaknesses: - The construction method of the dataset seems a bit artificial.
- I don't see analyses of how the number of images / context length affects accuracy. In general, I feel like important analyses are not present. There are no "slices" in the dataset to sanity check that expected properties hold. For example, I would expect that as context increases, accuracy falls. If I did not see this, I would be suspicious of the construction method. This is just an example. My main point is that the dataset and how the properties of the dataset affect evaluation are not analyzed enough or perhaps not explained in the text.
- I am not convinced by the proposed training method. It is not really motivated or explained well in the text, and there is no discussion (as far as I can tell, but it is very possible I have missed something) on how it relates to other similarly named methods in the literature for "multi-scale training".

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 2
questions: Would you consider adding an out-of-distribution test set to further evaluate generalization capabilities?
soundness: 3
strengths: The introduction of a challenging new task (IITC) with extensive multimodal input length sets a unique standard in evaluating models' interleaved image-text comprehension. The proposed VEGA dataset is extensive, comprising 295k high-quality, multi-turn entries, with a curated test set of 700 questions to assess models' nuanced comprehension abilities.

The benchmark evaluates both proprietary and open-source models, showing that current models, even at their best, are still only moderately successful in IITC tasks. Fine-tuning Qwen-VL-Chat leads to meaningful improvements, establishing a strong baseline for future models.

The dataset structure, including partitions for 4k and 8k token contexts, is well-conceived and allows for detailed performance analysis. Including specific references to image indices in responses (e.g., '[Picture 2](Figure. 3)') is a thoughtful design choice, aiding interpretability in output.
summary: This paper introduces a novel task, Interleaved Image-Text Comprehension (IITC), which evaluates models' capabilities to handle complex interactions in lengthy multimodal contexts. The authors present the VEGA dataset specifically designed for the IITC task, featuring up to 8,000 tokens and 8 images. The paper also benchmarks several models on this task, including proprietary and open-source options, and demonstrates the benefits of fine-tuning Qwen-VL-Chat for IITC. The results show that even state-of-the-art models face challenges in this setting, highlighting the dataset's rigor and the task's demands.
weaknesses: Presentation: Figures, particularly Figure 1 and Figure 2, are too small, making the text nearly unreadable without significant zoom. This compromises the clarity and accessibility of the paper's visual aids.

Test Set Generalizability: The test set, created through a process similar to the training data generation, may inadvertently advantage trained models by being in-distribution. Introducing an additional test set, created independently or using different datasets, could strengthen the evaluation by ensuring generalization beyond the training data distribution.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: - During the construction of the VEGA IITC subset, particularly in the answer modification step (see line 305), the authors adjusted the answers from the original dataset (SciGraphQA). Was this modification based on rules (such as keyword matching) or aided by an LLM (e.g., prompt GPT)? Additionally, were the modified answers subjected to quality checks and evaluations, such as through manual sampling inspection and calculating the pass rate?
soundness: 3
strengths: - This paper introduces the Interleaved Image Text Comprehension (IITC) task and creates the corresponding dataset VEGA. IITC and VEGA have the following unique features: (1) the interleaved images and text input; (2) multiple images input; (3) the task requires the LMM to determine which image is relevant to the question; (4) the input includes image and text information that is irrelevant to the question.
These features reflect the normal state of human reading and understanding of image-text materials and are essential for a multimodal AGI system. Existing visual language datasets often lack these features, especially the latter two. Therefore, IITC and VEGA fill this gap to some extent and help to advance the technical progress of the research community in this direction.

- The overall design of the dataset is scientific and reasonable, the construction process is detailed, and the provided statistical information is very convincing. The adopted multi-scale, multi-task training strategy also appears reasonable and credible.

- The main body of the paper is well-structured and written in a clear and concise style, making the article easy to read and understand.
summary: This paper introduces the Interleaved Image Text Comprehension (IITC) task and establishes the corresponding dataset, VEGA. Compared to existing visual language tasks and datasets, IITC and VEGA have the following unique features: (1) the interleaved images and text input; (2) multiple images input; (3) the task requires the LMM to determine which image is relevant to the question; (4) the input includes image and text information that is irrelevant to the question. Particularly, the latter two features are lacking in previous tasks and datasets.

By benchmarking existing mainstream close-source and open-source MLLMs on VEGA, it was found that the performance of these SOTA  models is not ideal, thus confirming the challenging nature of the IITC task.

Furthermore, this paper also finetunes Qwen-VL-Chat using the train split of VEGA and adopts a multi-scale, multi-task training strategy, laying a solid baseline for subsequent collaborative research.
weaknesses: - All training-related experiments in the paper were conducted solely on the Qwen-VL-Chat model and did not involve other open-source foundation VL models that support interleaved text-image processing. Therefore, the training strategies proposed in this paper and the conclusions from its ablation study have not been sufficiently validated for generalization across other models.
- The article has some issues with formatting and layout, such as the text in some images being too small, which affects normal reading (e.g., Figure 1). Additionally, the layout of the appendix also needs further optimization.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: Other than the weaknesses:

1. Is multi-scale training still effective when not used for multi-task training?
2. In Table 4, what is the model’s performance when trained only on the ITA task with and without multi-scale training?
soundness: 3
strengths: 1. Proposes VEGA, a complex dataset specifically crafted for multi-image understanding, challenging MLLMs with interleaved image and text contexts.

2. Establishes two tasks, Interleaved Image-Text Comprehension (IITC) and Image-Text Association (ITA), setting new benchmarks for interleaved image understanding.

3. Demonstrates that the multi-scale training method effectively improves performance on the ITA task.
summary: The paper introduces VEGA, a dataset designed for interleaved image and text understanding. It also proposes a multi-modal large language model (MLLM) baseline fine-tuned on this dataset. The fine-tuning process uses a multi-scale method, gradually incorporating more complex samples. The model is fine-tuned for two tasks in the VEGA dataset: Interleaved Image-Text Comprehension (IITC) and Image-Text Association (ITA). It demonstrates improved performance on both tasks.
weaknesses: 1. Key baseline models such as LLaVA-interleaved and InternVL2 are missing, limiting insight into comparative performance.
2. Similar datasets like SlideVQA, MMLongBench-Doc, DUDE, and DocVQA are not discussed or tested, which could demonstrate the generalization of the fine-tuned model.
3. The ITA task may have limited real-world applicability, as not all text paragraphs have corresponding images. In contexts like research papers, only a subset of text relates to specific images, and filtering irrelevant text often implies inferred relevance between remaining text and images.
4. The font in Figure 1 is too small and should be increased for better readability.

Meta-review: All reviewers gave positive evaluations. The authors introduce a new task called Interleaved Image-Text Comprehension and provide the corresponding VEGA dataset to support this task. This task challenges models to understand complex interleaved image-text content, a valuable contribution to the vision-language research community. The proposed multi-scale training strategy effectively improves model performance. The authors should clarify certain aspects of the dataset creation and training methods to enhance the paper's clarity and comprehensibility. Overall, this paper is expected to inspire further research in this field.

Decision: Accept (Poster)