Learning Interleaved Image-Text Comprehension in Vision-Language Large Models
Authors: Chenyu Zhou
Mengdan Zhang
Peixian Chen
Chaoyou Fu
Yunhang Shen
Xiawu Zheng
Xing Sun
Rongrong Ji
Venue: ICLR 2025 Poster (2025)
Keywords: MLLMs;Benchmark;Interleaved Image-Text Comprehension
Primary area: datasets and benchmarks

Abstract: The swift progress of Multi-modal Large Models (MLLMs) has showcased their impressive ability to tackle tasks blending vision and language.
Yet, most current models and benchmarks cater to scenarios with a narrow scope of visual and textual contexts.
These models often fall short when faced with complex comprehension tasks, which involve navigating through a plethora of irrelevant and potentially misleading information in both text and image forms.
To bridge this gap, we introduce a new, more demanding task known as Interleaved Image-Text Comprehension (IITC).
This task challenges models to discern and disregard superfluous elements in both images and text to accurately answer questions and to follow intricate instructions to pinpoint the relevant image.
In support of this task, we further craft a new VEGA dataset, tailored for the IITC task on scientific content, and devised a subtask, Image-Text Association (ITA), to refine image-text correlation skills.
Our evaluation of four leading closed-source models, as well as various open-source models using VEGA, underscores the rigorous nature of IITC.
Even the most advanced models, such as Gemini-1.5-pro and GPT4V, only achieved modest success.
By employing a multi-task, multi-scale post-training strategy, we have set a robust baseline for MLLMs on the IITC task, attaining an $85.8\%$ accuracy rate in image association and a $0.508$ Rouge score. These results validate the effectiveness of our dataset in improving MLLMs capabilities for nuanced image-text comprehension.

Published as a conference paper at ICLR 2025
VEGA: LEARNING INTERLEAVED IMAGE-TEXI COM- PREHENSION IN VISION-LANGUAGE LARGE MODELS
Chenyu Zhou Mengdan Zhang", Peixian Chen Chaoyou Fu, Yunhang Shen
Xiawu Zheng' Xing Sun, Rongrong Ji'
Key Laboratory of Multimedia Trusted Perception and Efficient Computing. Ministry of Education of China, Xiamen University
ABSTRACT
The swift progress of Multi-modal Large Language Models (MLLMs) has show- cased their impressive ability tackle tasks blending vision and language. Yet, most current models and benchmarks cater to scenarios with a narrow scope visual and textual contexts. These models often fall short when faced with complex comprehension tasks, which involve navigating through a plethora of irrelevant and potentially misleading information in both text and image forms. To bridge this gap. we introduce new, more demanding task known as Interleaved Image- Text Comprehension (IITC). This task challenges models discern and disregard superfluous elements in both images and text to accurately answer questions and follow intricate instructions to pinpoint the relevant image. support of this task, we further craft new VEGA dataset, tailored for the IITC task on scientific content, and devised subtask, Image-Texx Text Association (ITA), to refine image-text correlation skills. Our evaluation of four leading closed-source models, as well as various open-source models using VEGA, underscores the rigorous nature IITC. Even the most advanced models, such Gemini-1.5-pp and GPT4V, only achieved modest success. By employing a multi-task, multi-scale post-training strategy, we have set robust baseline for MILLMs on the ITTC task, attaining an 85.8% accuracy rate in image association and 0.508 Rouge score. These results validate the effectiveness of our dataset in improving MLLMs capabilities for nuanced image-text comprehension.
INTRODUCTION
The swift advancement of Multi-modal Large Language Models (MLLMs) OpenAl et al.] (2024); Team et al. ) Chen al. (2023); Liu al. y0 Li al. (2023b) has recently showcased their remarkable aptitude for tackling vision-language tasks. These models have significantly improved in areas such as logical reasoning Yue et al. (2024); Han et al. (2023). high-resolution image comprehension Dong et al. (2024), In-Context Learning (ICL) Zeng et al. (2024); Baldassini et al.) (2024). and Chain of Thought (COT) processing Ge et al. (2023a). leading to notable achievements.
Despite these advancements, current applications typically engage MILMS in the basic multi-modal comprehension task, which involves posing questions tied to narrow scope of visual content. This approach falls short when compared to human daily comprehension.
Human comprehension, particularly in document analysis. presents unique challenges not found in basic comprehension tasks such as Visual Question Answering (VQA) tasks Antol et al. (2015). As shown in Fig. these challenges include: 1) Interleaved Text-Image Distraction: MLLMs are forced to find the correct context from irrelevant text- text-image mixtures. 2) Long-Form Content: The inputs consist of intertwined sequences of images and text, demanding comprehension over long mul- timodal context. 3) Image- Referenefeee Answers: The model is tasked with identifying and linking the relevant image and text based on specific instructions before crafting an appropriate response.
"Equal Contribution
"ooresponniig Author Project Leader

---

Published as a conference paper at ICLR 2025
Existing Interleaved Image-text QA Tasks
Existing VQA Tasks
Context:
<image> The diagrams below show two pure samples of in iden- tical closed, rigid containers. colored represents one
Context:
William
garden simiar to the blue trapszo- in the following image <image1>. price per unit given the following figure <imape2>
Materiid
Price
Genin
Compare the averaee kinetic energies..
krttody bazgges
penamial grgges
S1
79
Q: What would cost
Anroe 1i8mn
A: The samples have the same temperature.
line fescut
Jocce
75
15
building this platform?
A: 18975.00.
Interleaved input
Answer with image reference
Interleaved Input
Answer image reference
image Inelevant
Multi-magee
Irrelevant mouti information
Proposed Task
Such techniques can be dessified into two categorics Iustratedd F qure IB....
Picture 3
Q: "What the difierence betw- modes?"
Picture
multiaame
function
approaches compared, the multi models, priority
address coupled
"he [Picture FFFguue model
Picture
Figure 4 shows how the Bis adapt the amount offloaded suffic to the unikeessed charnels based on the number of WAPs that are considered networt. in fart, the ability of axteptation network changes at the Bis alloss to show how the proposed mult game
WAPs, whiee
WAPS
Interleaved input
Answer with image reference
Multi-image
C Irrelevant multi multi-madal information
Relevant multi modal information the question
modal information question
Figure 1:
Comparison between existing VQA and IITC tasks. While existing VQA tasks use single images with relevant context, ITTC incorporates multiple images and extended text with both relevant and irrelevant information. Models must specify reference images in their responses.
Recent prominent open-source MLLMs, such as LLaVA 1.5 Liu et al. 2023), CogVLM Wang et al. (202b)), and PaperOwl Hu et al. (2023), lack support for multiple image inputs and struggle with such intricate comprehension task. Multi-image, multi-turn models Chen et al. (2023) and video-based MLLMs Li et al. (2024) face challenges in concurrently processing extended visual and textual contexts effectively. Additionally, benchmarks for evaluating such an intricate task are limited, obscuring our understanding of MLLMs' practical comprehension abilities.
To bridge this gap. we introduce a novel multi-modal task. Interlcaved Image- image-Text Text Comprehension (IITC), which involves locating the relevant text and image within complex context given question, providing an accurate answer, and outputting the corresponding image index. Meanwhile, we present the first benchmark MI.LM to handle the above challenges within the ITTC task.
Specifically, we observe that IITC is a complex task where simply increasing the number of Interleaved Image-Text QA samples insufficient for accurate comprehension. Therefore, we propose utilizing multi-task learning strategy to enhance the model's understanding capabilities: employing the Image-Texx Association (ITA) auxiliary task to strengthen the model's 's ability to accurately locate image-text paragraph correspondences. In addition, we introduce multi-scale training strategy, constructing the training data with progressively increasing context lengths and image numbers to enhance the model's robustness against gradually increasing redundant image-text content.
Ultimately, we have developed novel dataset, designated as VEGA. It is comprised of two subsets, one tailored for the IITC task and another for the ITA task. The longest interleaved image-text content in VEGA reaches up to 8 images and 8k tokens. We design the instruction of the IITC task to be question about only of the images, requiring the model to specify the image it refers in
2

---

Published as a conference paper at ICLR 2025
its answer. We assess the model'ss interleaved image-text reading comprehension ability by both the correct rate of associated images, and the text quality of the answer by ROUGELin (2004) and al. (2002). We have evaluated several oftte-of-theaar MLLMs on our dataset. validating the challenge of our tasks. Furthermore, we have fine-tuned the Qwen- Qwen-VI--Cha model Bai
BLEU Papineni
al. (2023) the VEGA dataset set robust baseline for the IITC task.
In conclusion, this study makes several noteworthy contributions to the field:
Identifying Novel Task. We introduce a novel Interleaved Image-Text Comprehen- sion (ITC) task. It evaluates and enhances MLLMs' capabilities of following complex instructions and extracting key cues in long and interleaved image-text scenarios.
Introducing the VEGA Dataset. We develop new VEGA Dataset for the IITC task that enables a comprehensive understanding of scientific literature, whose multi-modal context reaches up to 8,000 tokens in length and contains up to images.
Benchmarking MLLMs in ITT. We utilize the VEGA dataset to appraise the ITTC capabilities current state-of-thee the-art models, including GPT4V, Semini-1.5-ppa and Qwen- VL-Chat, experimentally validating the challenge of IITC.
Enhancing MLLMs' IITC Capabilities. We fine-tune the Qwen- VL-Chat model on the VEGA dataset with multi-scale, multi-task training strategy and our experimental findings demonstrate that it achieves an image association accuracy rate of 85.8%. This represents a significant improvement and establishes strong baseline for the IITC task.
2
RELATED WORK
2.1 MLLMS
Recently, there has been notable increase in the deployment of MILMs aimed at tackling more complex tasks that involve multiple forms of media. These advanced models are equipped to understand and interpret not just text, but also visual content like images and videos Ge al. (2023b). For instance, BLIP-2 2 Li et al. (2023)) features the Q-Former, key component that establishes a link between a static LLM and visual data, showing impressive results in VQA tasks. InstructBLIP Dai al. (2024) is specifically fine-tuned using a variety of instruction-basee datasets, which improves its understanding of visual scenes and dialogues. Multi-modal-CoT Zhang et al. (2023) brings the concept of chain-of-thought Wei et al. (2022) into the multi-modal domain, demonstrating strong performance on the ScienceQA benehmark Lu et al. (0022). LLaVA Liu et al. (2024b) uses a linear approach and fine-tunes the entire LLM to enhance its effectiveness. The LLa Liu et al. (2024a), compared to L.L.V--155. quadruples the pixel resolution of input images, improves visual guidance for data blending, and offers enhanced visual reasoning and OCR capabilities. In this work, our focus is on the ITCC task, so we concentrate only on models that support multi-image input, such as Qwen-VI Bai al. (2023) and Intern VL Chen et al. (2023). We use LL-Chat the base model for training to establish a baseline for the IITC task.
2.2 MLLM BENCHMARKS
The evolution of MILMs, driven by multi-modal pretraining and instruction tuning, has outpaced traditional benchmarks like visual question answering and image captioning. New benchmarks have emerged assess aspects such OCR capabilities, adversarial robustness, and hallucination
susceptibility. POPE Li et al. (023c)), HaELM HaELM Wang et al. (2023a). LAMM Yin et al. (2023), LVLM-eHub Xu al. (2023). and MM- H.M-Vet|Ye Vet al. (2023) provide insights into these areas and give holistic view of MLLM's performance.
Vision-language benchmarks like Winoground Thrush et al. (2022) RAVEN Zhang et al. (2019)
OK-VQA Marino et al.) (2019), and VCR Zellers et al. (019) aim to measure MLLMs nuanced reasoning abilities. TextVQA Wright et al. (2010), FigureQA Kahou et al. (017)), ScienceQA Lu et al. (2022), and Math Vista Lu et al. (2023) contribute to understanding MLLMS domain-specifie reasoning. Comprehensive benchmarks like MME et al. (2024). MMBench Fu et al. (2024), and SEED-Bench cover variety of reasoning skills. However, these benchmarks only evaluate model performance in scenarios with limited single-image context. To address this limitation, we created the VEGA dataset.
3

---

Published as a conference paper at ICLR 2025
(a)IITC
(b) ITA
Task Prompt
Documents
Segnents
Based the following known information, answer the user's quesli-cus concisely and professionally The arswer is dadived from an image, please indicito the answer by referncinng PPictimel 1I whom reprisents the imtaee number ecemple (Phonue 21PPgoere Please only thoose
Decuments: <Text I><Image Is... Text no-dmage m²
%
Question: "h is the batch szze and leaming rate larger the SSCD run?
7 Pictures
Task Prompt
Documents
" "Pizzee determine the cxrrapenneene betwoen the pictures and lext besed the cunieti, provited "jiclures and segments' Please use Piolue reppod. regresents indes
Relevant image [jicurr
fallowing format Pictom 2| Pictun (Pctrr))
Decuments: <Image <Image Text I>., n
What concspending onder
segments
O utput
Oulput
Answer: "The [Picture ((Pigum 4) slows that butch size
Arswer: (Pictue 5] T2: (Pictrr 1) P3/Pacuuee 4] [Pictue 2] T5 (Fssurr 37
The task definition of ITT and ITA tasks. (a) The ITT task takes long interleaved image-text content as input and requires the model to specify the image it refers to in its response. (b) The ITA task takes shuffled images and text segments from different articles input and requires the model to output the relationship between the text and the images. <Text *> and <Image *> represent text segment and an image, respectively. They are both tokenized and fed into the model along with the task prompt and the question.
Figure 2:
PROPOSED METHOD
In this section, we introduce our VEGA dataset by first elaborating the task definition (3.I), followed by outlining the process of construction (3.2) and statistics (3.3). Finally, we present the training details of our baseline MLLM model (3.4.
3.1 TASK DEFINITION
Interleaved Image-Text Comprehension (IITC). Our VEGA dataset meticulously designed address the challenging ITTC task, with the goal of advancing the real-world comprehension capabilities of MLLMs. As demonstrated in Fig. [ an MLLM is presented with the complex task of discerning relevant text and images in response
given question. The model is expected to only deliver an accurate answer but also to identify the specific image index related to the response. This requirement extends beyond the traditional boundaries of VQA tasks Antol al. (2015).
To facilitate this process, we have developed a specialized prompt that directs MLLM to carry out the desired instruction. The complete input for MLLM is organized as shown in Fig. (a). We utilize the notation "[Picture n]" within the text to establish clear reference to the n-th image, whether appears in various sections an article or across different examples. This approach enables the model to learn consistent and standardized format for image referencing.
In the ITTC task, the crux of the challenge lies in the model's 's proficiency in accurately linking the appropriate images with the corresponding textual information as dictated by the instructions. This precise pairing is crucial, for it the foundation upon which a coherent answer is constructed. To quantitatively gauge the model's aptitude for this association, we have crafted each question pertaining to a specific input image, necessitating that the model explicitly reference this image as part of its response.
The model's prowess in image-text association is quantified by its success rate in identifying the correct image. Meanwhile, the textual response's quality is evaluated using established linguistic
4

---

Published as a conference paper at ICLR 2025
Quuestin Fillering
Mudti-modal ContextConstrudd Construction
Answer Moditicaton
Question What en the upliniced price tale appcowch and the fieed price role apgroen??
betwo-
Answer The findings in the graph sigge St. that the stundard devintion
A Specific Refurence
2: Whtal some ons the Tensor Cores are not being utilived mom offetiivly?
Tes. File Parse
Answer 1:
The findings in the [Picture 2
174
More Emages Text
(Flgure. suggest that the standard devia- tion
Question What is the goal of graght?
A Anorer The NME noulis detictting facial landmaris.
Add Speciflc Reference
Pind retermnc
Add Specific Reference
A Answer The KME reults detecting facial landmarks. For information please [Pisture fjFFigure
Question 3: What is the matn goal of the Figure. 37
The construction process of the VEGA dataset. We use the SciGraphQA dataset foundation, to which we add more images and text to its context, modify questions and answers that lack clear image references, and incorporates references meet the requirements of the IITC task.
Figure 3:
metrics, namely the ROUGE Lin (0044) and BLEU Papineni al. (2022) scores. Additionally, inspired by LLaVA-Bench LivechLLiuaee et al. (2023), ) we also evaluated the text quality using the advanced LLM 1--tmaa--.--00--nntttee appendix. Together, these measures provide comprehensive assessment of the model's performance on the IITC task.
Dubey al. (2024), the prompt temple we used could be found in the
Image-Text Association (TTA). It is the auxiliary task designed to sharpen MLLM's precision in aligning text paragraphs with their relevant images. As illustrated in Fig. 2bb), ITA challenges the model with a shuffled array of inputs: (Pi.... Pa. Ti................ Tn). Each image. labeled as P. from i i= n, is linked to a unique article, while each text snipper T: provides context for its image P, The textual inputs are intentionally scrambled at the outset. ITTA's goal is for the model to accurately match and declare the correct pairings of each text segment T; with its image P.. as demonstrated in Fig. 2()). Compared to the IITC task, ITA places greater emphasis on the model's ability to associate images with text. It is anticipated that training with ITA will bolster this skill within the model, leading improved performance on the ITC task. For the ITA task evaluation, with set of n images, we count the number of correctly associated image-text pairs m, and use the ratio m/ n as the performance metric for the ITA task.
3.2 DATA COLLECTION
We develop the VEGA dataset upon the foundation of SeiGraphQA & Tajbakhsh (023). Sci- GraphQA is a multi-turn question-answeriin dataset for scientific graphs, containing 295k high- quality, multi-turn data entries. It collected part of the papers published on arXiv from 2010 to 2020, and extracted the pictures (denoted by [P.) is the total number of pictures), the captions ((C,) ((C),N) the pictures, and the first paragraphs ([M:) mentioning the pictures from the papers.
As outlined in Fig.Bl our enhancement involves meticulous process of questions filtering that pertain to the visual-textual context, constructing long visual and textual context, and formulating answers that directly reference images. The VEGA dataset is structured into two subsets, each specifically curated to train models on the IITC and ITA tasks, respectively.
3.2.1 SUBSET FOR HTT
Question Filtering. In the original SciGraphQA dataset, some questions about images are quite general, such as "What are the implications of the results shown in the graph?" and "What does the x-axis and y-axis of the graph represent?" For more examples, please refer to the supplementary materials. However, when these questions placed within the context of multi-image long articles, they encounter issues with ambiguous image references. Furthermore, the answers are heavily dependent the image content, rather than the multi-modal context information, which is not
5

---

Published as a conference paper at ICLR 2025
aligned with the objectives of the IITC task. To address this, as illustrated in Fig. B modify the training set by reducing the proportion of such data and incorporating explicit image references in the questions, thereby enhancing the model's image-reading capabilities. For the test set, filter out these questions and focus on evaluating the model's ability to comprehend complex interactions between text and images based on specific queries. Through manual screening, we curate over 2,000 questions and selecte 700 high-quality questions to form the final IIT test set.
Context Construction. We investigate two approaches for crafting extended multi-modal contexts. The first approach involves randomly merging images and their introductory paragraphs from various papers in the original SciGraphQA dataset into a long context designated < P.. M >< PJM > < P M > where k. I and m denote different papers, and then posing questions on each image and its textual surroundings. The second expands the text-image sequence from the original paper based on the paragraph ( (M:) The expanded sequence is denoted as {Fi J.M which can maintain the structure of the paper and capture the inherent multi-modal context.
For the IITC task's test set, we utilize the latter approach. During the training set construction, we assess models trained with both approaches, noting superior performance with the second, as mentioned in Sec. 4 Beyond the advantage of consistency in data construction, we identify two key reasons for this outcome. First, the disparity in image-text content across different papers is significant, and the first approach does not effectively support the model's ability to discern between relevant and distracting multi-modal information, thereby hindering its capability to accurately match questions with the correct image-text responses. Second, the brevity of the first mention paragraph M, often results in an insufficient textual context, impeding question comprehension and response. Consequently, we select the second one for assembling the multi-modal long context for IITC.
We elaborate on the second approach of context construction. Illustrated in Fig. 3. we retrieve the paper's Tex files from Arxiv based on the 'id' field from the original SciGraphQA dataset. We then match the 'caption' field with TeX commands 'ccaption_.]', 'llaelf.)'' and "ref{. J' to pinpoint the paper's paragraph that first mentions the picture. Next, we expand this paragraph's context by sequentially incorporating lines from the adjacent text. We intersperse 2 to 8 random pictures from the document into this text until achieving a specific context length. To evaluate the model's 's efficacy across varying token lengths, we craft two dataset versions capped at 4k and 8k tokens, ensuring balanced token count distribution as depicted Fig. 4 To accurately represent the text-image spatial relationship. maintain the sequence and placement of the inserted images as they appear in the TeX files. Lastly, we substitute the 'Iref(.]' markers related to the embedded images with the notation "[Picture n]", clarifying the link between these references and their corresponding images.
Answer Modification. ITC requires the model to specify the image it refers to in its response. To meet this requirement, we modify the answers in the original dataset. We replace the subjects in the original answers, such as 'graph' and 'figure, with the notation [Picture (Figure j), where i denotes the sequence number of the input image, and j signifies the image's index within the original document. Alternatively, we add 'For more information, please refer to [Picture i] j).
3.2.2 SUBSET FOR ITA
We craft the Image- Text Association (ITA) task to bolster the model's proficiency in correlating text with images and to fortify its accuracy in image indexing responses. For each image, we generate a textual context T: E, akin the IITC task, extending the text around the first mention paragraph M,, capped at 2,000 characters. Note that in the ITA task, the expanded context paragraphs E, does not include images outside of P,.
We streamline the ITA task by pairing images with text from disparate articles, randomizing them, and prompting the model to reorder these pairs. Despite the task's 's complexity, leading- methods like Qwen-MAX C0AMMMXX(ee---e VL. VL. Team (024) and Gemini-1.0- 1.0--ro Team et al.) (2024) exhibit only moderate success, particularly as the volume of pairs escalates, as evidenced in Table 2 At the same time, we find that direct training with expanded image-text pairs is insufficient for the model to master text-image associations. To mitigate the learning challenge, we implement a multi-scale training strategy in the ITA task. We design three textual scales for the training set, corresponding to the image caption C, the first mention paragraph M,, and the expanded context paragraphs E, We also design two image quantity scales, with sets of three and five images. Experiments have shown that
6

---

Published as a conference paper at ICLR 2025
(a) Image Number Distribution
Token Number Distribution
80
Train 4k
Frequency (%)
Teat 8k
Train Sk
Frequency (%)
60
Test
Trin S
40
10
30
20
10
0-1k
1-2k
2-3k
3-4k
5-6k
6-7k
7-8k
Image Number
Token Number
Figure 4: The distribution of the number of images and tokens in the IICC subset of the VEGA dataset. The number of tokens for each image is 256.
this multi-scale training set construction method make the model more adaptable the ITA task. In testing, we evaluate the model using solely the extended pairs, independent of other scaled data.
VISSA VBGAABBac-4 Boe-4k
GPTAY
Subset
Scale
Train
Test
Total
Gentini-1. 1.0
4k Tokens
208103
700
208803
ITC
8k Tokens
196947
700
197647
Accurscy
3 Pic C.
23991
23991
3 M.
23993
23993
3 Pic E.
47983
500
48483
ITA
23986
23986
5 Pic M.
23985
23985
45226
486
45712
Image Number
Table
The composition the VEGA
Dataset.
Figure 5: The relationship between accuracy and number of images the ITTC task.
3.3 DATASET STATISTICS
The VEGA dataset is an extensive repository featuring 50k scientific literature entries, over 200k question-and-answe pairs, and a rich trove of 400k images. includes the ITTC subset, which is segmented into two categories based token length: one supports up 4,000 tokens, while the other extends to 8,000 tokens. Here, images are equated to 256 tokens each. Both categories offer roughly 200k training instances and 700 meticulously curated, high-caliber test samples. The ITA task is categorized into six divisions, with two dedicated to image quantity and three to text length. Table. presents the train and data statistics for the VEGA dataset, while Fig. 4 details the distribution of image numbers and token counts within the IITC subset, providing insights into the visual and textual context lengths.
3.4 BASELINE MODEL
To further validate our dataset, we fine-tune the Qwen-VL-Chat Bai et al. (2023) model at two distinct maximum token lengths, 4k and 8k, training dedicated model for each configuration, denoted
7

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation results for various MLLMs. Models that fine-tuned and have the best performances are indicated with underlines. "LB." refers to the LLaVA Bench score, "LB. T indicates the LLa VA Bench score when the image is correct, and "LB.F" represents the LLaVA Bench score when the image is incorrect.
ITC 4k
ITTC Sk
ITA
Model
Rouge-L
BLEU
LB.
LB.T
LB.F
Rouge-L
BLEU
LB.
pic
5 pic
InterVVL-115
0.378
0.128
0.364
6.37
6.74
6.16
0.374
0.126
0.292
6.19
0,238
0.014
Owen Owen-VV..Ce VL.-Chat Chat
0.323
0.093
0,002
4.76
5.21
4.76
0.250
0.065
0.001
3.98
0.043
0.000
Owen2-VT.Al In-Istret
0431
0.137
0.017
6:55
6.92
6.54
0.393
0.124
0.006
6.36
0.385
0.029
Qwen-MAX
0.356
0.107
0.684
5.20
5.47
4.60
0.343
0.107
0.365
5.05
0.231
0.017
Gemini-1.Uppo
0.415
0.138
0.649
5.91
6.19
5.39
0.404
0.132
0.501
5.81
0,570
0.092
Somini-1.5-ppr
0.354
0.087
0.753
6.86
7.10
6.11
0.363
0.090
0,737
7.03
0.956
0.891
GPT4V
0.342
0,091
0.805
7.24
7.43
6.47
0.322
0.085
0.752
7.13
0.912
0,538
VEGA-Base-4k
0.508
0.252
0.858
5.65
5.87
4.31
0,488
0.228
0.789
5,43
0.998
0.992
15GAA-Bsc-88
0.492
0.241
0.845
5.58
5.83
4.23
0.473
0.223
0.775
5.41
0.994
0.936
Ground Truth
7.67
7.78
Random Guess
0.227
0.227
0.167
0.008
as and
For more training detail, please see our supplementary
materials.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We evaluate the performance of the current state-of-the-an art MLLMs on the IIT and ITA tasks.
We test three open-source MLLMs: 1) InternVL-11 13BChen et al. (2023). 2) Qwen-Vil VL-Chat 7B Bai et al. (002)), and 3) Wyen2-VL-Inntruue 7B Wang et al. (2024), as well as four advanced proprietary models: 1) Gemini- 1.5-pro Team et (2024), 2) Gemini-1 1imiimi-1..-pro0 Team et al. (2024), 3) GPT4V OpenAl et al. (2024), and 4) Qwen-Max Qwen- VL Team (2024). Finally, we evaluate the performance of our bascline models
and VEGA-
4.2 MAIN RESULTS
Table[[presente the evaluation results of mainstream sate-of-the-an MLLMs the VEGA dataset, highlighting the significant challenges posed by the IITC and ITA tasks. Intern Qwen-VL VL-Chat, and Owen2-VL-IInsrruce exhibit poor performance in the ITA task and image accuracy the IITC task, which can be attributed to these open-source models' limited ability to follow instructions. As a result, they struggle significantly with the complex contextual inputs and directives inherent in these tasks.
Among the proprietary models, GPT4V achieves the highest image-relation accuracy the ITTC task. However, eтror analysis reveals that the primary issues include interference from similar images and instability in following instructions. Gemini-l.5-pre demonstrates exceptional accuracy in the ITA task, underscoring its robust image-text comprehension capabilities. In terms of text generation quality, Gemini-11 1. 1.0-pro scores higher on the ROUGE-L and BLEU metrics compared to Gemini- 1.5-pro and GPT4V. This discrepancy arises from two main factors: (1) Gemmini-1.5-pp and GPT4V tend to summarize using their own language rather than directly incorporating descriptions from original text, and (2) GPT4V's outputs are often longer than the Ground Truth, which disadvantages it in automated evaluations. Nonetheless, both Gemini- 1.5-pro and GPT4V achieve higher LLaVA Bench scores.
We also collected the average scores each model in the ITC 4K task under correct image selection and incorrect image selection, denoted as LB.T and LB.F, respectively. The LB.T score is much higher than the LB.F score, indicating that image-text association is crucial for the quality of model answers.
Our model, trained using Qwen-VL-Chan VL-Chat 7B on the VEGA dataset, achieves state-of fi--the-rd art results across all metrics in both tasks, except for the LLaVA Bench score. After training, Qwen-VI.-
8

---

Published as a conference paper at ICLR 2025
Table 3: Analysis of the impact of VEGA general VQA performance. The base model we used is Qwen-VL-Chat The VQA dataset comprises training data from VQAv2, TextVQA, and ChartQA in 1:1:1 ratio. When training with both the VEGA and VQA datasets, the data ratio is 1:1.
ITC 4k
Model Name
Training
MMMU
MME
ChartQA
Rouge-L
BLEU
Acc
LB.
Qwen-VIe VL-Chat
0.330
1430.40
48,40
0.323
0.093
0.002
4,76
Qwen-VEGA VEGA
VEGA
0.320
1386.05
64.00
0,498
0.247
0.836
5.59
Qwen-VQA VEGA-VQA
VQA VEGA&VQA
0.340
1472.11
61.12
0.063
0.001
0.044
2.21
0.336
1460.21
61.52
0.490
0.235
0.823
5.37
Table 4: Analysis on the effectiveness of multi-task and multi-scale training strategies.
ITTC 4k
ITTC 8k
ITA
Multi-Task Multi-Scale
Rouge-L
BLEU
Acc
Rouge-L
BLEU
Acc
pic
pic
w/o
0.501
0.247
0.838
0.478
0.220
0.766
w/o
0.502
0.247
0.833
0.485
0.228
0.779
0.155
0.004
W
0.508
0.252
0.858
0.488
0.228
0.789
0.998
0.992
Chat's LLaVA Bench score improved from 4.76 to 5.65, surpassing the closed-source model Qwen Max. This demonstrates the effectiveness of our dataset enhancing model's s ability to process interleaved image-text inputs.
4.3 ABLATION STUDY
Impact of VEGA on General VQA Performance To investigate the impact of the VEGA dataset on the model's general VQA capabilities, we utilized VLMEvalKit Duan et al. (202)) to test the model's performance on MMMU Yue et al. (2024), MME Fu et al. (2024), and ChariQ Masry et al. (2022).
Based on the experimental results in Table we draw the following conclusions:
1. Qwen-VEGA shows some decline in performance MMMU and MME compared to Qwen- VL-Chat. This can be attributed to two main factors: 1) Qwen- VEGA focuses on enhancing the model's ability understand long-form text-image content within the VEGA dataset (specifically in the scientific paper domain), which may negatively impact its VQA capabilities in other scenarios. The improved performance of VEGA-VQA suggests that this issue be mitigated through joint SFT training. 2) After training on the VEGA dataset, Qwen-VErt VEGA tends to produce longer outputs. Although VLMEvalKit has powerful post-processing code for answers, there are still cases where the scores are lower. (In the MME test, the average output length is Qwen-VEGA: 102.32; Qwen-VL- Chat: 3.00; VEGA-VOA: 2.57).
2. Qwen- VEGA achieves high scores on ChartQA, likely due to the large number of chart-containing images in the VEGA dataset. We believe that training on the VEGA dataset enhances the model's ability to interpret chart images.
The similar scores REGA-VQA and Qwen-Vor indicate that incorporating the VEGA dataset into training does not compromise the model's s VQA capabilities.
In summary, we recommend using the VEGA dataset for multi-task joint training with other multi- modal datasets. This approach can enhance the model's ability to handle mixed text-image inputs while preserving its traditional VQA capabilities, making it more adaptable to a broader range of instructions and improving its generalization abilities.
Multi-task and Multi-scale Learning. We investigate the impact of multi-task and multi-scale training strategies model training. As shown in Table 4 our base model is Qwen-VL-Chat, where "Multi-Tass represents joint training on both the ITT and ITA tasks, while "without Multi-Ta Task" indicates training solely the IIC task. "Multi-Scale" denotes the of a multi-scall training strategy in the ITA task, otherwise training is only conducted on the extended context scale of the ITA task. The experimental results show that the model employing both Multi-Scale and Multi-Ta Task

---

Published as a conference paper at ICLR 2025
Table 5: Performance comparison of the base model SeiGraphQA and ITTC 8k Task with different training sets. VEGA" denotes the first context construction method for the IITC task that integrates images and their initial mentioning paragraphs from several papers, as detailed in Section 3.2.1
SciGraphQA
ITT 8k
Base model
Training Set
Rouge-I.
BLEU
Rouge-I.
BLEU
Acc
Qwen- Vh-Chat
SciGraphQA
0.538
0.290
0.406
0.153
0.000
Qwen-VL-Che VL-Chat
VEGA
0.522
0.266
0.473
0.223
0.775
Owen-VL-Chat
VEGA*
0.507
0.241
0.415
0.160
0.521
strategies improves the accuracy of image-text association by 2.3% compared to the model without these strategies, confirming the effectiveness of the multi-scale and multi-task training strategies. We also find that without multi-scale training, the model shows almost no performance improvement on the ITA task, with accuracy comparable to random guessing. Observing the model's failure case, organizes the pictures and text chaotically. Such failure originates from two primary causes: First, the substantial volume of both text and images adds complexity to the task, challenging the model's ability to grasp the connections between textual and visual contexts. This complexity hinders the model's 's compliance with the directives for producing the correct output. Second, the complexity of ITA is attributed to the necessity of navigating multiple visual-textual relationships rather than identifying a single image in the answer. By adopting a multi-scale approach that streamlines the learning process during training, we have improved the model's 's compliance with instructions, leading to a significant enhancement its performance on the ITA task. Additionally, we evaluate the impact of different lengths of visual context the model performance in Fig.
Comparison of SeiGraphQA and VEGA. We trained Qwen-VL.-Che Vh.-hat using the SciGraphQA Li & Tajbakhsh (2023) and VEGA datasets and tested the models' performance these two datasets. As shown in 5 the model trained with the VEGA dataset performed better on both tasks, while the model trained with SciGraphQA struggled with the IITC task. These results suggest that training with the VEGA dataset not only enhanced the model's s ability to handle long interleaved image-text inputs but also maintained its capability to process traditional VQA input patterns.
Comparison of Context Construction Methods. We assessed the impact of two context construction techniques on model performance for the IITC task. VEGA" represents the first method that builds multi-modal context for the IITC task by integrating images and their initial mention paragraphs from several papers, as detailed in Section 3.2.1 As shown in Table 5, the data reveals that VEGA' underperforms on both SciGraphQA and IITC tasks compared to our baseline model, underscoring our method's superiority. This inferior performance may stem from the fact that combining text and image context from the same paper could introduce extraneous material, inadvertently sharpening the model's proficiency in discerning relevant information from noise during training. The creation of the VEGA dataset goes beyond mere data accumulation; We carefully selected questions and constructed contexts based on the challenges faced in reading comprehension applications. This thoughtful design leads to more effective enhancement of the model's ability to associate images and text, well as its reading comprehension ability.
CONCLUSION AND LIMITATION
In this study, we present the VEGA dataset, tailored boost MLLMs' Interleaved Image-Tes Text Comprehension (IITC) in real-world applications. We employ the VEGA test set to evaluate lcading MLLMs, exposing the stringent requirements of ITTC tasks, particularly in multi-modal compre- hension with focus on instruction adherence. Our cutting-edge approach integrates multi-task and multi-scale learning, utilizing VEGA within the Qwen-VL-Chat framework to refine MLLMs' ability to interleave image-text comprehension and accurately generate image indexes following instructions. This approach outstrips premier proprietary models like Gemini 1.5 Pro and GPT4V, proving VEGA's potential to substantially improve IITC performance. Future enhancements to the dataset will involve embedding subtler instructions, increased image-text interactivity in answers, a wider array of data sources, and more cutting-edge evaluation methods for the IITC task.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported by the National Science Fund for Distinguished Young Scholars (No. 2o.662026633,, the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. U23A20383, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), and the Natural Science Foundation of Fujian Province of China (No. 2021J06003, No. No.2022106001.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv, 2023.
Folco Bertini Baldassini, Mustafa Shukor, Matthieu Cord, Laure Soulier, and Benjamin Piwowarski. What makes multimodal in-context learning work?, 2024.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvi: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv, 2023,
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards models with instruction tuning. NeurIPS, 2024.
vision-language
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang. Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. setemmmmxxcopooerr2-aae
A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv, 2024.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. The llama 3 herd of models, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
Jiaxin Ge, Hongyin Luo, Siyuan Qian, Yulu Gan, Jie Fu, and Shanghang Zhang. Chain of thought prompt uning in vision language models, 2023a.
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making Ilama see and draw with seed tokenizer, 2023b.
Xiaotian Han, Quanzeng You, Yongfei Liu, Wentao Chen, Huangjie Zheng, Khalil Mrini, Xudong Lin, Yigi Wang, Bohan Zhai, Jianbo Yuan, Heng Wang, and Hongxia Yang. Infimm-eval: Complex open-ended reasoning evaluation for multi-modal large language models, 2023.
Anwen Hu, Yaya Shi, Haiyang Xu, Jiabo Ye, Qinghao Ye, Ming Yan, Chenliang Li, Qi Qian, Ji Zhang, and Fei Huang. apllu-papproww Scientific diagram analysis with the multimodal large language model. arXiv, 2023.
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning. arXiv, 2017.
11

---

Published as a conference paper at ICLR 2025
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: 2: Bootstrapping -inaee pre-training with frozen image encoders and large language models. In ICML. 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. B Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023b.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2024.
Shengzhi Li and Nima Tajbakhsh. Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs, 2023.
Yifan Li. Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023c.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In ACL, 2004.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023,
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS. 2024b.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-We Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS, 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-We Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv, 2023.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019.
Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In ACL 2022, pp. 2262-22662222 2279, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022. findings-acl.177.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, al. Gpt-4 technical report, 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
Qwen-VL Team. Qwen-vl-max, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, et al. Gemini: A family of highly capable multimodal models, 2024.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Can- dace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR, 2022.
Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao Haiyang Xu, Qinghao Ye. Ming Yan, Ji Zhang, Jihua Zhu, Jitao Sang, and Haoyu Tang. Evaluation and analysis of hallucination in large vision-language models, 2023a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024.
12

---

Published as a conference paper at ICLR 2025
Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvim: Visual expert for pretrained language models, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022.
John Wright, Yi Ma, Julien Mairal, Guillermo Sapiro, Thomas Huang, and Shuicheng Yan. Sparse representation for computer vision and pattern recognition. Proc. IEEE, 2010.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvim-ehub: comprehensive evaluation benchmark for large vision-language models, 2023.
Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, Jing Shao, and Wanli Ouyang. Lamm: Language-assisted multi-modal instruction-tuniin tuninn dataset, framework, and benchmark, 2023.
Weihao Yu, Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi, From recognition to cognition: Visual commonsense reasoning. In CVPR, 2019.
Yuchen Zeng. Wonjun Kang, Yicong Chen, Hyung Il Koo, and Kangwook Lee. Can milms perform text-to-image in-context learning?, 2024.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In CVPR, 2019.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal of-inghf- reasoning in language models. arXiv, 2023.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A TRAINING DATA
Data Composition During the training the VEGA-Base model, we utilized the entire IITC subset of the VEGA dataset. As for the ITA subset, we incorporated all of the Expanded Context data, as well as half of the data from both the Captions and First Mentions. The training data distribution visualized in Figure 6
ITA Caption 10%
ITA First Mention
10%
IITC 59%
ITA Extend Context
21%
Figure The composition data for model training.
B LLAVA BENCH PROMPT
[Picture caption]: (Picture Caption]
[Picture Context]: (Picture First Mention
[Question]: [Question}
[Assistant 1]
(Answer 1}
[End of Assistant I]
[Assistant 2]
(Ground Truth)
[End of Assistant 2]
[System]
We would like to request your feedback the performance of two Al assistants in response the user question displayed above. The user asks the question on observing an image. For your reference, the visual content in the image is represented with few sentences describing the image.
Please rate the accuracy of their responses. Each assistant receives an overall score on a scale to 10, where higher score indicates better overall performance.
14

---

Published as a conference paper at ICLR 2025
Please only output single line containing only two values indicating the scores for Assistant and respectively. The two scores are separated by a space.
Please avoid any potential bias and ensure that the order in which the responses presented does not affect your judgment.
C QUESTION FILTERING
In the construction of the VEGA dataset, we filtered out the questions that lacked clear image references, shown below:
What are the implications of the results shown in the graph?
What are some the potential applications the graph?
What are some of the implications of this graph? Are there any other interesting aspects the graph that you would Like to highlight What is the signi significance ficance of the results shown in the graph?
What is the main goal of the experiment illustrated in the graph?
What are the key takeaways from this graph?
What are some of the key observations that can be made from the graph?
What are the key features of the mutation plot?
What are the implications of the findings in the graph?
What are the limitations of this graph?
What are the implications of the results of this graph? the conclusions that can be drawn from the graph?
Are there any limitations What are some of the key takeaways from this graph?
What are some the limirations of the graph?
What are the implications of the results of the graph?
What are some the implications of the results in the graph?
What is the significance of the different colors in the graph? What are the limitations of the graph?
What are the implications of the results shown in the figure?
What is the main focus of the graph?
What is the significance of the 3D plot?
What is the main message of the graph? what does the y-axis represent?
What are the key takeaways from the graph? What can be inferred from the graph?
What is the purpose of this graph?
How does the graph support the claims made the paper?
What are the key features of the graph?
What is the main idea of the graph?
What are the key takeaways from the figure?
What is the significance of the graph as a whole? What is the purpose of the graph?
Are there any other interesting aspects of the graph that you would like to point out What is the main purpose of the graph?
What is the main goal of the graph?
What is the significance of the markers in the graph?
What are the implication of this graph for future research? What does the graph show about the performance of the proposed method?
What is the main objective of this graph?
What is the main idea of the figure?
What is the significance of the graph in the context of the paper?
What does the x-axis represent?
What is the difference between the two figures in the graph?
What does the x-axis and y-axis of the graph represent?
What are the two main lines in the graph?
What are the implications of the results in this graph? What does the graph show?
15

---

Published as a conference paper at ICLR 2025
What is the significance of this graph?
What is the main takeaway from the graph?
What are the implications of the findings from this graph?
What are the main takeaways from the graph?
What is the purpose of the two figures in the graph?
What are the key findings of the graph?
What are the two main axes the graph?
What do the colors in the graph represent?
What are some the implications of the findings presented in the graph?
16

---

Published as a conference paper at ICLR 2025
D
CASE STUDY
This section presents the performance of VEGA-Base-4k, Gemini-1.5 and GPT4V on the IITC 4k task. Fig. 7 and iillstraaee cases where all three models failed, highlighting the challenging and complex nature of the IITC task. The models were misled by deceptive textual content, resulting in incorrect answers. Fig. 9 and I0] shows instances where all three models provided the correct response. Fig.
I1 12 and 14 depict scenarios where VEGA-Base-4e answered correctly, while Gemini-1.5-pro and GPT4V did not. VEGA-Base-4k successfully navigated through distractions from similar images and irrelevant text, accurately interpreting the information the images to arrive at the correct answers, demonstrating its robust capability to process interleaved image-text inputs.
Context
the access pattern of DFS does not match this layout. This increased randomness of access to pages means that there is less locality to exploit on each jump than occurred for Linear Search, and hence less gain versus Nswap compared Linear Search.
Figure 11[Picture shows different execution times DFS for various counter threshold sizes. ElasticOS achieves at best about 1.5x improvement delay over Nswap across wide range of counter thresholds, namely larger than 64. However, for small values of threshold less than or equal to 64, DFS performs worse.
Figure 2] shows that when the threshold value is very small, DFS experiences a large number of jumps. Also, our tests showed that DFS's best performance happens when the threshold value is large compared to other algorithms as shown in jubbb))n)
00030
Bluui-06
100.0
-0000
-00
30030
790.0
10030
Ocurter
Figure 11
Figure 12
The shape the graph in DFS can also impact the memory access pattern. For example increasing the depth the graph would make branches longer, resulting in a longer branch that occupies more memory pages, Increasing the chance of single branch having pages located both on local and remote machines. This would increase the chances of jumping more and performing poorly. Figure 13 3] shows DFS performance on ElasticOS for different graph depths with a fixed jumping counter size of 512. Increasing the graph depth eventually results poorer performance. Figure 14[Picture 4) shows that this poorer performance occurs when there is excessive jumping for deep graphs. To make ElasticOS perform better on such graph depth we need to increase the jumping counter size values larger than 512, avoid jumping too much.
6:00
350
8.30
3500
4.00
2003
0.00
1300
100
Figure
Figure 14
Figure 7: The context for Case
17

---

Published as a conference paper at ICLR 2025
Question: What the relationship between graph depth and performance ElasticOS?
Ground Truth: The (Picture 3](Figure. 13) depth DFS search tree the maximum number of levels tree. the graph depth increases, number branches
tree also increases. This can longer branches, which occupy memory pages. single branch pages located both local remote machines, this increase chances jumping more and performing poarly.
VEGA-Base-4k:
The (Picture 4](Figure. 14) shows that the graph depth increases, the performance ElasticOS decreases. is because as the graph depth increases, the number of jumps increases, which makes it more path.
find
rouge-l: BIEU: 099
Gemoni-1.5-pm The performance ElasticOS degrades as the graph depth increases. [Picture 3] This because deeper graphs lead more jumps, as shown [Picture 4], which negatively impacts performance. rouge-l: 0.320; 0.149
The relationship between graph depth and the performance of ElasticOS is that the graph depth increases, the performance of ElasticOS tends to decrease. This is evidenced by [Picture 3](Figuee. 13), which shows that a deeper graph results longer execution times for ElasticOS. The performance decline is further intensified due to the increased frequency of jumping to non-local memory pages the depth grows, which observed in (Picture 4](Figure. 14), where excessive number jumps occur deeper graphs. mitigate issue improve performance deep graphs, would necessary than 512.
GPT4V:
threshold value the jumping
Score: rouge-l: I: 0.349; BIEU:0.143
Analysis: VEGA-Base-4k may have selected the wrong image due being misled by related text description of [Picture 14]. fact, [Picture 14J describes the relationship between Graph Depth indicating that VEGA-Base-4e
number of jumps, content.
attention to the content the picture might be less than that Both Gemini-1.5 1.5 and GPT4V correctly understood the content that the context wanted to express, but they did not follow the instructions well and interpreted two pictures at the same time.
Figure 8: Models' answer and analysis for Case 1. The pictures referred in the Ground Truth marked blue, the incorrect images referred to the model's 's answer are marked in red, and the correct images are marked in green.
18

---

Published as a conference paper at ICLR 2025
Context
Rodycial somm
Ossosriider
Figure 3
Figure 4
Figure
the description our results by emphasizing individual realizations of the interface density $Arhoş remain always active for any non-zero value of the noise, as also the case for the variable $n$ (see Fig. veffffmmnnneiiiaiiinnnggeee for Barab\'asi-Alber free network corresponding. respectively, to the bimodal [panel (textfffa)]] and the unimodal regime [panel [textb([b]]] While in the first of them (Sa < a_c$) the system fluctuates near full order, with sporadic excursions of different duration and amplitude towards disorder; in the second ($a > a_c$), the system fluctuates around a high level of disorder, with some large excursions towards full order.
example, show in "6[Piccuue 4] realizations the dynamics
Figure
Introducing the annealed approximation for uncorrelated networks described above into the definition of the order parameter given equation oererefffrrhoddeffitttonn definition), and focusing on the steady state average value, we obtainin way, an explicit solution the steady average interface density be namely, in terms of the variance S1sigma*2 [st][n]S (see Appendix
by expressing
terms
analytical results presented "Vref(a:order_ pprammterrrar..paa the Interface density) for details), This expression can be contrasted with numerical results in Fig. (iPicuure 5), where present the steady state average interface density Salangle \rho \rangle_ [st]S as function of the noise parameter $a$ for different types of networks. The mean-field pair-approximation result derived in \ctef0lakonova2015) is also included for comparison.
Extli Bihel nameroo
Barabial Alac'
Mogesrate pe copres
Figure 7
As we can observe in Fig.-7[Picture 5], our approach correctly captures the behavior of the system for both small (Sa Vesssim 10*(-3]5)am and very large values (Sa lgtrssmm 3$) of the noise parameter: both the asymptotic
convergence towards VmbllsSVangle vho trangle_ (attle_sss__a_______))) 05) for small $a$ (voter model result for finite systems) and convergence towards VSagle \rho Irangle_(st) 1/25) large Sa$ (full disorder) are well reproduced. the contrary, our analytical approach fails reproduce the numerical results for intermediate values the parameter (\mbox($10'(-3) Vesssim Veessim 3$]]. The origin of this discrepancy lies the annealed network approximation: when replacing the original network by weighted fully-connected topology, all track of local effects is lost preccselly those measured by the order parameter. The fact that this discrepancy only present for intermediate values of $a$ can be explained, on the one hand, by the lack of locally ordered structures in the fully disordered, large $a$ regime and, on the other hand, by the development of global order order--morr -more and more independent of local effects- for decreasing values of $a$. Thus, an accurate fit the numerical results for any Supplementary approximation successfully
value $a$ can only be expected for topologies where local effects are absent or negligible. Figure presented results fully-connected network. The good accuracy the results presented above for the variance of $nş suggests that the discrepancy between analytical and numerical results appears only when the annealed network approximation used to derive relationship between $Vanglee \rho \rangle_ [st]$ and $1sigma"2 ( and not in the de derivation of the latter, for which only global correlations are relevant.
Figure S1) we show
Figure 9: The context for Case
19

---

Published as a conference paper at ICLR 2025
Question:
What are the three different types of networks that are studied in this figure?
Ground Truth:
The three different types of networks that are studied in (Picture 5](Figure. 7) are Erdos-Rény) random networks, Barabási- Albert scale-free networks, dichotomous networks. Erdös-Ré Rénii random
networks networks which the nodes connected randomly. Barabási-Albert scale-free networks networks in which the nodes connected preferential attachment manner. Dichotomous networks networks in which the nodes either connected or not connected.
BaAA-Base-4k: The three different types of networks that are studied in this [Picture S](Figure. 7) are Erdös-Rényi, Barabási-Albert, and Dichtotomous. These networks are all random graphs, but they have different degree distributions. Bédds-éényi
graph power-law degree distribution, Barabási-Ae Albert scale-free degree distribution, and the Dichtotomous network random graph
network random graph power-law degree distribution.
Score: rouge-l: 0,451; BIEU: 171
-emonn--15-pro: [Picture 5] The three types networks studied this figure are: Erdös-Rényl, Barabási-Albert, and Dichotomous.
rouge-l: 0.408;
three different types networks studied [Picture 5][Figure. 7) Erdös-Rényl. Barabási-Albert, Dichotomous networks.
GPT4V:
Score: rouge-l: 0.417; BIEU:0.022
Analysis: The text does not explicitly detail the three different types of networks depicted [Picture 5] (Figure 7). However, the three models are correctly linked to the appropriate images and provide accurate responses. This demonstrates the model's capability for text-image correlation and image comprehension.
Figure 10: Models' and analysis for Case 2. The pictures referred in the Ground Truth marked in blue, the incorrect images referred to in the model's s answer are marked in red, and the correct images are marked in green.
20

---

Published as a conference paper at ICLR 2025
Context
The experimental results show that the (laigA) factorization verystable. Figure 4[Picture 1] displays the growth factor of (lalgA]for random matrices of size varying from 1024 to 8192 and for sizes ofthe panel varying from 8 to 128. We observe that the smaller the sizeof the panel is, the bigger the element growth is. In fact, for asmaller size of the panel, the number of panels and the number ofupdates on the trailing matrix is bigger, and this leads to largergrowth factor. But for all panel sizes, the growth factor of (ValgA) issmaller than the growth factor of GEPP. For example, randommatrix size 4096 panel size 64, growth factor onlyabout 19, smaller than growth factor obtained by GEPPand expected, much smaller than the theoretical upper bound (409955)))
Figure
Tables (ref(TabRRQRLU randnilu) and Vre((TabLUPRRPP spec1] in Appendix Bpresent more detailed results showing the stability of the (valgA/factorization for random matrices and a set of special matrices.There, we include different metrics, such as the norm of the factors, the value of their maximum element and the backward error of the LUfactorization.W evaluate the normwise backward stability by computing three accuracy tests as performed in the HPL frigh-Performance Linpack) benchmark.In In HPL, the method considered to be accurate the values of thethree quantities are smaller than $16$. More generally, the valuesshould of order $O(1)S. For the (algA) factorization HPL1 is most $8.09$,HPL2 We also display the normwise backwarderror, using the 1-norm,and the componentwise backward errorwhere the computed residual $r-b-A (x]S. For our tests residualsare computed with double-working precision.
most $8.0 ('tmmes 10M(-2]S and HPL3 most $1.60 51.60\simms (0*(-
Figure S[Picture 2] summarizes all our stability results for([algA). This figure displays the ratio of the maximum between thebackward error and machine epsilon of (|algA) versus GEPP. Thebackward error measured using three metrics, the relative error SYPPA-LU \/ / ///A VS,, the normwise backward error $1eta,, and thecomponent- wise backward error SwS of (\algA) versus GEPP, themachine epsilon. We take the maximum the computed error epsilon since smaller values mostly roundoff error, and so taking ratios lead extreme values with little reliability. Results for all the matrices in test set arepresented, that $20$ random matrices for which results arepresented Table (ref(TabRRQRLU randnilu), and $37$ special matricesfor which results are presented in Tables TTbGEEP [ andalmost all ratios are between $0.5$ and $2Ş. For special matrices,ther there are few outllers, up to $ 23.71$ (GEPP is more stable) for thebackward error ratio of the special matrix \texti((hadamard) and downto $2.12\times 10M(-2] ([aagAA) is more stable) for the backwarderror ratio of the special matrix (texiit(moler).
ICAU
7000
304U
YEAR
65AJ
TOAL
7562
754
DECEEES SIEERE ETP
Figure
Between max([algA)'s backward erros, machine epsilon) and max(GEPP's backward error, machine epsilon) for all the test matrices mean that (ValgA)'s backward error is larger, and bars below $1$ mean that GEPP's backward error is larger. For each matrik and algorithm, the backward error is measured ways. For the first third of the bars, labeled SAIPA- LUN /AJA/S S, the metric is the backward error, using the Frobenius norm. For the middle third of the bars, labeled "normwise backward error", the metric is $1etaš in equation legref(eq:eta). For the last third of the bars, labeled "componentwise backward error", the metric $wS equation teqref(eqccomp) The test matrices are further labeled either as "randn", which are randomly generated, or "special", listed in Table \ref(Tab allrestMatrikk.
Each vertical bar represents such ratio
test matrix, Bars above $10*0=15
Figure 11: The context for Case
21

---

Published as a conference paper at ICLR 2025
Question:
What does the graph show about the stability CALU PARP?
The (Picture 2](Figure. 5) shows that CALU PRRP stable algorithm, with backward errors are most times larger than those of GEPP random matrices, and most times larger special matrices. This consistent
Ground Truth:
the results reported in the paper, which show that CALU PRRP has relative error of 9.14 x 10-14, normwise backward error of most 1.37 x 10-14, and componentwise backward error of most 10-8 8--00-ffo for Demmel matrix.
VEGA-Base-4Ke The (Picture 2](Figure. 5) shows that CALU PRRP is more stable than GEPP for all test matrices the set. This is evident from the fact that the backward error ratio of CALU PRRP generally lower than that of GEPP. For example, special
hadamard,
10-2, while This that PRRP accurate than GEPP for this
backward ratio of GEPP Score: rouge-l: O 0.408; BIEU: 0.093
Gemoni-1. [Picture 1] This graph shows the stability of the CALU PRRP algorithm comparing its growth factor to the growth factor of GEPP for different matrix sizes and panel sizes. The growth factor of CALU PRRP is smaller than the growth factor of GEPP for all panel sizes. As the size of the panel increases, the growth factor of CALU PRRP decreases. This indicates that CALU PRAP is more stable algorithm than GEPP, especially for larger panel sizes.
rouge-l: 0.379;
The graph in [Picture 1F[Fiuure. 4) shows the average growth factor when using the CALU PRRP factorization for random matrices of various sites and panel sizes. The results indicate that the smaller the size of the panel, the larger the growth factor is, which means less stable factorization. However, for panel sizes, the growth factor is smaller than that
GPT4V:
(another algorithm for matrix factorization), suggesting that CALU comparatively stable GEPP. graph indicates for CALU PRRP increases controlled manner, significantly below theoretical upper bounds, showing good stability characteristics the CALU PRRP algorithm.
increases, growth factor
Score: rouge-l: 0.350; BIEU: 0.0398
Analysis: VEGA-Base Base-4k picked the correct image and drew the right conclusion. Both Gemini-1.5-pr and GPT4V the correct conclusions based on information. GPT4V's
chose the wrong images, yet they response significantly longer compared Ground Truth, resulting Rouge-l and BLEU
Figure 12: Models' answer and analysis for Case 3. The pictures referred in the Ground Truth marked in blue, the incorrect images referred to the model's 's answer are marked in red, and the correct images are marked in green.
22

---

Published as a conference paper at ICLR 2025
Context
discriminative classifiers, we train three-layer \times \times MLP network one input layer ($7845 neurons), hidden layer ($100$ neurons) and one output layer ($105 neurons) for classification. In training stage, classifiers are trained with radial basis function (RBF) kernel for these two data sets. We built $10$ "one-versus-all" SVM classifiers for the $ 110ss classification problem, and assign the class label with the greatest margin to the test image. The DBN used in our experiments is $768 \times 500 \times 500 \times 2000 \times 10$ network which is trained by two stages: pre- pre-training stage and fine-tuning stage. In its pre-training stage, each hidden layer is trained with $505 epochs using all tra vaiining data, and in its fine-tuning stage, a pack-prooaaation training algorithm interested readers
the learning
$0.015, and the number of iterations $5000S. SVM
$200$ epachs adopted the discriminative purpose. We (hinttnn for detailed descriptions this deep belief network.
Figure 2
our proposed Kernel-Distortion classifies, we use SK=100$ kernels, set Sp 35, $q 405, Sisigma_ _ 0.03$ and $1sigma dd2 0.9$. select better kernels, the proposed iterative kernel selection algorithm applied. In Fig. 2[Picture 11, show total likelihood $QS function iteration for $3000$ iterations for digit "$6$", when the MNIST data set is used example. Although purely monotonic, shows steadily increasing likelihood \footnote|A video demonstration for the process of kernel selection is provided as Supplementary Material. One can see that the class-wise likellhood is steadily increased with the number of iteration, and that the "best to remove" digits seem really bad at first, and then at the end, they look almost the same as the "best to add". J. The steady increasing likelihood demonstrates the effectiveness of our proposed iterative kernel selection algorithm. We stop the kemee-selection algorithm the $5005 iteration both MNIST, and data sets, although of overall testing classification error generative classifiers for these two data sets. However, proposed generative classifier can greatly improve the prediction performance compared with other generative classifiers and outperform some other well-trained discriminative classifiers. It performs best for USPS handwritten digits dlassification and ranks third for MNIST handwritten digits classification, which demonstrate the effectiveness of our proposed Kernel- Disttl--sstotioon classifier. LDA, GMM, GKDE, LKDE, MLP-NN, DBN, RBF SVM, Sk$ NN (5k=15), and our Kernel- Distortion classifier. For each dataset, we highlight the best result with (underline (Boll)) value, the second one with (textbf(Bold) value, third one (textit[ttalic) value, among all classifiers.
likelihood still increases. Table (ref(tab all) shows the comparison results in shown that the discriminative classifiers usually outperform the
2-1
Figure Figure. 3The largest value of $a$ tried was 40. The performanceat the far right the Euclidean distance metric wlassifier,which is equivalent to $q=N^2$. Figure. Avariance ratio S1sigma"2 $ysigma"24//sigmar> __tee______________ee Figgurr SThe parameters of the proposed approach include $p5, $q$, SKS, $\sigma_ _a and $tsigma_ dd2aaa__dddaeeae_ea Given the MNIST data set as an example, we further study the classification performance with different parameter values. Firstly, the performance different $q$ values from $25 to $40$ measured, as shown Fig. 3[Picture 21, when SK 1005, right performance the Euclidean distance
Figure
Figure
Sp 3$, Slsigma_ o*2 0.035 and S\sigma_dd 1iligma_d22=2_ 0.95. At metric, which equivalent $q=N* The performance the best $q=405, the highest $Q$ value tried experiments. Due the slow performance improvement and the high storage requirement, function of $'gammma is shown in Fig. 4[Picture 3), indicating
not higher
value. For $p=3$ and $q=40S, the performance that $1gamma=30$ is a reasonable value. Notice that our approach the Euclidean distance metric classifier when $1gamma=1$. We also evaluate the performance as function of $p$. The result is shown in Fig. S[Picture 4), which shows no advantage for Sp$ larger than $3$. We note that it is still possible to achieve better parameter settings. and different classification tasks would have different optimal parameter settings.
Figure 13: The context for Case
23

---

Published as a conference paper at ICLR 2025
What does the graph show about the relationship between variance ratio and total classification error? The (Picture 4][Figure. 5) shows that there is negative correlation between variance ratio variance ratio increases, the total classification error decreases. This likely because higher variance distinguish between different classes.
Question:
Ground Truth:
classffication error. This means
that data spread which makes easier
The [Picture 4](Figure. shows that the total classification error decreases as the variance ratio increases. This because higher variance ratio means that model more sensitive to the input data, and thus more accurate predictions. However, decrease learn the underlying structure the data and make
VEGA-Base-4k:
not linear,
error eventually
plateaus. This because the model eventually accurate predictions without the need for more variance.
Score: rouge-l: 0.447; BIEU: 0.168
The graph [Picture shows the relationship between the variance ratio total classification error the variance ratio increases, error rate initially decreases, reaching lowest point around point, variance continues Score: rouge-l: 0.349; BIEU: 0.156
Gemoni-1.5-pro
relatively with only fluctuations
GPT4V:
The graph in [Picture 3](Figure. 4) shows that as the variance ratio (v) increases, the total class dassifiiation error decreases, reaching an optimal point at y=30. Beyond this point, the error rate levels off, indicating that increasing the variance ratio further does not significantly affect the error rate. This suggests that a variance ratio of around 30 classification
reasonable
Score: rouge-l: 0.349; BlEU:0.143
Analysis: VEGA-Base picked correct image drew the conclusion. Both Gemini-1.5 and GPT4V selected the wrong images and responded based the related text content of incorrect images, resulting in lower ROUGE scores. A possible reason could be that they were misled by the labels on the y-axis of (Picture 3)(Figure. 4).
Figure 14: Models' answer and analysis for Case 4. The pictures referred in the Ground Truth marked in blue, the incorrect images referred to the model's 's answer are marked in red, and the correct images are marked in green.
24


Reviews:
[{"review_id": "fpRpSv1ADT", "reviewer": "Reviewer_Mss2", "summary": "Long-context interleaved image-text comprehension is an important skill required to read scientific papers, for example. Humans can do this well, but characterization of this ability is understudied. This paper introduces a multimodal task called Interleaved-Image-Text Comprehension which requires answering a question using interleaved image and text and grounding the answer in one of the in-context images. The authors develop a dataset called VEGA to evaluate models on this task and auxilliary simpler task called image-text association. They evaluate multiple frontier LLMs and some open LLMs on the task, and develop a training strategy which they show improves performance on this task.", "questions": "Please see the weaknesses section. Most of the weaknesses are minor. The weakness I care about and the one where an author response would cause me to increase my rating is \"analyses\". What analyses have the authors done on how properties of the dataset (e.g. number of images, context length, etc) affect the performance of MLLMs? Also, can the authors discuss why there are multiple evaluation metrics and what each of them signify?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- There are no resources (as far as I know of) to study and characterize _long-context_ interleaved image-text comprehension or empirical studies of this comprehension ability. This is one of the first.\n- The experimental evaluation is broad, covering multiple vision language models including a mix of closed source LLMs and open source LLMs.\n- I especially appreciate the honest of the authors in showing that training on Vega _does not_ in general improve performance on other VQA tasks.", "weaknesses": "- The construction method of the dataset seems a bit artificial.\n- I don't see analyses of how the number of images / context length affects accuracy. In general, I feel like important analyses are not present. There are no \"slices\" in the dataset to sanity check that expected properties hold. For example, I would expect that as context increases, accuracy falls. If I did not see this, I would be suspicious of the construction method. This is just an example. My main point is that the dataset and how the properties of the dataset affect evaluation are not analyzed enough or perhaps not explained in the text.\n- I am not convinced by the proposed training method. It is not really motivated or explained well in the text, and there is no discussion (as far as I can tell, but it is very possible I have missed something) on how it relates to other similarly named methods in the literature for \"multi-scale training\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C48AQdPs5k", "reviewer": "Reviewer_QKxm", "summary": "This paper introduces a novel task, Interleaved Image-Text Comprehension (IITC), which evaluates models' capabilities to handle complex interactions in lengthy multimodal contexts. The authors present the VEGA dataset specifically designed for the IITC task, featuring up to 8,000 tokens and 8 images. The paper also benchmarks several models on this task, including proprietary and open-source options, and demonstrates the benefits of fine-tuning Qwen-VL-Chat for IITC. The results show that even state-of-the-art models face challenges in this setting, highlighting the dataset's rigor and the task's demands.", "questions": "Would you consider adding an out-of-distribution test set to further evaluate generalization capabilities?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The introduction of a challenging new task (IITC) with extensive multimodal input length sets a unique standard in evaluating models' interleaved image-text comprehension. The proposed VEGA dataset is extensive, comprising 295k high-quality, multi-turn entries, with a curated test set of 700 questions to assess models' nuanced comprehension abilities.\n\nThe benchmark evaluates both proprietary and open-source models, showing that current models, even at their best, are still only moderately successful in IITC tasks. Fine-tuning Qwen-VL-Chat leads to meaningful improvements, establishing a strong baseline for future models.\n\nThe dataset structure, including partitions for 4k and 8k token contexts, is well-conceived and allows for detailed performance analysis. Including specific references to image indices in responses (e.g., '[Picture 2](Figure. 3)') is a thoughtful design choice, aiding interpretability in output.", "weaknesses": "Presentation: Figures, particularly Figure 1 and Figure 2, are too small, making the text nearly unreadable without significant zoom. This compromises the clarity and accessibility of the paper's visual aids.\n\nTest Set Generalizability: The test set, created through a process similar to the training data generation, may inadvertently advantage trained models by being in-distribution. Introducing an additional test set, created independently or using different datasets, could strengthen the evaluation by ensuring generalization beyond the training data distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7kvwQqrHWa", "reviewer": "Reviewer_x1Yo", "summary": "This paper introduces the Interleaved Image Text Comprehension (IITC) task and establishes the corresponding dataset, VEGA. Compared to existing visual language tasks and datasets, IITC and VEGA have the following unique features: (1) the interleaved images and text input; (2) multiple images input; (3) the task requires the LMM to determine which image is relevant to the question; (4) the input includes image and text information that is irrelevant to the question. Particularly, the latter two features are lacking in previous tasks and datasets.\n\nBy benchmarking existing mainstream close-source and open-source MLLMs on VEGA, it was found that the performance of these SOTA  models is not ideal, thus confirming the challenging nature of the IITC task.\n\nFurthermore, this paper also finetunes Qwen-VL-Chat using the train split of VEGA and adopts a multi-scale, multi-task training strategy, laying a solid baseline for subsequent collaborative research.", "questions": "- During the construction of the VEGA IITC subset, particularly in the answer modification step (see line 305), the authors adjusted the answers from the original dataset (SciGraphQA). Was this modification based on rules (such as keyword matching) or aided by an LLM (e.g., prompt GPT)? Additionally, were the modified answers subjected to quality checks and evaluations, such as through manual sampling inspection and calculating the pass rate?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This paper introduces the Interleaved Image Text Comprehension (IITC) task and creates the corresponding dataset VEGA. IITC and VEGA have the following unique features: (1) the interleaved images and text input; (2) multiple images input; (3) the task requires the LMM to determine which image is relevant to the question; (4) the input includes image and text information that is irrelevant to the question.\nThese features reflect the normal state of human reading and understanding of image-text materials and are essential for a multimodal AGI system. Existing visual language datasets often lack these features, especially the latter two. Therefore, IITC and VEGA fill this gap to some extent and help to advance the technical progress of the research community in this direction.\n\n- The overall design of the dataset is scientific and reasonable, the construction process is detailed, and the provided statistical information is very convincing. The adopted multi-scale, multi-task training strategy also appears reasonable and credible.\n\n- The main body of the paper is well-structured and written in a clear and concise style, making the article easy to read and understand.", "weaknesses": "- All training-related experiments in the paper were conducted solely on the Qwen-VL-Chat model and did not involve other open-source foundation VL models that support interleaved text-image processing. Therefore, the training strategies proposed in this paper and the conclusions from its ablation study have not been sufficiently validated for generalization across other models.\n- The article has some issues with formatting and layout, such as the text in some images being too small, which affects normal reading (e.g., Figure 1). Additionally, the layout of the appendix also needs further optimization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UTvUwuXNrA", "reviewer": "Reviewer_qCo5", "summary": "The paper introduces VEGA, a dataset designed for interleaved image and text understanding. It also proposes a multi-modal large language model (MLLM) baseline fine-tuned on this dataset. The fine-tuning process uses a multi-scale method, gradually incorporating more complex samples. The model is fine-tuned for two tasks in the VEGA dataset: Interleaved Image-Text Comprehension (IITC) and Image-Text Association (ITA). It demonstrates improved performance on both tasks.", "questions": "Other than the weaknesses:\n\n1. Is multi-scale training still effective when not used for multi-task training?\n2. In Table 4, what is the model’s performance when trained only on the ITA task with and without multi-scale training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Proposes VEGA, a complex dataset specifically crafted for multi-image understanding, challenging MLLMs with interleaved image and text contexts.\n\n2. Establishes two tasks, Interleaved Image-Text Comprehension (IITC) and Image-Text Association (ITA), setting new benchmarks for interleaved image understanding.\n\n3. Demonstrates that the multi-scale training method effectively improves performance on the ITA task.", "weaknesses": "1. Key baseline models such as LLaVA-interleaved and InternVL2 are missing, limiting insight into comparative performance.\n2. Similar datasets like SlideVQA, MMLongBench-Doc, DUDE, and DocVQA are not discussed or tested, which could demonstrate the generalization of the fine-tuned model.\n3. The ITA task may have limited real-world applicability, as not all text paragraphs have corresponding images. In contexts like research papers, only a subset of text relates to specific images, and filtering irrelevant text often implies inferred relevance between remaining text and images.\n4. The font in Figure 1 is too small and should be increased for better readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)