MileBench: Benchmarking MLLMs in Long Context
Authors: Song Dingjie
Shunian Chen
Guiming Hardy Chen
Fei Yu
Xiang Wan
Benyou Wang
Venue: COLM (2024)
Keywords: multi-modality, benchmark of long-context model, benchmark of vision-language model
TL;DR: We introduce MileBench, a new benchmark that reveals most Multimodal Large Language Models struggle with handling long context and multiple images.

Abstract: Despite the rapid progression of Multimodal Large Language Models (MLLMs) and their impressive performance on various benchmarks, the applicability of these results to real-world tasks remains uncertain. 
This ambiguity primarily stems from the benchmarks' limited consideration for long-context and multi-image tasks, which are critical elements in real-world applications.
Existing benchmarks often focus on single-image and short-text samples, and when assessing multi-image tasks, they either limit the image count or focus on time-series captioning tasks, potentially masking MLLMs' performance challenges such as hallucination in long-context situations. 
To address these limitations, we introduce \textbf{\dataset}, a pioneering benchmark designed to rigorously test the \textbf{M}ult\textbf{I}modal \textbf{L}ong-cont\textbf{E}xt capabilities of MLLMs. 
This benchmark comprises a mix of text and images, long contexts, multiple tasks, and tasks requiring both comprehension and generation. 
We establish two distinct evaluation sets, diagnostic and realistic, to systematically assess MLLMs' long-context adaptation capacity and their ability to complete tasks in long-context scenarios.
Our experimental results, garnered from testing 19 models, revealed that while closed-source model GPT-4(Vision) outperforms others, most open-source MLLMs display inadequate performance in long-context situations. 
Hence, we strongly encourage an intensification of research efforts towards enhancing MLLMs' long-context capabilities, especially in scenarios involving multiple images.

Published as a conference paper at COLM 2024
MILEBENCH: Benchmarking MLLMs in Long Context
Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, Benyou Wang" The Chinese University of Hong Kong, Shenzhen Shenzhen Research Institute of Big Data wangbenyou@cuhk. edu. cn https://milent wwwe/nch..eeeecc...ommmmc github. io/
Example
BED-Bench
Multi-image Examples from MileBench
wouther
What differencn
How
What happeree after the perus
pelnx tnics
100%
& It's fogur
It's surry
raining hanily.
there people
lig
pirture on light
A this
C fiee
there?
Closed the rriige
lastop
90%
sirewoll and one the car
Opered the Kincion
80%
70%
Performance
60%
50%
Performance
40%
Gap (~45%)
30%
20%
Performance Drop (~40%)
10%
0%
Single (1)
(2-5)
Medium (6-31)
Many (32-109)
Number Images
GPT-4V
Gemini 1.5
LLaVA-1.5-7B
LLaVA-1.6-BB
Y--L
Closed-Soarce MLLMs
Open-Source MLLMs
Figure 1: MLLMs' performance fluctuates with the image count in datasets. Open-source MLLMs demonstrate remarkable performance drop as the number of images increases. The performance gap between open-source and closed-source MLLMs expands as well. For
single-image performance, we refer to SEED-Bench (Li et al. 2023b), given the absence of single-image samples in MILEBENCH.
Abstract
Despite the advancements and impressive performance of Multimodal Large Language Models (MLLMs) on benchmarks, their effectiveness in real-world, long-context, and multi-image tasks is unclear due the bench- marks' limited scope. Existing benchmarks often focus single-imas image and short-text samples, and when assessing multi-image tasks, they either limit the image count focus specific task (e.g time-series captioning), potentially obscuring the performance challenges of MLLMs. To address these limitations, introduce MILEBENCH, a pioneering benchmark de- signed to test the MultImodal Long-contExx capabilities of MLLMs. This benchmark comprises not only multimodal long contexts, but also multiple tasks requiring both comprehension and generation. We establish two distinct evaluation sets, diagnostic and realistic, to systematically assess MLLMs' cong-contett adaptation capacity and their ability to complete tasks long-context scenarios. Our experimental results, obtained from testing 22 models, revealed that while the closed-source GPT-4o outper- forms others, most open-source MILMs struggle long-context situations. Interestingly, the performance gap tends to widen with increase in the number of images. We strongly encourage an intensification research efforts towards enhancing MLLMs' long-context capabilities, especially in scenarios involving multiple images.
*Corresponding author.

---

Published as a conference paper at COLM 2024
SEED-Bench2
Momentos
Avg
15000
Images:
2.7
100-
Arg Images: 11.6
Count
10000
MileBench (Avg Words:422.3)
50-
DEMON (Avg Words:98.9)
5000
SEED-Bese Bench2 (Avg Words:25.3)
10a
Momentos (Avg Words: 18.ds:18.00
10
20
40
Count
DEMON
MileBench
10a
6000
Ima 3.5
Avg
1500
Arg Images: 15.2
Count.
4000
1000
2000
500
10a
0
+48
2000
Word Number
6000
8000
10000 12000
Images Number
0
10
Images Number
20
40
(a) Distribution of Word Number
(b) Distribution of Images Number
Figure 2: Visualization of distribution of images and word number in present MLLM bench- marks. The range and mean of both word and image number per sample in MILEBENCH far exceed those of previous works.
1 Introduction
The recent swift development of Multimodal Large Language Models (MLLMs) (OpenAl
2023: Anil et al. 2023 Liu et al. 2023a Awadalla et al. 2023) has displayed outstanding
performance across diverse range of multimodal tasks Yang et al. 2023 Wu et al. 2023).
Meanwhile, a surge of benchmarks for evaluating MLLM performance has emerged (Lu
al. 2023a Fu et al. 2023
Ge et al. 2023 Li et al. 2023a offering insights into their general
capabilites Li al. 2023b Liu et al. [202cc] Yu et al. 20023) and task-specific capabilities (Liu
et al. 2023d
Yue et al. 22023
Wang et al. 2024).
However, a critical aspect is often overlooked. Real-world applications frequently demand the processing of long contexts and multi-image tasks that include multi-round dialogues based on multiple images (Li et al. 2022), action prediction tasks (Wu et al. [2021), navigation tasks in 3D space (Krantz al.) 2020), and understanding tasks with lengthy documents interspersed with images on Wiki pages (Hannan et al. 2020
Despite this, existing benchmarks primarily focus on single-image and short-text sam- ples (Liu et al.] 2023a Fu et al., 2023 Liu et al. 2023c Li et al.) 2023b), thereby failing to fully capture the complexity and diversity of real-world scenarios. While some benchmarks evaluate multi-image tasks, they either have limited number of images provided per sample (e.g., SEED-Bench-2 (Li et al. 2023a), DEMON (Li et al. 2023c)) or only include time-series In addition,
captioning tasks (e.g., Mementos (Wang et al. 2024)), shown in Figure 2 this omission could potentially neglect the hallucination issue that MLLMs might exhibit in long-context situations (Huang et al. 2023). Given the aforementioned shortcomings with existing benchmarks, we identify a pressing need for a more holistic evaluation that fully encapsulates the long-context and multi-image task demands prevalent in real-world applications.
Addressing this need, we introduce MILEBENCH, the first benchmark specifically designed to test the MultImodal Long-contExt capabilities of MLLM To systematically assess the capabilities of MI.LM in multimodal long contexts, our benchmark consists of two distinct evaluation sets, diagnostic evaluation and realistic evaluation. The former explores the long-context recall abilities of MLLMs, using needle-in-a-haystee tasks, while the latter stress-tests the model in a manner akin to real-world conditions using both temporal multi-image tasks and semantic multi-image tasks. To construct our evaluation sets, we gather 6,440 multimodal long-context samples from 21 pre-existing or self-constructed datasets, with an average of 15.2 images and 422.3 words each, as depicted in Figure 2 and we categorize them into their respective subsets.
and image retrieval
We define "multimodal long contexts" as long text content integrated with two more images, or content composed of multiple images.
2

---

Published as a conference paper at COLM 2024
After evaluating 22 models, the closed-source GPT-46 excelled in both diagnostic and realistic evaluations, achieving impressive scores of 99.4% and 60.3%, although it still falls short of perfect 100% score. On the contrary, most open-source MLLMs struggled with long-com cong-context tasks as depicted in Figure 1 Only Mantis and Qwen-VI.-77 VL.-7B managed average scores of 47.5% and 37 37.2% in realistic and diagnostic evaluations respectively. These results underscore that there are "miles to go" towards fully-realized long-context MLLMs, prompting a call for increased research focus on such tasks, especially those involving numerous images.
2
Related Work
2.1 Multi-image and Long-Context MLLMs
Beyond training on single-image-text pairs, recent developments in MLLMs are also ori- ented towards handling multiple and interleaved image-text sequences (Awadalla et al.; 2023 Li et al. 2023c). However, these models have relatively limited contexts (i.e., up to 4K) compared
leading proprietary MLLMs such GPT-4V (OpenAl 2023) and Gem- ini (Anil et al. 2023 Reid et al. 2024), which exhibit capabilities for long-contens processing, supporting up to 128K and 10M tokens, respectively. However, there remains notable gap in open-source MLLMs capable of long-context comprehension. Currently, the only open- source models equipped for long contexts are those designed for video, which are trained to process multiple frames, inherently managing multiple images and long contexts (Liul let al. 2024a Zhang et al.) [2023 Luo et al. 2023 Li et al. 2023d e). this paper, we release evaluation set specifically designed for mult- and long-context MLLMs.
2.2 Evaluation of MLLMs
Most of the MLLM benchmarks only evaluate multimodal tasks with a single image (Liu
et al.] 2023a Fu et al., 2023 Liu et al. 2023c Li et al. 2023b Yu et al. 2023 Ge et al. 2023 Yue et al. 2023). As a complementarity. SEED-Bench2 (Li et al. 2023a and DEMON (Li et al. 2023c) test multimodal capabilities with multiple images but limit the evaluation
to around three images, which is inadequate for thorough multi-image comprehension
assessment. Mementos (Wang et al. 2024) involves data samples with up to approximately 11 images, mainly focusing on temporal understanding and limited context scenarios. This focus overlooks the wide range of scenarios involving multiple images and long context. We provide an overview of existing MLLM benchmarks in Appendix A To the best of our knowledge, MILEBENCH is the first comprehensive benchmark that evaluates MLLMs across both multi-image and long-context dimensions, catering to broader spectrum of general scenarios.
MILEBENCH
3.1 Evaluation Taxonomy
MILEBENCH consists two major components: Realistic Evaluation and Diagnostic Eval- uation, depicted in Figure 3 Realistic Evaluation requires MLLMs to address tasks within multimodal long-context scenarios, emphasizing the models' proficiency in com- prehending and reasoning across extended multimodal contexts. Conversely, Diagnostic Evaluation demands MLLMs to retrieve information from the provided context, high- lighting the model's capability of long-range information retrieval and the elimination of distractors. The detailed taxonomy of MIL.BBENCH is illustrated in Table 4
3.1.1 Realistic Evaluation
The realistic evaluation designed to assess an MLLM's ability to comprehend, integrate, and infer information in multimodal long context. We categorize the tasks into two main groups: Temporal Multi-Image tasks and Semantic Multi-Image tasks. Temporal
httpp:: http://oppmmaaccco /iddex haaam//dde/ellxggttetaeee hello-gpt-4o We adapted the taxonomy from MVBench eL 2024 and DEMON Li et al. 2023c to suit our multimodal long-context setting.
3

---

Published as a conference paper at COLM 2024
Temporal Muttl-Image
Realistic Evaluation
Semantic Multi-Image
(T-1) Action Understonding and Prediction
(S-1) Knowledge Grounded
(S-4)
(S-2) Text-Rich Images QA
Dialogue
(S-3) Visual Relation Inference
(S-5) Space Understanding
Instruction: Based on provided images, answer the question related the sequence of action You must choose your answer from the Choice List. Question: What happened before the person held the food?
Choice List:
A. Opened the bag.
B. Closed the closet/cabinet. sandwich.
Opened the laplop.
Instruction:
Upon viewing pictures cuptured from street view vehicle, role
various
(T-2) Object and Scene Understanding
answer questions about the presented scene. You must choose your answer from the Choice List.
(T--) Visual Novigation and Spatial Locolization
Question: how many yellow trucks are there? five
(T-4) Counterfactual Reasoning and State Change
Choice three
Diagnostic Evaluation
Needle a HayStack
Image Retrieval
(N-1) Text Needle
S (1-1)Image Retrievol
Instruction: Answer question concisely and directly based on the provided images and text.
Instruction: Identify the candidate image exhibits the similarity in color, texture, and shape the given anchor image.
Context: You write programs that manipulate
them. Just look for reasonable domain expert proposing something that sounds wrong. And I don't have to know if bet on everything just being on the server.
Image2:
Image3:
Imagel
But its origins as medel of computation gave it power and elegance that other languages couldn't match. The special magic Budapest number 562653d There
ImageS:
Anchor
Image4:
Image:
Always Room, ......
Question: Choose the answer from the choice list,
Question: What is the special magic Budapest number?
Choice List:
A. imagel image2
C. image3
image4
E. image5
(N-2) Image Needle
Figure 3: Taxonomy and four multimodal long-context examples in MILEBENCH.
Multi-Image tasks test the MLLM's ability to discern temporal relationships among several time-related images, emphasizing the model's predictive capabilities in real-world scenarios. On the other hand, Semantic Multi-Image tasks challenge MLLMs to process multiple images that are possibly temporal-irrelevant but are semantically interconnected.
Temporal Multi-Image Tasks. Temporal Multi-Image tasks include four temporal-related multi-image tasks. Each task contains multiple subtasks, each with 200 samples.
T-1 Action Understanding and Prediction task involves interpreting and forecasting actions of objects or characters in sequential scenarios based on a series of images. This task is divided into three subtasks, Action Localization (Gao et al. 2017) assesses the model's ability to identify time actions within a sequence. Action Prediction (Wul et al.] 2021) tests the model's capacity to predict character's 's actions. Action Sequence (Wu et al. 22021) evaluates the model's understanding of the chronological order of character' actions.
T-2 Object and Scene Understanding task involves identifying and understanding objects within a sequence. It comprises four subtasks: Öbject Existence (Yi et al., 2020) tests the model's ability to detect and track an object's movement. Moo- ing Attribute (Yi al., 2020)) assesses the model's understanding of moving object attributes. Object Interactio (STAR) evaluates the model's 's comprehension of inter-
*T, S, N. denotes Temporal Multi-image Tasks, Semantic Multi-image Tasks, Needle in a Haystack Tasks, and Image Retrieval Tasks, respectively. The numerical value is the index of the task within the set of tasks.

---

Published as a conference paper at COLM 2024
actions between people and objects in complex scenarios. Object Shuffle (Patraucean et al. 2023) gauges the model's ability to locate hidden objects amidst disturbances.
T-3
Visual Navigation and Spatial Localization task tests the model's understanding of spatial and directional concepts through two subtasks. One is Egocentric Naviga- tion (Krantz et al. 2020), involves the model interpreting motion-relates instructions and image sequences from robot's perspective to predict the next action. Moving Direction (Yi et al. 2020), assesses the model's ability to determine object movement direction,evalluttnnn
direction, evaluating its understanding of spatial orientation.
T-4 Counterfactual Reasoning and State Change task evaluates the model's logical reasoning within image sequences, focusing on causality and state changes. It comprises four subtasks. Counterfactual Inference (Yi et al. (020) tests the model's ability to predict outcomes under hypothetical changes. State Change (Patraucean) al. 2023) assesses the model's understanding of object state changes. Character Order Paatraucean et al. 2023) examines the model's 's reasoning of the order of letter appearances over time. Scene Transition (Huang et al. 2020) evaluates the model's
understanding of scene changes and the associated causality.
Semantic Multi-Image Tasks. Semantic Imalt tasks include five semantic-related multi-image tasks. Each task contains multiple existing artificially constructed datasets, each with 200 samples.
S-1 Knowledge Grounded QA task centres on knowledge-based reasoning, where models synthesise multimodal knowledge for single- or multi-hop reasoning tasks. The task employs four datasets: Webpage QA (Chang et al. 2022) for open-domain multi-hop web search, Textbook QA (Kembhavi et al. questions with diagrams and images, Complex Mullimodal QA (Talmor et al. 2021 for complex Wikipedia questions with tables and images, and Long Text with Images QA for long text with images questions from Wiki documents.
] (017) for multimodal textbook
S-2 Text-Rich Images QA task demands the processing and understanding of rich text information embedded directly in images. It calls for models capable of recognizing textual information from images and integrating this textual information with complex reasoning to answer questions. It contains Slide QA (Tanaka et al. 2023) for multi-slides question answering requiring multi-hop and numerical reasoning, OCR QA (Mishra et al., 2019) for book cover image text reading, and Document QA (Mathew al. 2021) for document images with focus on understanding document structure.
S-3 Visual Relation Inference task is centered around understanding and inferring visual relationships. It aims to detect subtle variations between two images, such as changes in objects and positional shifts, and subsequently generate accurate descriptions of these changes. This necessitates the model to possess robust capabil- ities in capturing visual details and generating response. The task leverages Visual Change Captioning (Jhamtani & Berg-Kirkpatrick] 2018 Hosseinzadeh & Wang 2021 to describe differences between similar images, image change capturing, and Visual Relationship Expressing (Tan al. 2019) for generating relationship captions between images.
S-4 Dialogue task primarily involve the fusion of visual data and natural language dialogue understanding. This task needs the model to understand and process multimodal data, including both textual and visual information, while demonstrat- ing consistency and complementarity in the task. The datasets involve Multimodal Dialogue (Li et al. 2022 for multimodal conversational question answering and Conversational Embodied Dialogue (Shridhar et al. 2020) for mapping from language commands and visuals to action sequences.
S-5 Space Understanding task requires the model to perceive the spatial environment using multi-image information. It uses the muscenes (Caesar et al. 2020) dataset, specifically designed for self-driving car technology. It contains sensor data for object detection and tracking, with 1000 annotated scenes for location and properties of objects.
5

---

Published as a conference paper at COLM 2024
1.3:9.3%
9-5933
Table 1: Key Statistics of MILEBENCH. Note that we use tokenizer of LLaMA2 to calculate the token number.
Image
Retnievol
Needle in HayStack
9. 9.3%
T-2 12:4%
99% Diagnostic Evel 19 3%
Comporoo Muiti image
Statistic
Number
Total samples
6,440
Total images
97,855
T:3 822
Average images
15.2
Range of images
2 "009
Realistic: Eval
Range of words
"18821
Semantic Multi -image
80, 7%
(Estimated) Average Tokens
T-4 12.4%
- Image token=0
342.2
Image token=32
1,028.4
40.4%
Imagee token=256
4,432.1
Image token=576
9,294.4
9.39
Samples by Image Num Level
Few (2)5)
2,959
Medium (6')1)
2,389
Figure 4: Tasks Distribution of MILEBENCH.
Many (32 109)
1,092
3.1.2 Diagnostic Evaluation
The diagnostic evaluation focuses on the MLLMs' capability to retrieve information without being distracted in multimodal long context. We transform the tasks of "Needle in a Haystack" from NLP and "Image Retrieval" from CV into a multimodal format for assessment. This transition preserves the core of the conventional tasks while offering more challenging and realistic measure of MLLMs' performance.
Needle in Haystack. The "Needle in Haystack" task requires the model to find preset password from a long context. This is widely used in diagnostic evaluations of long-context language models (Kuratov et al. 2024). In this study, we reintroduce this novel task from multimodal perspective to evaluate the context perceptual and retrieval abilities of MILMS.
Specifically, constructed two tasks, Text Needle In A Haystack (N-1) and Image Needle In A Haystack (N-2). Examples are shown in the lower left corner of Figure 3 and Figure in Appendix B.2 respectively. Compared to unimodal Needle in a Haystack task, Text Needle In A Haystack's s haystack includes both text and images and the model is required to recall randomly generated 7-digit password from this multimodal haystack. Image Needle In A Haystack changes the modality of the "needle", inserting it as text within an image. This cross-modal Needle in a Haystack task requires MLLMMs have not only good perceptual and retrieval abilities but also robust OCR capabilities.
Image Retrieval. Image Retrieval task (I-1) (Schall et al., 2022) requires the model to retrieve images from a set of candidates images given an anchor image (an example is shown in the lower right corner of Figure 3). addition to perceptual and retrieval abilities, this task also necessitates that the MLLM possesses robust abilities in both object and conceptual recognition. We consider this a traditional computer vision task and see image retrieval "Needle in Haystack" task with image modality queries.
3.2 Data Collection and Review Process
We have established a robust data collection process and meticulous review procedures to maintain the integrity and quality of our datasets.
Data Collection. We collected the samples from two sources: (1) For most of the tasks, we selected and sampled 200 instances from the test sets of the pre-existing datasets for each task, giving priority to multi-image samples. For video data, we used the Katna (Keplerlab) 2021) to convert the video into a multi-image format by extracting one frame per second. The choice of these pre-existing open-source datasets was driven by their well-established reputation and the fact that they have been published in top-tier conferences and journals, ensuring their credibility and reliability. Please refer to Section 5 and Appendix D.3 for information on the data's licensing and data contamination issue. (2) For the new tasks Long Text with Images QA, Image Retrieval, Text Needle In A Haystack, and Image Needle In A Haystack, we created synthetic data (more details in Appendix B.3). Ultimately, we collected 6,440 samples with varying context lengths. Detailed statistics of these samples are presented in

---

Published as a conference paper at COLM 2024
Table 1 The task distribution is demonstrated in Figure 4 A comprehensive breakdown of the datasets, tasks, and taxonomy can be found in Appendix B.1
Review Process. For the open-source dataset comprised of the benchmark, we sample 10% of the data for manual verification. Our review team, composed entirely of authors, was assigned to scrutinize the precision of the sampled data, resulting in an Inter-annotator Agreement (IAA) of 95%, indicating high level of consistency among reviewers. For the datasets we formulated independently, equivalent manual verification was carried out on the entirety of the dataset, yielding similar IAA of 98%, thus ascertaining the data quality. Additionally, the error rate was found to be less than 1% for both datasets, affirming that these datasets maintain an exceptionally high quality and are virtually devoid of errors.
5
Experiment
4.1 Experiment Setup
Evaluation Models. In this study, we conducted an evaluation of several models across three distinct categories that may handle multimodal long contexts, including five closed-
source models (GPT-4V [OpenA] 2023). GPT-4o, Gemini 1.0 (Anil et al. 2023). Gem-
ini 1.5 (Reid et al. 2024), Claude 3 Opus 6 twelve open-source image models (Qwen-
VL-Chat (Bai et al., 2023), MiniGPT-v2 [Cen et al. 2023), Cheetor (Li al.. 2023c),
Open flamingo (Awadalla et al. 2023), LLaVA-1.5-7B 13 (Liu et al. 2023a), LLa VA-1.6-
7B/13B (Lu et al. 2024b), ALLa VA-Longer (Chen et al. 2024), Yi-L Young et al. 2024),
VILA (Lin et al. 2023), Mantis (Jinng et al. 2024)), and five open-source video models (Video-LLaMA- (Zhang et al. 2023), Valley (Luo et al. 2023), VideoChat2 (Li et al. 2023d),
LLaMA-VID [Li et al., 2023e), LWM (Liu et al., 2024a)). The details of the models and and their version information are in Appendix C.1 All models used greedy decoding to generate answers, with designated generation length between 1 and 512. We conducted all experiments on NVIDIA A100 GPÜs.
Prompts and Metrics. To save costs, all evaluations were performed only in a zero-shot setting. The format of the prompts detailed in the Appendix C.2 When the input length exceeds the maximum context length of the model, we keep the instruction, and truncate the interleaved image-text question from left so as to keep the question of a sample, as instruction and question are critical information and the importance of the last image is higher in many tasks, e.g. multimodal dialogue. Metrics for each dataset, as shown in Table 4 are consistent with the original work for tasks built on previous datasets. For open-ended generation tasks, the popular n-gram-based metric ROUGE-L is adopted, and accuracy is the metric for multiple-choice and needle-in-a -hassta tasks.
4.2 Main Result on MILEBENCH
We present the results of our experiments in Table 2 and summarize our findings as fol- lows: (1) Closed-source MLLMs outperform open-source MLLMs in multimodal long- context tasks date, particularly in diagnostic evaluation of long-context adaptability where the gap between closed-source MLLMs (average: 79.2%, max: 99.4%) and open- source MLLMs (average: 10.1%, max: 37.2%) is significant. In realistic evaluation, all open-source models, except for VILA and Mantis, lag considerably behind. (2) Open- source image models generally perform better than open-source video models. Even the best video model, LLaMA-VID, falls short in realistic evaluation with 31.8%, score that is lower than eight image models. This may be due to the inability of video mod- els to capture detailed information in images in the same way that image models can. (3) Training with multi-image data can improve performance. Models like Qwen-VL- L- Chat, VILA, and Mantis that were trained on diverse multi-image datasets show notably better performance, demonstrating the benefits of varied visual inputs in multimodal learning. (4) The ability to adapt to long contexts and perform long-context tasks are not necessarily linked. For example, while Qwen-VL-Chat scores the highest in diagnostic evaluation among open-source models, it trails behind Mantis in task com- pletion (39.1% <47.5%) highlighting our evaluation's 's diversity and comprehensiveness.
5A degree of similarity among the annotations made by different annotators on the same data. htth://wwwwaathhppccco/ee/////ds
7

---

Published as a conference paper at COLM 2024
Table 2: Experiment Result on MICHBENCH. T-1 refers to the task number introduced in Section 3 NH and IR refers to Needle in Haystack and Image Retrieval. The highest scores for flosed-soures models, open-source image models, and open-source video models are marked in red, blue, and green respectively.
Model
Size
Temporal Mulgi-
Semanttc Multi-image
NH
IR
Overall
Overall
T-1
1-2
S-1
S-2
S-3
S-5
N-1
N-2
1-1
Real.
Diag.
Ramdows
25.0
25.1
24.6
0.8
0/3
22.3
5.5
Cloast-szuroe MLLMS
GPT-VV
51.7
50.1
22.3
58.5
82.8
77.3
11.1
81.0
99.7
99.1
86.7
53.0
99.4
GPT-lo
61.8
56.8
38.0
68.3
85.3
83.3
15.3
47.2
86.5
99.7
99.1
88.8
60.3
99.4
Gemini 1.0
41.5
46.0
27.3
51.6
74.4
64.7
16.9
47.8
73.0
73.1
27.5
12.5
49.2
37.7
Gemini 1.5
55.0
56.5
34.0
57.3
73.9
72.7
17.8
44.7
62.5
99.8
96.3
85.0
51.7
94.5
Claude Opus
37.7
59.7
72.5
Openvaaurree
rinsage models)
VoLager
3B
20.8
32.4
28.0
31.3
33.9
20.7
12.1
17.7
25.5
8.1
0.0
9.3
24.7
5.8
Yi-VI
26.8
57.1
34.2
33.55
35.5
12.5
0.0
7.7
Cheetor
7B
24.2
23.3
18.3
28.0
28.5
24.8
21.9
27.5
32.0
17.8
0.0
8.5
25.4
9.8
Owees-LL-Cht
34.7
37.3
52.3
35.3
62.8
37.2
LLaVA-15-7B
7B
37.8
45.5
31.5
46.4
30.8
32.0
11.3
24.2
62.5
0.0
0.0
6.2
38.0
2.1
MiniGPT--2
7B
9.2
14.4
16.0
20.3
34.5
25.2
766
17.0
16.5
15,9
0.0
1.2
17.8
5.7
VILA
0.3
12.8
LLaVA-16-78
7B
38.8
46.4
28.5
33.0
31.6
40.3
9.9
26.9
69.3
10.6
0.9
10.2
38.
72
Mantis
7B
5:.8
$9.9
25.3
18.9
65.8
55.5
17.4
31.9
78.0
27.5
0.0
32.
17.5
19.9
Open Eamingo
0.0
LLaVA-15-130
38.8
33.0
66.9
49.2
24.7
0.0
10.8
11AVA-16-188
32.8
47.5
23.3
33.0
39.1
34.0
9.5
25.6
72.5
6.9
1.9
10.0
37.5
6.
Ojensaneer M.J.M (Vain)) sundAts)
Video-LLMMA22
7B
0.5
3.9
1.3
6.9
11.5
3.3
4.7
3.6
10.0
0.0
0.0
4.5
5.1
1.5
Valley
7B:
17.0
29.8
12.5
27.0
18.7
18.7
7.6
26.9
31.0
0.0
7.7
10.5
21.0
6.1
VideoChat2
7B
10.8
27.1
12.8
15.9
12.1
21,8
14.4
21.0
31.5
3.4
0.0
2.7
22.3
2.0
LLaMA-VID
78
26.2
37.5
26.5
40.5
43.5
26.0
11.7
26.0
46.5
48.4
0.0
10.7
31.8
19.7
LWM
7B
0.5
7.3
14.0
107
15.0
30.6
0.0
10.2
LWM suffered from a significant decline in performance due to its training on video-text pairs in the final stage. However, after repeating image multiple times, we observed an improvement.
(5) Interestingly, the majority of open-source mod- els scored zero in the Image Needle in Haystack task. Upon inspection, we found that many of these models partially answered the needle nu- meric string without completely getting it right. This suggests that open-source models need to im- prove their ability to retrieve information from im- ages, particularly their OCR capabilities. Detailed results from the realistic evaluation can be found in Appendix [C.4 We also selected individual ex- amples for error analysis, details of which can be found in Appendix
Minti//ev2
Cheebor
Gemini
LLaV8-15-7 1577
LLAMANND
Gemini 1.5
ELatte11 66-7
Viteocl LAMA--2
Chicde Opox
VILA
Villey
ALE.VA
Manii
Vileochat2
Bantingo
IMM
VL-Chat
0.7
Performance
0.5
0.4
0.2
4.3 Analysis
0.0
Row
(.:0
Medium d6 331
Many B3 100)
Image Quantity Level
In this section, we delve into a meticulous analy- sis of the results, focusing on two research ques- tions that revolve around the MILEBENCH: "How do MLLMs perform given contexts with different lengths?" and "Do MLLMs also get Lost in the Middle in multimodal long contexts?" We also analyzed three important questions: "Does com- bined image help multi-image comprehension??" "How diverse and comprehensive is MILEBENCH?" and "Does data contamination issue exist in MIL.FRENCH?" Detailed analysis results are provided in Appendix D.1 Appendix D.2 and Appendix D.3. respectively.
Figure 5: Average performance across various levels of image quantity.
4.3.1 Performances Decline as the Number of Images Increases for Most MLLMs
To investigate the performance of MLLMs with varying numbers of images, we divide our dataset into three levels: Few, Medium, and Many, based on the number of images per sample. The specific quantities for each level can be found in Table average performance of the model on the three types of data with different numbers of images. It can be observed that as the number of images increases, the performance of most models significantly declines (as indicated by steep slope in the curve), especially for the LLaVA-1.5 series models. This is likely because most models have only been trained on single image, resulting in insufficient generalization for multi-image test data. However, the performance of GPT-4V, GPT-4o, Gemini 1.5, Claude 3 Opus and Qwen-VI-Cha on the
the
8

---

Published as a conference paper at COLM 2024
5.0
Dopth Percent
1.
Depth
1.4
-62
Token Limit
Token Limit
(a) TextNeedle GPT-4V
(b) TextNeedle for Qwen-VI. VL-Chat
100
Deptn Purcent.
Degtth Purvent
12
000000
Limit
Image Limit
(c) ImageNeedle for GPT-4V
(d) ImageNeedle for Owen-VL-Cher
Figure 6: Visualization of results varying in depth and context length in needle haystack. The x-axis represents the number of tokens images in the context, while the y-axis indicates the depth of the context where the needle resides. Green squares indicate successful extraction of the needle at that position, while red squares denote failure.
Medium level surpasses that of the Few level. This could be attributed to their training mulli-image data, where a larger number of images can provide more information to some extent, thereby aiding the model in task completion. Despite their outstanding performance on multi-ime mule tasks, their performance still declines when the number of images reaches the Many level. This leaves room for future development in modeling for multi-image context.
4.3.2 "Lost in the Middle" for MLLMs
Liu et al. (003b) pointed out that in needle-in-a-haystacck tasks involving long texts, models may experience the "Lost the Middle" phenomenon, where they struggle find the needle located in the middle of the context. We investigated whether MLLMs would exhibit multimodal contexts. We chose the two best- performing models from closed-source and open-source models in the Needle in Haystack task for analysis. As can seen from the results Figure 6 MLLMs displayed varying behaviors. In multimodal long contexts, GPT-4V did not "get lost in the middle" and managed to complete the two tasks impressively with the scores 99.7% (N-1) and 99.1% (N-2). On the other hand, ignoring the scenarios where the data exceeds its maximum context length (8192 tokens or 32 images) and gets truncated, Swenn-V-CChe showed certain degree of "lost the middle", particularly evident in the image needle task. This indicates that the "lost in the middle" phenomenon also exists in multimodal scenarios. However, a strong ability manage long context can significantly reduce this risk.
the "Lost the Middle" phenomenon
Conclusion and Future Directions
In this study, we introduced MILEBENCH, a pioneering benchmark designed to rigorously evaluate the multimodal long-context capabilities of MLLMs. We have established the diagnostic and realistic evaluation sets, designed to systematically assess the MLLMs' ca- pacity for long-context adaptation and proficiency in task completion within these contexts. Despite some impressive performances, our experimental results underscore the urgent need for more focused research enhance MI.LMs' capabilities in these complex scenarios.
Moving forward, we suggest two primary research directions: (1) Long-context MLLM: Given the ubiquity mixed media content, there pressing need for models that can adeptly process multiple images in long-context scenarios. (2) Scaling MILEBENCH to Larger Contexts and Other Modalities: As real-world tasks continue evolve, benchmarks should also adapt, incorporating larger contexts, complex task structures, and additional modalities to stimulate the development of more versatile MLLMs. These efforts will help equip MI.LMs better for our increasingly multimodal world.

---

Published as a conference paper at COLM 2024
Limitation
The limitations of this study include that the results from closed MLLMs may vary over time, and there is a risk that some of the test data may subject to leakage in the future.
Ethics Statement
The dataset we're using is an aggregation of publicly accessible datasets licensed under the Creative Commons license (CC-BY) or other open-source licenses. We've meticulously ad- hered required legal procedures to incorporate this data into our research, recognizing the importance of transparency in data licensing for proper attribution and suitable data utilization. Our dataset also encompasses images derived from publicly accessible datasets and language data created through the GPT-4V API. While measures have been put in place to secure suitable content, we acknowledge the potential existence of problematic content. Should you come across any such content, urge you to inform us immediately make the necessary adjustments to sustain dataset free from inappropriate content. We unwavering in our commitment maintain high-quality, ethically responsible dataset and promise to uphold principles of privacy and transparency throughout our work.
can
Acknowledgement
This work was supported by the Shenzhen Science and Technology Program JCYY2022081810300000)) Shenzhen Doctoral Startup Funding (RCHS5022100803330065) Tianyuan Fund for Mathematics of National Natural Science Foundation of China (NSFC) (12326608), Shenzhen Key Laboratory of Cross-Modal Cognitive Computing (grant number ZDSYS202306260913020000 and Shenzhen Stability Science Program 2023, Shenzhen Key Lab of Multi-Modal Cognitive Computing.
10

---

Published as a conference paper at COLM 2024
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Has- son, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Mon- teiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning, 2022.
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable mul- timodal models. CoRR, abs/ 2tt.//331.111100 2023. doi: 10.48550/ ARXIV URL https://doi. wrg.///...rg////.0 10. 48550/arXiv 2312 11805
Anas Awadalla, Irena Gao, Josh Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision- language models. CoRR, abs /atbs//30001100 2023. doi: 10.48550/ https://dol: org/10. 18550//ariiig 23w8.. 01390
0044850/ARXIV.2800 ARX.V URL
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-vl: A frontier large vision-language model with versatile abilities. CoRR, /ttp///3....9./. 2023. doi: 10.48550/ ARXIV.2308.12900 URL org/10 10. 48550/ wttpv//arivv... 2308. 12966
https://doi
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In 2020 IEEE/CVF Conference on Com- puter Vision and Paltern Recognition, CVPR 2020, Seatfle, WA, USA, June 13-19, 2020, pp. 11618-11628. Computer Vision Foundation IEEE, 2020. doi: 10.1109 /CVPR4260) 2020.01164. URL https:://ppeacceeeet thecvf. cwm/ www.ttent.cvR..2222 wtt.22220/ttm//assaaa nuScenes Multimodal Dataset for Autonomous Driving CVPR 2020 paper.htpp
Yingshan Chang, Guihong Cao, Mridu Narang, Jianfeng Gao, Hisami Suzuki, and Yonatan Bisk. Webqa: Multihop and multimodal QA. In JEEÉ/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 16474- 16483. IEEE, 2022. doi: CVPR52688. 2022.022222 01600
URL hntpps://dols orw.:o.rgg/.... 1109/
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing sptdv-synthesized data for A lite vision-language model. CoRR, abs/2402.11664, 2024. https://:://do org/10 48550/arXiv....//// 2402. 1168
doi: 10.48550/ 10:4855//RXXIV224022 AXX.V.22402.166.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt- v2: large language model as unified interface for vision-language multi-task learning. CoRR, /etp///331009747 2023. doi: 10.48550/ 00.4550//ARXV.231000.... A1X.V.2310.09478. URL
48550/arXiv. 2310. 2310.09478
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehen- sive evaluation benchmark for multimodal large language models. CoRR, abs/ 13306.13394,
11

---

Published as a conference paper at COLM 2024
2023. doi: 13394.
URL http:/ss//0....0////44455///////
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. TALL: temporal activity local- ization via language query. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 5277-5285, IEEE Computer Society, 2017. doi: M4.100//CCCV2000
URL https:t//ti/oorrrrtterr 10t.://...0//...../...0.00 2017. .53
Wentao Ge, Shunian Chen, Guiming Chen, Junying Chen, Zhihong Chen, Shuo Yan, Cheng- hao Zhu, Ziyue Lin, Wenya Xie, Xidong Wang, Anningzhe Gao, Zhiyi Zhang, Jian- quan Li, Xiang Wan, and Benyou Wang. Mllm-bench, evaluating multi-modal llms using GPT-4V. CoRR, abs/2311.13351, 2023. doi: 10.48550/ ARXIV.2311.13951. URL https://doi. https::/doi.oor//00 10. 48550/arXin 2311 13951
Darryl Hannan, Akshay Jain, and Mohit Bansal. Manymodalga: Modality disambiguation and QA over diverse inputs. In The Thirty-Fourye AÅAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7879-7886. AAAI Press, 2020. doi: 10.1609/ AAALV34105.6299 URL httphtttp///d...coo
10. 6099/aai. htt.//aaaivv3........ 6294
Mehrdad Hosseinzadeh and Yang Wang. Image change captioning by learn- ing from an auxiliary task. In TEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 2725-2734. Computer Vi- URL https://opemaccess.. thecvf. http:///ooeeeeet///////////////e Imwgeeechenmmmaeeeeeaaeeeee html]
sion Foundation IEEE, 2021. doi: 111009/VVPR44437220000
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocatin. arXiv preprint arXiv:2311.17911, 2023.
Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV, volume 12349 of Lecture Notes in Computer Science, pp. 709-727. Springer, 2020. doi: 00000779783-0005500 88 41. URL httpp:://oi.. org/ 10. 1007/978 3-000- 58548-88
Harsh Jhamtani and Taylor Berg-Kirkpatrick, Learning to describe differences between pairs of similar images. In Ellen Riloff, David Chiang, Julia Hockenmaier, and lun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. Computational Linguistics, 2018, doi: 10.18653/V1/DDD 1436. URL https:: 000/10 18653/v1/d18- 1436
Association for
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. ar preprint ar Xrxi:2405.00455 2024.
Aniruddha Kembhavi, Min Joon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirii. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 5376-536 5384. IEEF Computer Society, 2017. doi: 10.1109/CVPR.2011
URL https: / /toi://oiiiooo 0tt.:oi.rr/11..101000 1109/
Keplerlab. Katna: Keyframe extraction library for optimal video summaries. https:// 2021.
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision-and-langugge navigation in continuous environments. In Andrea
12

---

Published as a conference paper at COLM 2024
Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIIII, volume 12373 of Lecture Notes in Computer Science, pp. 104-120. Springer, 2020. doi: 10.1007
7. URL httpp:://oii.og
Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. In search of needles in a 10m haystack: Recurrent memory finds what Iims miss. Xiv preprint ar axXiv:2402.10990 2024.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench-2: Benchmarking multimodal large language models. CoRR, abs/2311.11092, 2023a. doi: 10.48500/ARXXV.2311..77 17092
URL hhtpps: wrw..//.....0///.........ooooo
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal IIms with generative comprehension. CoRR, abs abs
2023b. doi: 10.4550//RRXI.22077112 16125
URL /ttts:://doi..rg//1....e
Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, and Yueting Zhuang. Empowering vision-language models to follow interleaved lision-languag instructions. CoRR, abs/2308.0000 04152, 2023c. doi: (0.48550/ARXIVV22 22..2...00....1.... URL http:::/o/// http://.............//////////// 48550/arr100/... 2308. 04152]
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video understanding benchmark. CoRR, abs /ttp//311..00000 2023d. doi: 10.48550/ ARXIV. 2311.17005. URL htttp:://doi.org/11/ wrr..////.....//r//....... 2311 17805
48550/arXiv....
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video understanding benchmark, 2024.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. CoRR, abs/ /ttps/231...000. 2023e. doi: 10.48550/ ARXIV.2311.1043. URL https://doi. org/10. 48550/aa5ii..... 2311. 17043
Yongqi Li, Wenjie Li, and Liqiang Nie. Mmcoqa:
Conversational question answering over text, tables, and images. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 4220-4231. Associ- ation for Computational Linguistics, 2022. doi: (0118633/V112222AA4L-2 https://doi org/ / 10. 18653/v1/2022 ac1 -alllclllnnaeea long.
URL
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. VILA: on pre-training for visual language models. CoRR, abs 2312.0733,, 2023. doi: 10.48550/ AA0555////AII1.22207533... URL https //doi. 40111e..555//ariiv 231.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. CoRR, /b00//24000666 2024a. doi: 10.48550/ ARXIV.2402.05268. URL URL/https://6oi.rr//// 48550 art.v.///axiv.... 2402, 08268
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. CoRR, abs/ /0///3310.03744, 2023a. doi: 10.48550/ ARXIV.2310.03744 10:40//////AA/Y22100000444 URL
https://doi.org org/10 03550/arXiv.23144 03744
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL http::/www.....iiiooooooo/////22 -aaaa-a-eeaa-eettt
https://llava-vl. thub. wtt.iiooooo//4-0000000
13

---

Published as a conference paper at COLM 2024
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. CoRR, abs/ abs//230..01122 03172, 2023b. doi: 10.48550/ARXIV22222 ARXIV230070011722 URL https://doi.org:
http://aaxxv..22......
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? CoRR, abs/2307.06281, 2023c. doi:
10.48550/ 10.48550/ARXI1220077 ARXIV.2307.06222222222
URL /ttp:://///oo..//////// 48550/arXii
Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, and Xiang Bai. On the hidden mystery of OCR in large multimodal models. CoRR, abs 2330500889, 2023d. doi: 10.48550/ ARXIV. 2305.07895. https//s:///o1. http::///ooooorgg.... 48550/048550///// arxiv. 2305. 07895
Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced
ability. CoRR, 000022306007007 2023. doi: 10.48550/ ARXIV URL https 2306. 072077
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for VQA on document images. In IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Walkoloa, HI, USA, Jamuary 3-8, 2021, pp. 2199-2200 IEEE, 2021. doi: 10.1109/ WAACV483302021100222
URLhttts:://oi..ook org.0/....../.... /09////C48830.0... 2021.00225
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCR- VQA: visual question answering by reading text images. In 2019 International Conference on Document Analysis and Recognition, ICDAR 2019, Sydney, Australia, September 20-25, 2019, pp. 947-952. TEFE, 2019. doi: (0.110//CCDAR.200 000055.. URL httpt:tt///oooooo
10. 19.1199/CDAR 2019 00156
OpenAl. GPT-4 technical report. CoRR, abs abs /00///333308774 2023, doi: 008550/ARXX.22033 08774. /tt/::t::://ooo.rrg//
48550/arXiv. 005550/arXiv.233..88111 2303.08774
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adrià Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alexandre Fréchette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Si- mon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception test: A diagnostic benchmark for multimodal video models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, Decem- ber 10- 16, 2023, 2023. URL http::/ nips. cc/paper fi fiwes/paper /2t/3/2222///// Bwwccaaan...naass....m html
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint ar arXio:2403.05530, 2024.
Konstantin Schall, Kai Uwe Barthel, Nico Hezel, and Klaus Jung. Gpr1200: a benchmark for general-purpose content-based image retrieval. In International Conference Multimedia Modeling, pp. 205-216. Springer, 2022.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han,
Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark
for interpreting grounded instructions for everyday tasks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, Pp. 10737-10746. Computer Vision Foundation / IEEE, URL https://openaccess:
2020. doi: 10.1109/ /t/p//CVR222202222222 220.000..22.00075... content CVPR 2028/html/Shridhan ALFRED Benchmark for Interpreting Grounded Instructions for Everyday Tasks CVPR 2020 paper.
cnecvf.comm
paper.html html
14

---

Published as a conference paper at COLM 2024
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodalga: complex question answering over text, tables and images. In 9th International Conference on Learning Repre- sentations, JCLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
Hao Tan, Franck Dernoncourt, Zhe Lin, Trung Bui, and Mohit Bansal. Expressing visual relationships via language. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 1873-1883. Association for Computational Linguistics, 2019. doi: 10.18653/ /V1////1PP191//// URL 18613/11/1119--111/
https://doi.org// http://b...rrgi111..16653///////
Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. In Brian Williams, Yiling Chen, and Jennifer Neville (eds.), Thirty-Sevent AAAl Conference on Artificial Intelligence, AAAI AAAII 2023, Thirty-Fifth Conference Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pp. 13636- AAAI Press, 2023. doi: 10.1609 AAA1..37711.26599 URL https: //tpps::/doi.ooo org/10.g///oooo 1609/aaai v37111.
Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, Huaxiu Yao, and Furong Huang, Memen- tos: A comprehensive benchmark for multimodal large language model reasoning over image sequences. CoRR, abs/ /00.//400.1122 2024. doi: 10.48550/ 005500/ARXIV.2401100222 URL https://doi. org/ 405.0/8550/r/ www..iv......1...0 2051.
Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang. Xilin Luo, Xuejie Wu, Lu- nan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanqi Sun, Yifu Chen, Yongyi Peng, Xiaojuan Liang, Shuicheng Yan, Han Fang, and Yahui Zhou. Skywork: A more open bilingual foun- dation model. CoRR, abs/ 2sss/331..13311 2023. doi: 10.48550/ ARXIV.2310.19341. URL https://doi. https://doi.orr/10 48550/arXiv. 2310. 19341
Bo Wu, Shoubin Yu, Zhenfang Chen, Josh Tenenbaum, and Chuang Gan. STAR: A benchmark for situated reasoning in real-world videos. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NewrlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. httpp::://aattsttt--eennaasssss wwwpaardiir..ooooeeiiiii........ neur witrrppsccom http:ccccpppppeeeeee/eeeeee /2021 /hash/ Tet25999bb/9aaa44444000002722222
Yang Wu, Shilong Wang, Hao Yang, Tian Zheng, Hongbo Zhang, Yanyan Zhao, and Bing
Qin. An early evaluation of gpt-4v(ision), CoRR, abs/ 2at.. 16534, 2023. doi: 10.48550/
ARX1V.2310.1533.. URL /tt/ttts:: /ttt:s///ooo..or///..8855////// 48550//555axii 2310. 16534
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with git-ivi CoRR, abs/ 2209, 17421, 2023. doi: 48550/arXiv. 230.x....239..744 17421
URL
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. CLEVRER: collision events for video representation and reason- ing. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL Hd=HKXYZANYDB
netfoorumm
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. ar Xiv preprint ar Xrx3::2403.0453 04652,, 2024.
15

---

Published as a conference paper at COLM 2024
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. CoRR, 230.00240800 2023. doi: 10.48550 0438550/ARX1122388802249999 URL https
//doi.org/10. wttp 2308. 02490
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. CoRR, abss2231..16500 2023. doi: 10.48550/ ARXIV.2311.16002. URL http::://oi.oorg/// axiii//.vi///xii...o. 2311. 16582
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-turre audio-visual language model for video understanding. In Yansong Feng and Els Lefever (eds.), Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 System Demonstrations, Singapore, December 6-10, 2023, pp. 543-553. Association for Computational Linguistics, 2023. URL httpss//accaaattoooooooo///////
emnlp- demo. 49
16

---

Published as a conference paper at COLM 2024
Appendix Table of Contents
A
Comparison of Current MLLMs Benchmarks
18
More Information on MILEBENCH
18
B.1 Details of Taxonomy
18
B.2 More Examples in MILEBENCH
19
B.3
Construction Process of Synthetic Data
21
More Information on Experiments
21
Details of Evaluation Models
21
Details of Input Prompt Construction and Instruction Application
24
Error Case Study
26
4 Detailed Experiments Result
26
D More Analysis Results
26
D11 Experiment on Combined-image Set
26
D22 Inter-task Correlation in MILEBENCH
27
D.3 Risk of Data Contamination for MILEBENCH
28
17

---

Published as a conference paper at COLM 2024
A Comparison of Current MLLMs Benchmarks
As shown in 3 we enumerate the ten most extensively utilized MLLM benchmarks used recently. These benchmarks are compared across six dimensions: Visual Modality, Customized Question, Average Images, Max Images, Total Samples, and Question Type. It is noteworthy that our benchmark presents greater challenge, particularly in the areas of Average Images and Max Images, in comparison to other benchmarks. Simultaneously, in terms of data modality, question types, and quantity, it is comparable to existing works.
Table The Comparison of MLLMs Benchmark. OE: open-ended. MC: multi-choice
Benchmark
Visual Modality
Average
Max
Total Samples
Question Type
Images
Images
Single-image Benchmarks
LLaVA-Bench
Image
1
150
OE
MME
Image
2,194
MC
MMBench
Image
1
2,974
OE
MM-Vet
Image
1
218
OE
MLLM-Bench
Image
1
420
MMMU
Image
1
11,550
OE & MC
SEED-Bench
Image & Video
1
19,242
MC
Multi-inage Benchmarks
SEED-Bench2
Image Video
2.7
24,371
MC
DEMON
Image Video
3.5
11
18,176
OF MC
Mementos
Image & Video
11.6
26
4,761
OE
MILFBENCH
Image Video
15.2
109
6,440
OF MC
B More Information on MILEBENCH
B.1 Details of Taxonomy
In Table we present a detailed taxonomy of the dataset, task composition, as well as the number of samples and metrics corresponding to each task.
Table Detailed Statistics and Taxonomy of MILEBENCH.
Categury
Task
Datavet
Data Source
Count Metric
Realistir Fooeuution
Action Undentanding and Prediction
Action Localization
Kan 200
Accuracy
Accunacy
Sequence
Object and
Object Existence
CLEVRER
11
700
Accuracy
Understanding (T-2)
Object Interaction
STAR
WWaeee PU14
230
Accunacy
Temporal Mulii image
Mowing Attribue
CLEVV
ME
10710
2.0
Accuracy
Soullle
Pereeption Tex Plcnikar De
2023
200
Accuracy
Navigation Spatial Localiration
Egueentric Navigation
Kryntzst
Ascunacy
Moving
Counteriactual Reasuning and State Change (I-4)
Counteriat tual Infereoor
CLEVRER
Si Mi
2021
230
Aucunacy
State Change
Penception Test
2022
230
Accuracy
Chanater Order
Perorption Test
200
Accuracy
Scane Transtion
MovsoNet
Filiin 14 mi kimm
200
Accuracy
Knuwledge Grounded (5-1)
Wibpagg QA
WerQA
Chanres
Accunacy
Accuracy
Multimedal Text with Images
MuldfMudniOO
Accuracy
Text-Rich Images QA (8-2)
Slide QA
SlideYOA
Chntka 2320
230
Accuracy
Semantic Multi-image
OCR OA
OCRAVQA
200
Accunacy
Document QA
DocVQA Nathor 1F 1014
200
Accuracy
Relation
Viraal Change Captiontnn
CLEVR-Chump
ROUGEL
Vissal Relationship Expressing
Iedir
ROUGE-L
Dislogue (5-4)
Multinodal Dalaga
JMMCQQ In Airilaa
200
Accuracy ROUGE-
Convirstiocal Emiledeed Dislogue
ALFIED Seatiin bille andll
200
Space Understanding (5-5)
Space Undestanding
Aocuracy
Digguottic Evulustion
Needle In
Text Needie (N-I
Text Needie in Haystack
Mockeeiiinntaassted
320
Accuracy
A Haystack
Image Needle IN-21
Image Noedle In Haystack
ImageNodidd
320
Accuracy
Image
Image Ketrieval
Imraee Retneval
GPRI210
Accuracy
18

---

Published as a conference paper at COLM 2024
B.2 More Examples in MILEBENCH
We provide additional examples from each dataset of the temporal multi-image and semantic
multi-image, and Image Needle In A Haystack task, illustrated in Figure / Figure [
and Figure 9 respectively. In the instance of Image Needle In A Haystack task, text needle embedded within the first image.
Temporal Multi-image
Action Understanding and Prediction (T-1)
Object and Scene Understanding (T-2)
Instruction: Review the supplied imagery, and find the hidden object. You must choose your answer from Choice List. Question: The person uses maltiple similar objects to play an occlusion game. Where the hidden object the end the game from the persoe's 's point of view?
Instruction: Examine succession events the supplicd visuals und answer the relevant query. You must choose your answer from the Choice List.
Question: happened after person opened closet/cabines?
Choice List:
Cholce List:
A. Under the first object from the left.
A. Put down the towel. B. Put down the clothes.
B. Under second object from the left.
C. Put down the food. Threw shoe.
C. Under thind object from the
Ground
Ground Truth: B. Under the second object from the left.
Visual Navigation and Spatial Localization (T-3)
Counterfactual Reasoning and State Change (T-4)
Inspect the presented illustrations and conclide the scene transition. You must choose your answer from the Choice
Instruction:
Instruction: From presented pictures and instructions, infer course
You choose your
Choice List.
Question: Can you choose option that matches how scenes change the video?
Question: This is navigation video of an agent following insiruction: "Gio down the stairs your right and stop when
Choice List:
get way
should
bedroom rooflap.
garden..
Choice List:
krtchen
A. Turn left and move forward. B. Step.
C. From the office to the park..
Mo forward.
D. Turn right and forward.
D). From the living room to the beach.
Ground
right move forward.
Ground Truth: From the kitchen the rooftop.
Figure 7: Examples in temporal multi-image category.
19

---

Published as a conference paper at COLM 2024
Semantic Multi-image
Knowledge Grounded QA (S-1)
Dialouge (S-4)
Instruction: Upon viewing pictures cuptured from cameras on a street view vehicle, your role is to answer about the presented unswer from the Choice List.
You choose your
Your Main Goal: Read book using the light of lamp. Step Details:
Instruction: Examine the succession of events in the supplicd visuals and answer the relevant query. You must choose your
Question: Question: Identify deltoid the following images:
Choice List:
A. U. C.P.
D.
Ground
Stepfil: Turn to your right and walk to the desk.
Stepá2: the book
StepR3:
Question:
Rirt-Rich Images (S-2)
Approach the lamp far right of desk.
Current Step:
desk.
Ground Truth: Turn the lamp.
Instraction: Review the supplicd imagery, and find the hidden your from Question: Question: In which markett was the number of higher: Central Neida
Space Understanding (S-5)
object. You
weighted average BSP at the end of Q1-2014 was 2,991?
List:
A. the area with weighted average BSP of 2,876 Noids.
C. the area with weighted average BSP of 2,658. weighted average
Ground Truth: B. Central Noida.
Visual Relation Inference (S-3)
Instruction: Upon viewing six pictures captured from various
vehicle, your questions about the presented scene. You must choose your List.
answer
Instruction:
Identify the transformations the second image
compared
Question: Differnce::
Ground Truth: the after image has person in a blue shirt. the after has walking before the person blue the image dark person slanding by big
Question:
how many yellow tracks are there?
Choice three
Ground Truth: A. three
Figure Examples in semantic multi-image category.
Image Needle HayStack
precise responses to any queries regarding displayed images.
Context:
Question: What special magic Delhi namber in the imagus?
Ground Truth: 2854716
Image
Image
Figure 9: An example in Image Needle In A Haystack task.
20

---

Published as a conference paper at COLM 2024
B.3
Construction Process of Synthetic Data
Prompts
Please construct a multiple-choice question using the provided images and accompanying text. The question must be relevant to multiple images. The four answer choices should be as comparable as feasible, with only one correct answer. Present your response in the following structure:
'Question: <Question>>
Options: <Option 1><Option 2> 2><Option 3> 3><Option 4>
Answer: <Answer>'.
Figure 10: Prompts for WikiVQA data construction.
Within the MILEBENCH, three datasets were constructed by our authors. We employed systematic approach for our task-specific data construction.
The Long Text with Images QA (WikiVQA) is derived from multimodal English Wiki documents. We collected documents that contain more than two images and have text length ranging between 1000 and 8000 words. For each data entry, we extracted images and their associated captions from each document and leveraged GPT-4V generate a question, options, and an answer, based on a prompt illustrated in Figure 10 Subsequently, we structured each data entry of WikiVQA by using the original multimodal document as the context, accompanied by the question, options, and the corresponding answer.
For Image Retrieval, we used images from the GPR1200 (Schall et al., 2022), ) which encompasses image data collated from a myriad of sources across 120 categories. For each data entry, we select two images from one category as anchor and positive samples, and then uniformly sample 4 to 62 images from other categories as negative samples.
Both Text Needle In A Haystack and Image Needle In A Haystack tasks involve sampling between 2 to 64 images. For Text Needle In A Haystack, a text needle containing the password is embedded into 10 equidistant positions within a long text fragment, sampled at ratio of 1 image to 250 words. Conversely, the Image Needle In A Hayslack task involves embedding the needle directly into the images themselves, placed at 10 equidistant positions.
C More Information on Experiments
C.1 Details of Evaluation Models
We provide basic information of benchmarked models in Table as well as brief introduction for each model to highlight their characteristics:
GPT-4V (OpenAI 2023) and GPT-407 are developed by OpenAl and are deemed the most powertul vision-language models for comprehension and generation. We report testing GPT-4V (opt-4-tubo-2204--44-00 on April 12, 2024 and GPT-4o Sopt-4o-2224-05-11 on May 2024.
Gemini 1.0 (Anil et al. 2023) and Gemini 1.5 RRid et al. 2024 are models developed by Google, demonstrating competitive ability among closed-source MLLMs. We report testing Gemini 1.0 (gemini-1, e-pro-vision-e
on March 20, 2024, and Gemini 1.5 (gemini-1. 5-pro-latee on April 12, 2024. Gemini 1.0 has a limitation of accepting up to 16 images as input.
Claude 3 Opu is vision-language model recently released by Anthropic, impress- ing the community with its extra-long 200K context length. We report testing Claude
http://wwwwaaahhppccccc//////e/s
21

---

Published as a conference paper at COLM 2024
Table 5: Sizes, context length, number of tokens per image and multi-image training infor- mation of benchmarked models.
Model
Size
Context Length
# Tokens per Image
Multi-Image Training
Closed-source MLLMs
GPT-4V GPT-4o
/
128K
/
128K
Gemini 1.0
12K
Gemini 1.5
1M
Claude Opus
200K
Opencsource MLLMs (Image models)
ALLgvA-Longer
3B
2048
Yi-VL
6B
4096
X
Cheetor
7B
4096
Ch-Cht
7B
8192
256
LLaVA-1.5-7B
7B
4096
576
MiniGPT-v2
7B
4096
256
VILA
7B
4096
576
LLaVA-1.6-7B
7B
4096
576
Mantis
7B
8192
576
Open flamingo
9B
2048
256
L.vVA-1.5-13B
13B
4096
576
LLaVA-1.6-13B
13B
4096
576
X
Open-source MLLMs (Video models)
Video-LLaMA-2
7B
4096
32
Valley
7B
4096
256
VideoChat2
7B
2048
96
LLaMA-VID
7B
4096
2
LWM
7B
1M
257
3 (claude-3-opel 3000--0---pu--2222229) March 16, 2024. Claude 3 Opus has a limitation of accepting up to 20 images as input
LLaVA-v1.5 (Liu al. 2023a) introduces a simple yet effective modality alignment strategy, based on which many other models are developed.
MiniGPT-v2 (Chen et al. 2023) is a concurrent work with LLaVA-v1.5. It adopts similar structure to the latter and is grounding-enhancee model.
LLaVA-v1.6 (Liu et al. 2024b) takes step further on the basis of LLaVA-v1.5. It is able process images with any resolution and releases more variants based on different L.Mss.
ALLaVA-Longer (Chen at 2024) is a lite version of LLaVA with enhanced complex reasoning ability, even achieving competitive results with larger models.
Yi-VI. (Yungg et al. 2024) is based on LLaVA architecture and adopts 3-stage
training process.
Visual Prompt Generator to capture residual visual details, which might be vital for model performance.
Cheetor (Li et al. 2023c proposes
Qwen-VL-Chat (Bai et al. 2023
7B model trained on billions of multimodal
samples.
VILA (Lin al.
2023 adopts LLaVA-like structure but is trained with interleaved image-text data.
Mantis (Jiang et al. 2024 uses LLaMA3 as language model and is trained with
multi-image data.
Open flamingo (Awadalla et al. 2023) the open-source implementation of Flamingo (Alayrac et al. 2022), which serves as the foundation of subsequent multi-image and video comprehension models.
hhtt::///stthbbccmm/mmmaalaam//a
22

---

Published as a conference paper at COLM 2024
Video-LLaMA-2 [Zhan et al. 2023) is an MLLM that can process vision and audio data by adopting two distinct encoders and Q-Formers.
Valley (Luo et al. 2023) video comprehension model that uses a temporal mod- eling module to encode video inputs into embeddings, which are then concatenated with textual features to perform downstream tasks.
VideoChat2 (Li et al. 2023d) trains Q-Former to extract visual features, and the
LLM remains frozen with the added LoRA layer tuned.
LLaMA-VID (Li et al. 2023e) conducts an extreme compression of images, repre- senting an image with only 2 tokens.
LWM (Liu et al. 2024a is progressively trained with multimodal data to extend its context length. is by far the only open-source MLLM that can handle up to 1M tokens its inputs.
23

---

Published as a conference paper at COLM 2024
C.2 Details of Input Prompt Construction and Instruction Application
In Figure 11, we illustrate the prompt structure employed during our evaluation phase. We crafted unified input prompt structure for each dataset, incorporating dataset-specific instructions. For detailed view of the specific instructions utilized, please refer to Table for temporal multi-image tasks, Table semantic multi-image tasks, T for diagnostic evaluation.
Prompts
Prompt for Multi-choice QA: Instruction: [Instruction]
Question: [Interleaved image-text question]
Choice List: {choices)
Answer:
Prompt for Open-ended QA: Instruction: (Instruction]
Question: [Interleaved image-text question]
Answer:
Figure 11: Prompts for evaluating models.
Table 6: Instructions for the datasets in temporal multi-image tasks.
Dataset
Instruction
Action Localization
Analyze the provided visuals and determine the timing of the event in question. You must choose your answer from the Choice List.
Action Prediction
Analyze the provided visuals and forecast the individual's subsequent move. You must choose your answer from the Choice List.
Action Sequence
Based the provided images, answer the question related the sequence of action You must choose your answer from the Choice List.
Object Existence
Based the provided images, answer the question related the existence of objects. You must choose your answer from the Choice List.
Object Interaction
Based on the provided images, answer the question related to the interaction of objects. You must choose your answer from the Choice List.
Moving Attribute
Based on the provided images, answer the question related to the moving attribute. You must choose your answer from the Choice List.
Object Shuffle
Based on the provided images, and find the hidden object. You must choose your answer from the Choice List.
Egocentric Navigation
Analyze the provided visuals and instructions, then determine the subsequent step. You must choose your answer from the Choice List.
Moving Direction
Based on the provided images, answer the question related to the moving direction. You must choose your answer from the Choice List.
Counterfactual Inference
Based the provided images, answer the question related the counterfactual from the Choice List.
inference. You must choose your
State Change
Analyze the provided visuals and determine the change of the state in question. You must choose your answer from the Choice List.
Character Order
Analyze the given visuals and answer question about the order of character. You must choose your answer from the Choice List.
Scene Transition
Analyze the provided visuals and determine the transition of the scone in question. You must choose your answer from the Choice List.
24

---

Published as a conference paper at COLM 2024
Table 7: Instructions for the datasets in semantic multi-image tasks.
Dataset
Instruction
Webpage QA
I will give you several images and question, your job is to seek information in the slide and answer the question correctly. You must choose your answer from the Choice List.
Textbook QA
Provided with series of diagrams from textbook, your responsibility to correctly answer the following question. You must choose your answer from the Choice List.
Complex Multimodal
Given collection of relevant data, which includes images, text, and tables, your task is to respond accurately to the ensuing question. You must choose your answer from the Choice List.
QA
Long Text with
Analyze the given context and associated images, draw inferences from the combination of both, and provide responses to posed questions.
Images QA
Slide QA
will give you several slides and a question, your job seek information slide and answer the question correctly, You must choose your answer from Choice List.
OCR QA
I will give you two pictures the book cover. Please look at pictures and answer question You must choose your answer from the Choice List.
Document QA
I will give you some pictures, and each group pictures will correspond to a question. Please answer it briefly. You must choose your answer from the Choice List.
Visual Change Captioning
What's the difference between 2 images?
Visual Relationship Expressing
Please give editing Request to describe the transformation from the source image to the target image.
Multimodal Dialogue
Provided with variety of pertinent information, including images, text, tables, and previous Q&A history, your role is to answer the upcoming question accurately.
Conversational Embodied Dialogue
Give you main goal, your job is to figure out what to do now by looking at current envirments. Your past views as well as decisions are also provided.
Space Understanding
Given six images taken from different cameras street view car, your task is to answer questions about the depicted scene. You must choose your answer from the Choice List.
Table 8: Instructions for the datasets in diagnostic evaluation.
Dataset
Instruction
Text Needle Hayslack
Answer user question conciscly and directly based on the provided images.
Image Needle In A Haystack
Answer user question concisely and directly based on the provided images.
Image Retrieval
Given the anchor image and the candidate images, identify which candidate image is most visually similar to the anchor image.
25

---

Published as a conference paper at COLM 2024
C.3
Error Case Study
Instruction: Using the final goal as your guide, refleet on your past successful stritegies smart agent. Observe the information in the image inform your present docision.
two hoxes of tissue from counter toilet. Step Details: Stepe]: Proceed to counter front of t'iisuu boxes. Stepli2:
Question: Your Main Goal:
of
Pick up middle tissue box from counter. Stepf::; Tum around, go to counter toilet.
Current Step:
Ground Truth: Place tissue box buck of toilet, in front of cundle stick. v
GPT-4V: I'm sorry, I cannot provide assistance as requested. X (Refusal to reply)
Owen-VL-Chat: Step complete. X (Failure follow instractions)
LLaVA-L55-B:: The image a 3D computer-generaate image of bathroom. X
(Failure to follow instructions)
Instruction: What's difference between images?
Question: Difference:
Ground Truth: There are 2 people on the left image. V Owea-VL-Chat: There existence any difference images. X (Fatiture to recognize)
Valley: The image shows a building with large number of cars parked in frunt of The are parked in parking lot, and the building
uppears to he university building, The cars are parked in a row, and the image taken from distance, showing the entire parking
Ierge1
(Failure to follow instructions)
Figure 12: Two error cases from Space Understanding (above) and Visual Relation Inference (below) tasks in MILEBENCH.
We conducted an error analysis to further investigate the flaws of the models. An example of the Space Understanding task is displayed in the upper part of Figure 12 When recognizing spatial positions and current actions, GPT-4V declined did not correctly follow the instructions to answer the question. They merely generated captions for the images, which could be related to them not having been trained on multi- image QA data, emphasizing the importance of multi-image training. In a Visual Relation Inference task example (Figure (2 (otttm)) Qwwn-VVI-Cher and Valley struggled with image differentiation and instruction following, resulting in inaccurate inferences. This suggests MLLMs could improve in recognizing subtle image differences, possibly due to their low- resolution visual models. The illusion issue in multi-image inputs for video models also highlights the need for ample multi-image training data.
respond, while other models
C.4 Detailed Experiments Result
To delve deeper into the analysis, we present the performance of all models on both the Multi-image Set and the Combined-image Set. For collections involving multiple images, Table 11 illustrates the performance of all models under the temporal multi-image tasks, while Table demonstrras their performance on semantic multi-image tasks. Regarding the combined-image set, ex the results of all models for Temporal Multi-image Tasks, and Table 14 reveals their results on Semantic Multi-image Tasks.
D More Analysis Results
D.1 Experiment Combined-image Set
To overcome the constraint that models support only minimal number of image inputs, we introduced Combined- -inage Set, and to distinguish it from the original MILEBENCH, we will henceforth refer to MILEBENCH as the Multi-image Set. In the Combined-image Set, multiple images are merged into one large image, positioned at the beginning of the input. The original images in the text are then substituted with placeholders. To save the cost, we only selected three closed-soures MLLMs to evaluate.
We show the result on combined image set in Table and summarize our findings follows: (1) The performance of proprietary models still surpasses that of open-source models in both realistic evaluation (average: 44.8% v.s. 29.6%, max: 48% v.s. 44.9%) and diagnostic
26

---

Published as a conference paper at COLM 2024
Table 9: Experiment Result on Combined-image Set. T-1 refers to the task number intro- duced in Section 3 NH and IR refers to Needle in a Haystack and Image Retrieval. The highest scores for closed-source models, open-source image models, and open-source video models are marked in red, blue, and green respectively.
Model
Size
Temporal Multi-image
Semantic Multi-image
NH
IR
Overall
Overall
T-1
1-2
1-3
T-4
S-1
5-2
S-3
S-4
S-5
N-1
N-2
I-1
Realistic
Diagnosiic
Rendows
25.0
31.9
25.0
31.6
25.1
24.6
0.0
25.3
0.0
0.0
0/0
11.4
22.3
5.5
Closed scurce MLLMs
GPT-VV
40.8
42.6
49.5
20.5
100.0
9.1
38.2
Gemini 1.D
32.3
4588
31.0
51.1
70.1
20.7
16.1
44.8
70.0
71.3
83.8
26.5
48.0
60.5
Claude 3 Opus
32.5
38.8
20.0
50.0
65.3
57.8
14.4
36.5
51.0
100.0
13.5
18.5
40.7
44.0
Open source MLLMS Chaage mosiels)
ALLaVA-LLomer
38
22.7
31.6
30.5
36.6
34.0
21.2
10.5
25.1
29.5
6.9
0.0
10.0
26.9
5.6
YI-VI
68
26.8
34.3
28.5
44.3
57.8
33.5
9.9
301.7
42.5
27.2
0.0
10.5
34.2
12.6
Cheetor
7B
23.5
30.0
19.5
30.5
29.5
26.0
22.8
25.9
31,0
15.6
0.0
10.7
26.0
8.5
Owen-VLLChat
7B
27.5
37.1
23.8
40.5
55.1
44.8
29.9
53.0
50,9
1.9
8.8
35.7
20.6
LLaVA-155-B
7B
30.0
38.1
30.3
41.8
53.8
29.0
15.5
30.4
65.5
18.8
0.0
7.2
37.1
8.6
MiniGPT--2
78
26.5
32.3
25.3
38.3
43.0
25.3
10.6
19.9
44.0
21.6
0.0
11.2
29.5
10.9
VILA
34.5
11.3
LLaVA-116-7
7B
27.3
38 5
27.3
90.8
46.3
33.0
11.8
25.6
56.5
15.9
0.0
10.5
34.1
8.8
Mantis
7B
13.0
65.1
34.0
54.5
68.8
53.2
17.3
272
61.0
85.3
0.0
11.8
41.9
32.4
Open flamingo
90
16.2
22.1
17.3
21.8
25.5
28.5
19.4
0.0
10.8
LLaVA-15-138
138
33.5
44.1
29.5
52.4
64.8
39.0
13.7
36.0
64.0
20.0
0.0
10.0
41.9
10.0
11AVA-16-13B
13B
32.0
44.5
29.5
43.44
55.4
30.3
10.0
27.8
64.0
17.8
0.0
10.3
$7.4
9.4
Open-souncce MLLMs (Vádeo mudels)
2
7B
12.5
21.5
18.5
11.6
15.6
10.3
3.7
4.0
25.9
0.0
10.2
11.4
12.0
Valley
78
21.3
31.9
24.3
26.1
24.0
23.8
13.9
7.8
26.5
10.5
5.3
0.0
22.2
5,3
VidenChat2
14.3
35.5
0.0
92
LLDMA-VID
70
25.8
33.3
25.8
35.66
44.9
24.0
12.0
28.8
38.0
51.3
0.0
10.7
29.8
20.6
LWM
78
1.3
10.4
0.8
3.1
8.4
10.3
9.9
6.8
7.0
43.1
0.0
0.2
6.4
14,4
evaluation (average: 51.2% v.s. 12.3%, max: 60.5% v.s. 32.4%). (2) In comparison to the results on the multi-image set, the performance of proprietary models declined, except for Gemini 1.0, which is limited by the number of images uploaded on the multi-image set. The potential reason is that to maintain performance on the combined image set, models need to possess high-resolution vision models that can effectively distinguish multiple images combined together. For instance, Gemini 1.0 adjusts images of excessively large resolution to size of 3072 3072. On the other hand, GPT-4V and Claude Opus resize the images to dimensions of 768 x 768 and 1568 x 1568, respectively. The lower resolution input of GPT-4V and Claude Opus, in comparison to Gemini 1.0, could potentially be factor for their diminished performance. set, the performance of some open-source models with short contexts improved, such as ALLaVA-Longer (from 24.7% to 26.9%) and MiniGPT-v2 (from 17.8% to 29.5%). The possible reason is that these models have only been trained on single images and cannot effectively generalize to multi-image scenarios. Combining images can effectively alleviate this issue.
(3) Compared to the results on the multi-image
D.2 Inter-task Correlation in MILEBENCH
To investigate the multi-task characteristics of the realistic evaluation in our MILEBENCH, we ana- lyzed the performance of all models across the nine tasks within this evaluation and calculated pair- wise correlations between different tasks, as shown in Figure 13 (1) We found that, aside from Task S-3 (Visual Relation Inference), tasks within the same category (either temporal multi-image or semantic multi-image) exhibited high correlation. Task S-3, being a challenging one, showed little variation in scores across models. (2) We also noted that Task T-3 (Visual Navigation and Spatial Localization) demonstrated lower correlation with other tasks, possibly due to its requirement of unique cognitive skills such as understanding the world from a first- person perspective. These observations suggest that the realistic evaluation of MILEBENCH encompasses broad range of task types, offering a more comprehensive assessment in the context of multi-image long-context scenarios.
4-1
-3
43
74
9/1
9/2
4/3
8/A
9/
E11
99
3:91
0:72
8:
9.
0:81
0:38
77
9:55
E2
0.87
E-3
0.72
8.72
1.00
0.66
0.50
0.56
0.35
0.50
9.64
0.02
1.00
0.50
0.02
0.50
0.50
2.90
0.60
0.30
2.50
0:95
0:24
0.72
0.00
8-2
0.501
2.05
0.56
0.92
0.93
3.00
0:35
0.77
3.58
8-3
0.38
10335
0.45
0.34
0:23
1:00
0.55
8-1
0.50
8.8G
0.72
0.77
0.55
1:50
8-5
0.66
0.05
0.64
0.83
0.90
0:80
0.67
1.00
Figure 13: Spearman correlation be- tween each pair of tasks in realistic evaluation.
27

---

Published as a conference paper at COLM 2024
D.3 Risk of Data Contamination for MILEBENCH
Considering MILEBENCH's use of public datasets, there's potential risk of data contamination, Our investigation, which involved excluding models trained solely on single-image tasks, selected four open- Cheetor, Open Flamingo, and VILA-and one closed-source model, GPT-4o (details in Ap- pendix C.1 We referred to Wei et al. (2023) and constructed an Adversarial (ADV) V) Set with shuffled options and paraphrased ref- erence answers and evaluated the differ- ence between original and ADV results. The results (Table 10) indicate negligible per- formance drop, ranging from 0.1% to 1.2% for the open-source models and 3% for the closed-source model, suggesting minimal likelihood that these models were trained our dataset.
Model
Regular
ADV
A
Cheetor
25.4
25.3
0.1
VILA
44.4
43.6
0.8
Qwen-VI--Chat
39.1
38.2
0.9
source models- -eee--eee-----ie
Open flamingo
27.4
26.2
1.2
GPT-4o
60.3
57.3
3.0
Table 10:
Contamination Detection. We present Regular (result on MILEBENCH), ADV (result on the ADV set) and their difference A.
Table 11: Experiment Result on Temporal Multi-image Tasks. AL: Action Localization. AP: Action P'rediction. AS: Action Sequence. CO: Character Order. Cl: Counterfactual Inference. EN: Egocentric Navigation. MA: Moving Attribute. MD: Moving Direction. OE: Object Existence. OI: Object Interaction. OS: Object Shuffle. ST: Scene Transition. SC: State Change.
Model
AL
AP
AS
CO
EN
MA
MD
OE
01
OS
ST
SC
Random
25.0
25.0
25.0
33.3
34.8
25.0
36.0
25.0
33.3
25.0
33.3
25.0
33.3
Closed source MLLMs
GPT-4V
36.5
64.0
54.5
32.0
16.0
48.5
66.0
35.0
83.0
46.0
GPT-4o
34.5
73.5
77.5
86.5
46.0
44.5
58.0
31.5
52.0
79.5
37.5
86.5
54.0
Gemini 1.0
30.0
48.0
46.5
39.0
41.0
28.5
48.5
26.0
53.0
50.5
32.0
84.5
42.0
Gemini 1.5
40.0
63.5
61.5
75.0
33.5
32.5
60.5
35.5
54.0
68.0
35.5
79.0
41.5
Claude 3 Opus
32.5
41.5
39.0
56.5
30.5
24.5
46.5
17.5
45.5
47.5
29.0
71.5
36.0
Open-source MLLMs (Image models)
ALLaVA-L.onge
19.0
23.5
20.0
25.5
23.5
25.5
29.0
30.5
47.0
21.5
32.0
38.5
37.5
Yi-VL
31.5
22.0
27.0
40.0
34.0
33.5
32.5
29.5
45.5
28.5
33.0
51.0
39.5
Cheetor
25.5
24.5
22.5
40.0
23.0
24.5
30.5
12.0
11.5
21.5
29.5
9.0
40.0
Qwen-VL--Catt
32.5
34.0
37.5
38.0
30.5
22.0
42.5
22.5
41.0
40.5
36.0
71.5
38.5
LLaVA-1.5-7B
24.0
48.5
38.5
26.0
28.5
31.5
47.0
32.0
53.0
47.5
33.5
71.5
30.0
MiniGPT-v2
9.5
6.5
11.5
17.5
39.0
15.0
20.5
17.0
14.5
16.5
6.0
5.5
19.0
VILA
23.0
53.0
45.0
48.5
38.5
35.0
54.0
35.5
54.5
51.5
36.5
78.0
32.0
LLaVA-1.6-7B
25.5
50.0
41.0
23.0
17.0
28.5
41.0
28.5
51.5
46.5
38.5
58.5
33.5
Mantis
25.0
60.5
49.0
41.5
34.0
20.5
55.5
30.0
55.5
57.0
31.5
66.5
53.5
Open flamingo
26.5
25.0
24.0
35.5
38.0
25.0
31.5
27.5
30.0
29.5
39.0
33.0
34.0
LLaVA-1.5-13B
25.5
45.5
42.5
43.0
36.0
26.0
48.5
37.0
46.0
45.0
42.5
70.5
36.0
LLaVA-1.6-13B
27.0
34.0
31.5
29.5
20.0
23.5
49.0
23.0
52.0
50.0
39.0
48.5
34.0
Open Open-source MLLMs (Video models)
Video-LLaMA-2
1.5
0.0
0.0
9.5
8.5
2.0
3.5
0.5
5.5
1.5
5.0
0.0
9.5
Valley
16.0
15.0
20.0
32.5
28.0
12.0
33.5
13.0
39.5
30.0
16.0
17.5
30.0
VideoChat2
6.5
16.0
10.0
16.5
25.0
17.5
44.0
8.0
37.5
11.0
16.0
10.0
12.0
LLaMA-VID
27.5
29.0
22.0
42.0
39.5
25.0
46.0
28.0
30.0
34.0
39.0
49.0
31.5
LWM
0.5
1.0
0.0
12.0
2.5
0.5
13.5
4.5
40.0
5.0
0.0
0.5
10.5
28

---

Published as a conference paper at COLM 2024
Table 12: Experiment Result on Semantic Multi-image Tasks. CED: Conversational Em- bodied Dialogue. VCC1: Visual Change Captioning (CLEVR-Change). DQ: Document QA. VRE: Visual Relationship Expressing. MD: Multimodal Dialogue. CMQ: Complex
Multimodal QA. SU: Space Understanding. OQ: OCR QA. SQ: Slide QA. VCC2: Visual
Change Captioning (Spot-the-Diff) TQ: Textbook QA. WQ: Webpage QA. LTTQ: Long Text
with Images QA.
Model
CED
VCC1
DQ
VRE
MD
CMQ
SU
0Q
SQ
VCC2
TQ
WQ
LTIQ
Random
0.0
0.0
25.0
0.0
0.0
25.3
25.3
23.0
25.7
0.0
25.0
25.3
25.0
Plosed-source MLLMs
CPT-4V
15.4
15.5
92.5
5.1
69.5
80.0
81.0
57.5
82.0
12.7
76.5
79.0
95.5
GPT-4o
24.4
20.3
7.6
70.0
85.0
86.5
18.1
80.0
79.5
96.5
Gemini 1.0
41.1
21.1
75.5
9.6
54.5
76.0
73.0
52.5
66.0
19.9
66.0
67.5
88.0
Gemini 1.5
26.9
21.3
88.5
12.8
62.5
71.5
82.5
52.0
77.5
19.4
74.0
71.5
78.5
Claude 3 Opus
19.6
19.7
81.5
8.7
68.5
56.0
58.5
31.0
66.5
11.7
70.0
61.5
71.5
Open-source MLLMS (Image models)
ALLaVA-Longer
10.4
15.8
28.0
5.7
25,0
27.0
25.5
2.5
31.5
14.8
49.5
25.5
Yi-VL
30.0
15.2
53.5
37.0
48.0
35.5
8.0
13.7
51.5
57.0
72.0
Cheetor
32.1
29.0
18.0
13.9
23.0
32.5
32.0
28.0
28.5
22.7
25.0
29.5
27.0
Qwen- VL-Chat
19.4
15.3
58.0
8.5
34.0
66.0
59.5
42.5
56.5
19.2
53.5
44.0
65.5
LLaVA-1.5-7B
14.9
16.4
1.2
33.5
62.5
10.0
16.5
58.5
31.5
MiniGPT-v2
1.9
5.5
25.5
7.4
32.0
34.0
16.5
21.5
28.5
10.0
21.0
52.0
31.0
VILA
20.0
16.6
44.0
7.1
41.5
72.5
73.0
31.5
48.5
13.6
55.5
65.0
79.5
LIVVA-1667B
11.8
13.7
44.0
5.9
42.0
60.0
69.5
32.0
45.0
10.2
49.0
38.0
59.5
Mantis
26.3
17.6
37.5
78.0
26.0
62.5
Open flamingo
23,9
17.6
26.5
13.3
41.5
29.0
28.5
20.5
24.0
17.8
29.0
29.5
20.0
TiVAA-155-1B
19.0
15.9
46.0
9.7
40.0
74.0
59.0
50.5
51.0
15.7
55.0
65.0
73.5
LLaVA-1.6-3BB
12.7
12.8
41.0
5.7
38.5
69.5
72.5
13.5
47.5
10.1
53.5
45.0
68.5
Open--ourre MLLMs (Video models)
-ideo-LLaMA-2
7.2
5.0
0.0
10.0
4.2
0.5
27.5
Valley
20.7
9.1
19.0
3.7
33.0
24.5
31.0
21.5
15.5
9.9
20.5
21.5
8.4
VideoChat2
13.0
22.6
30.5
7.6
35.0
53.5
31.5
7.0
28.0
12.9
31.0
33.5
50.5
LLD
19.9
14.6
35.0
6.5
36.0
37.5
46.5
14.5
28.5
13.9
47.0
37.0
52,5
LWM
13.3
6.5
3.5
17.5
15.0
17.5
1.5
29

---

Published as a conference paper at COLM 2024
Table 13: Experiment Result on Temporal Multi-image Tasks of Combined-imaae Set. AL: Action Localization. AP: Action Prediction. AS: Action Sequence. CO: Character Order.
CI: Counterfactual Inference. EN: Egocentric Navigation. MA: Moving Attribute. MD:
Moving Direction. OE: Object Existence. Ol: Object Interaction. OS: Object Shuffle. ST:
Scene Transition. SC: State Change.
Model
AP
MD
OE
01
OS
ST
SC
Random
25.0
25.0
25.0
33.3
34.8
25.0
36.0
25.0
33.3
25.0
33.3
25.0
33.3
Closed-soure: MLLMS
GPT-4V
38.0
48.0
36.5
36.0
22.5
58.0
42.0
23.5
75.0
Gemini
27.0
31.5
38.5
45.0
37.5
35.5
26.5
56.5
45.0
36.0
81.0
41.0
Claude Opus
35.5
28.0
34.0
54.5
34.0
24.5
43.5
15.5
44.0
35.0
32.5
73.5
38.0
Open-source MLLMs (Image models)
AlLaee-Longer
22.5
21.5
24.0
34.0
27.5
32.5
28.0
28.5
46.0
23.0
29.5
49.5
35.5
Yi-VL
27.5
25.5
27.5
41.0
36.5
36.5
31.5
20.5
46.5
26.5
32.5
60.5
39.0
Cheetor
31.0
21.0
17.5
27.5
32.5
30.0
23.5
41.0
10.5
26.5
23.5
26.5
12.5
28.5
24.0
30.0
35.5
25.0
27.5
46.0
20.0
47.0
25.5
30.0
65.5
36.0
LLaVA-1.5-7B
20.0
38.0
32,0
33.0
28.5
28.0
48.5
32.5
38.0
37.0
29.0
71.5
34.0
MiniGPT-v2
29.5
25.0
25.0
32.5
38.0
22.0
26.0
28.5
31.5
32.5
39.0
36.5
46.0
VII.A
63.5
28.5
35.5
39.5
37.0
35.0
31.5
46.5
33.0
51.0
29.5
29.5
73.5
LLaVA-1.6-7B
27.0
29.0
26.0
37.0
39.0
27.0
42.5
27.5
40.5
32.0
39.0
50.5
36.5
Mantis
33.0
47.0
49.0
44.5
39.0
37.5
50.0
30.5
50.0
48.5
32.0
85.0
49.5
Open flamingo
15.0
17.5
16.0
34.0
18.0
24.0
15.5
10.5
22.0
20.0
31.0
31.0
4.0
LLaVA-1.5-13B
31.0
33.0
36.5
46.5
38.0
27.0
51.5
32.0
45.5
37.5
42.0
79.5
45.5
LLaVA-1.6-13B
35.5
25.0
35.5
35.0
38.5
33.0
55.5
26.0
52.5
37.0
33.0
58.0
42.0
Open-source MLLMs (Video models)
aide--I.laMA-2
20.5
16.0
1.0
16.0
1.0
14.5
22.5
42.5
13.5
22.0
6.5
23.0
Valley
17.5
25.5
21.0
30.0
29.5
23.5
32.0
25.0
48.5
32.5
14.5
19.0
26.0
VideoCha12
17.5
19.5
29.0
25.0
18.5
47.5
26.0
28.5
24.5
LLaMA-VID
29.5
27.0
21.0
39.5
36.0
23.5
33.0
28.0
30.0
31.0
39.0
31.5
35.5
LWM
0.5
0.5
3.0
8.5
0.0
1.0
9.5
0.5
5.0
2.5
24.5
1.0
3.0
30

---

Published as a conference paper at COLM 2024
Table 14: Experiment Result on Semantic Multi-image Tasks of Combined-imaae Set.
CED: Conversational Embodied Dialogue. VCC1: Visual Change Captioning (CLEVR- Change). DQ: Document QA. VRE: Visual Relationship Expressing. MD: Multimodal
Dialogue. CMQ: Complex Multimodal QA. SU: Space Understanding. OQ: OCR QA.
SQ: Slide QA. VCC2: Visual Change Captioning (Spot-the-Diff), TQ: Textbook QA. WQ:
Webpage QA. LTIQ: Long Text with Images QA.
Model
CED
VCC1
DQ
VRE
MD
CMQ
SU
0Q
5Q
VCC2
TQ
WQ
LTIQ
Randon
0.0
0.0
0.0
25.3
23.0
25.7
0.0
25.0
25.3
25.0
Closeul source
GPT-4V
11.6
17.4
60.0
5.1
58.5
70.0
70.5
56.0
66.5
14.1
68.0
72.5
Gemini 1.0
41.5
20.3
88.0
10.3
48.0
60.0
70.0
56.0
68.0
17.5
61.5
68.0
91.0
Claude 3 Opus
18.0
18.4
69.0
9.1
55.0
57.5
51.0
36.5
68.0
15.7
66.5
62.5
74.5
Open source MLLMs (Image models)
ALIaVA-LLonge
11.7
15.5
30.0
4.7
38.5
30.0
29.5
3.0
30.5
11.2
41.5
52.0
12.5
Yi-VL
22.8
12.0
5.4
38.5
49.0
42.5
7.0
42.0
12.3
54.5
57.5
70.0
Cheetor
31.7
29.8
23.5
15.5
22.0
32.5
31.0
25.0
29.5
23.2
34.5
26.5
24.5
Ow-n-VL-Chat
20.4
11.1
38.5
6.6
39.5
60.5
53.0
44.0
52.0
11.0
49.5
49.5
61.0
LLaVA-115-7B
19.8
16.7
39.0
8.6
41.0
62.5
65.5
4.5
43.5
21.2
40.5
59.5
52.5
MiniGPT-v2
5.7
9.6
10.6
34.0
440
18.0
11.6
38.0
48.5
43.0
VILA
20.0
16.9
40.0
7.5
39.5
64.5
63.5
21.5
44.0
16.1
48.5
69.0
65.5
LLaVA-116-BB
13.8
16.8
35.0
5.5
37.5
52.0
56.5
21.5
42.5
13.2
48.5
37.5
47.0
Mantis
14.9
19.5
46.0
8.6
39.5
71.0
61.0
62.5
51.0
23.7
54.0
62.0
86.0
Open flamingo
12.1
1.4
38.5
28.5
21.0
2.2
25.0
LLaVA-1.5-13B
30.6
15.7
42.5
5.9
41.5
70.5
64.0
27.0
47.5
19.7
51.5
68.5
68.5
UTVAA-16--3B
13.5
15.0
44.5
4,9
42.0
61.0
64.0
3.0
43.5
10.1
54.0
55.0
51.5
Open-source MLLMS (Video models)
Video-LLaMA-2
5.4
7.5
2.5
5.5
2.0
13.5
4.0
15.0
13.5
1.6
9.5
11.5
28.0
Valley
9.3
9.8
4.3
26.5
34.5
27.5
25.0
VideoChat2
19.4
18.4
29.0
33.0
51.5
25.0
4.0
26.0
13.0
36.0
45.0
44.0
LLaMA-VID
19.2
16.6
31.0
5,9
38.5
39.5
38.0
12.5
28.5
13.5
44.5
35.5
60.0
LWM
8.1
12.8
10.0
6.4
5.5
11.5
7.0
8.5
12.5
10.5
13.5
2.5
6.0
31


Reviews:
[{"review_id": "F5M9R5uoka", "reviewer": "Reviewer_oeBf", "summary": "This paper presents a new benchmark, with a focus on multi-image scenarios. The benchmark mainly evaluates the long-context performance on the M-LLMs. \n\nThe paper provides a comprehensive evaluation of several existing models, and shows that long-context capability is demanding for improvements.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's positive comments on the comprehensiveness of our taxonomy and the detailed error analysis. In response to the concerns raised, we provide the following clarifications:\n\n> Q1: Dataset Composition and Metric Selection\n\nWe acknowledge the reviewer's concern regarding the dataset composition. Currently, most of the data in MileBench is sourced from **existing public datasets' test sets**, with only a few examples being manually constructed. In the manually constructed data portion, we have demonstrated the potential of using such data as test cases. We aim to include a broader range of data sources and employ more rigorous methods to expand our multi-modal long-context benchmark in the next version.\n\n> Q2: The Use of ROUGE-L and Accuracy Metrics\n\nThe reason we use ROUGE-L and accuracy as metrics is that we are **following the standards set by the datasets we collected**. These metrics are widely used and accepted for the task types addressed in our benchmark [1,2]. Our current tasks do not yet consider long-context outputs, but we will consider incorporating such tasks in the next version.\n\n> Reference:\n>\n> 1.Kuratov Y, Bulatov A, Anokhin P, et al. In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs Miss[J]. arXiv preprint arXiv:2402.10790, 2024.\n>\n> 2.Reid M, Savinov N, Teplyashin D, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context[J]. arXiv preprint arXiv:2403.05530, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents a comprehensive performance benchmarking. \n\n- Compared to prior studies, the proposed MileBench benchmark can be suitable for evaluating multimodal long-context abilities of MLLMs.\n\n- The evaluation taxonomy is clearly defined to evaluate two major categories: realistic and diagnostic evaluation. Each category includes a diverse set of evaluation tasks for evaluating the performance of multi-image scenarios.\n\n- The paper offers error analysis to help understand the challenging cases in the proposed dataset.", "reasons_to_reject": "- The testing samples of the proposed dataset are mostly manually borrowed from existing datasets. There is no reliable pipeline to further collect and scale up. \n\n- It remains difficult to automatically evaluate these models which have free-form text outputs. The experiments only consider ROUGE-L and retrieval accuracy as the main metric. This may limit the long-context output scenarios.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S5kYtUy6nr", "reviewer": "Reviewer_DvvC", "summary": "The paper introduces MILEBENCH, a benchmark designed to assess the capabilities of Multimodal Large Language Models (MLLMs) in handling long-context scenarios involving multiple images and texts. This benchmark is motivated by the limitations of existing benchmarks that predominantly focus on short texts and single images, which do not adequately represent real-world multimodal tasks that require handling complex, lengthy contexts with multiple images. MILEBENCH includes two types of evaluations: diagnostic and realistic, aiming to thoroughly assess the performance of MLLMs in complex scenarios that mimic real-world applications. The benchmark tests 19 different models, highlighting that while the proprietary model GPT-4(Vision) excels, most open-source models struggle with these tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of the significance of our work and the comprehensiveness of our taxonomy. In response to the concerns raised, we provide the following clarifications:\n\n> Q1: Potential Data Leakage\n\nWe have discussed the potential risk of data leakage in Section 4.3.3 and have confirmed that MileBench currently has **minimal risk** of such leakage. The data we extracted from other datasets is solely from their test sets.\n\n> Q2: Potential Biases in Data Sources\n\nThe data sources for MileBench are either existing datasets or publicly available wiki documents. For the former, these datasets have undergone peer review and are widely used, ensuring that any bias issues are addressed by their original creators. For the data generated using wiki documents, while we have implemented measures to ensure the suitability of the content, we acknowledge the potential existence of problematic content. We are committed to continuously improving our dataset to mitigate these issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The introduction of MILEBENCH addresses a significant gap in the evaluation of MLLMs by focusing on long-context multimodal scenarios, which are crucial for real-world applications but often overlooked in other benchmarks.\n2. The paper does a good job of designing a variety of tasks that test different aspects of MLLMs, such as temporal and semantic understanding, across both realistic and diagnostic settings.", "reasons_to_reject": "1. Majority of the curated benchmark comes from existing dataset, which may have data leakage and lead to unfairness and bias. Moreover, which part of the existing dataset did you extract the data from, training set, validation set, or test set? \n2. While MILEBENCH incorporates a variety of tasks and sources, there is little discussion on the potential biases inherent in these sources or how they might affect the benchmark's generalizability.", "questions_to_authors": "Please check the \"Reasons To Reject\"", "extra_scores": {"ethics_concerns_details": "This paper release new datasets."}}, {"review_id": "tkQcZOtlCK", "reviewer": "Reviewer_FSLA", "summary": "This paper focuses on evaluating the long-context understanding capabilities of multimodal large language models. The authors introduce MILEBENCH, a comprehensive benchmark encompassing multiple dimensions of multimodal long-context understanding, such as Temporal Multi-image Understanding, Semantic Multi-image Understanding, Needle in a Haystack, and Image Retrieval. Additionally, the paper provides a brief evaluation of various multimodal models, ranging from closed-source to open-source, trained on images and videos, presenting a comparative analysis of their performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of the significance and concise experiments of our work. In response to the concerns raised, we provide the following clarifications:\n\n> Q1: Justification for the Proposed Benchmark\n\nOur dataset differs from video benchmarks in that our benchmark is **more comprehensive** and **focuses on the overall multi-image and long-context understanding capabilities** of MLLMs, while videos are just one form and source of data.\n\n> Q2: Inclusion of Recent Models in Evaluation\n\nThank you for your suggestions:\n1. We have evaluated the models mPLUG-Owl2, MMICL, and Mantis (MM1 has not been open-sourced). The evaluation results are as follows:\n\n| Model         | Size | T-1 | T-2 | T-3 | T-4 | S-1 | S-2 | S-3 | S-4 | S-5 | N-1 | N-2 | I-1 | Overall Real. | Overall Diag. |\n|---------------|------|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|---------------|---------------|\n| mPLUG-Owl2    | 7B   | 49.0| 45.3| 23.8| 46.6| 57.5| 57.3| 16.3| 25.9| 64.5| 18.4| 9.4 | 10.7| 42.9          | 12.8          |\n| MMICL         | 12B  | 59.0| 48.1| 24.5| 49.9| 66.6| 37.0| 11.9| 34.1| 69.0| 50.3| 0.0 | 37.2| 44.5          | 29.2          |\n| Mantis        | 8B   | 54.8| 49.9| 25.3| 48.9| 65.8| 55.5| 17.4| 31.9| 78.0| 27.5| 0.0 | 32.2| 47.5          | 19.9          |\n\n2. Due to the variety of strategies used by different models, we **cannot** conclusively determine that a single strategy is beneficial for multi-image understanding. However, we can still make some general observations based on models with similar strategies and append to the final version, such as:\n    1. **Training with multi-image data can improve performance**: Qwen-VL-Chat, VILA, Mantis.\n    2. **LLMs with long-context capabilities aid in multimodal long-context understanding**: Qwen-VL-Chat, Mantis.\n\n> Q3: Fundamental Abilities for Tackling Multimodal Long-Context Problems\n\nMantis is an LLaMA3-based model trained on a large multi-image dataset. Our analysis of Mantis' results suggests that **training with multi-image samples can enhance a model's understanding of multimodal long-context patterns**. However, **this alone is not sufficient to achieve promising performance**, especially in the \"Needle in a Haystack\" tests where visual perception is also critical. In the final version, we will include more model results and detailed analysis and discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Long-context capabilities, especially in the domain of multi-image, long-context understanding, are crucial functionalities of multimodal large language models, yet they are absent in many current models. A comprehensive benchmark can facilitate the development of these capabilities.\n2. This paper provides an interesting and significant insight: numerous models that excel in common benchmarks show weak performance in multi-images scenario.\n3. This paper is well organized, offering concise experiments across a range of popular multimodal large language models.", "reasons_to_reject": "1. This paper collects data from publicly available datasets/benchmarks, yielding only 200 samples for each task. However, some of the pre-existing datasets used herein (e.g., START [1]) encompass a vast collection of video clips and annotations potentially suitable for directly assessing multi-image capabilities. It's not clear whether these datasets alone are inadequate. The authors can provide substantial evidence to justify the necessity of the proposed benchmarks.\n2. Recent developments have seen certain models displaying capabilities in multiple images understanding (e.g., MM1 [2], mPLUG-Owl2 [3], MMICL [4]). These models should be included in the evaluation to provide a comprehensive understanding of their capabilities and how training strategy/training data/model designing affect the multimodal long-context understanding capabilities.\n3. The experiment results indicate that some models struggle to follow instructions and perform poorly on several samples. It begs the question of whether these models inherently lack the multimodal long-context understanding or if their performance is hindered by unfamiliar patterns that might be improved with training on a few of data samples.  In essence, the authors are encouraged to delve deeper into discussing which fundamental abilities, which preventing the models from effectively tackling multimodal long-context problems, are absent.\n\n[1] Wu, B., Yu, S., Chen, Z., Tenenbaum, J. B., & Gan, C. (2021, August). Star: A benchmark for situated reasoning in real-world videos. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track (Round 2).\n[2] McKinzie, B., Gan, Z., Fauconnier, J. P., Dodge, S., Zhang, B., Dufter, P., ... & Yang, Y. (2024). Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611.\n[3] Ye, Q., Xu, H., Ye, J., Yan, M., Liu, H., Qian, Q., ... & Zhou, J. (2023). mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04257.\n[4] Zhao, H., Cai, Z., Si, S., Ma, X., An, K., Chen, L., ... & Chang, B. (2023). Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv preprint arXiv:2309.07915.", "questions_to_authors": "Refer to the Reasons To Reject.", "extra_scores": {}}, {"review_id": "QF4vml5P2c", "reviewer": "Reviewer_u8ee", "summary": "The paper presents a new benchmark dataset for multimodal (image+text) LLMs, called MileBench.\n\nThe novelty of this benchmark is in its focus - testing multimodal models in long contexts with many images. The collected dataset is a subset of existing datasets and synthetically generated examples. The examples from the existing datasets were picked in a way which agrees with the benchmark's focus. The resulting dataset consists of around 6k examples categorized in 12 different tasks.\n\nThe authors also measured the performance of various models on this benchmark and report the results. Notably, the closed-source models emerged as clear winners on most of the tasks.\n\nOverall, the paper is well written and presents a relevant and potentially useful contribution.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of the benchmark’s usefulness and the thoroughness of our experiments. In response to the concerns raised, we provide the following clarifications:\n\n> Q1: Clarification on Data Sources\n\nThank you for your query regarding the datasets used for data collection. We have listed the datasets in Table 5 of Appendix B.1. We agree that it is important to emphasize and highlight the data sources in the final version of the paper for better clarity.\n\n> Q2: Choice of Metrics\n\nRegarding the choice of metrics, we have adhered to the metrics used by the original datasets. For the datasets we constructed ourselves, we employed standard metrics for multiple-choice questions and extractive question types. Due to the high cost of using LLM-based metrics, the instability of the evaluator model versions, and the self-enhancement bias present in the evaluation models, we did not consider using LLM-based metrics for evaluation in the current version. Of course, in future versions, with the addition of new tasks, LLM-based metrics will be an effective supplement to the existing metrics, and we will consider adding LLM-based metrics as appropriate.\n\n> Q3: Evaluation of Closed-Source Models\n\nDue to cost considerations, we did not initially evaluate the closed-source models. However, based on your suggestion, we agree that it is necessary to include these evaluations. We conduct the necessary experiment of GPT-4o and include the result:\n\n| Model         | Regular | ADV   | $\\Delta \\downarrow$ |\n|-------------------|---------|-------|----------------------|\n| GPT-4o            | 60.3    | 57.3  | 3.0                  |\n\nThe results show that GPT-4o has certain data leakage issues, with an overall score drop of about 3 points. However, overall, the data leakage situation is not serious, and the score drop caused by changing the options did not significantly affect the ranking order of the model.\n\n> Q4: Clarity in Table 2\n\nWe will modify Table 2 to clearly indicate which numbers represent accuracy and which represent ROUGE-L scores for better clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* A new benchmark dataset which might be useful for multimodal LLM research.\n* Includes baseline scores for many publicly available models, making comparison using this benchmark more accessible.", "reasons_to_reject": "* It is not very clear which datasets have been used for the data collection. References are given to the tasks, so I am assuming the data comes from there, but this should be listed explicitly, perhaps in the appendix.\n* Inadequate attention given to possible data contamination in discussion about the results (see my question below).", "questions_to_authors": "* Have you tried using different metrics than ROUGE-L? I think for open ended generation tasks it might make more sense to use a LLM-based metric instead. I realize that selecting a good metric is secondary to this paper's focus, but by introducing a benchmark like this, you basically dictate which metric to use if someone else decides to use the benchmark to compare with other models.\n\n* Given that the collected benchmark is a subset of existing datasets, is it possible that the success of the closed-source models is determined by a data leakage? (https://aclanthology.org/2024.eacl-long.5/) Why are these models omitted in the analysis in section 4.3.3?\n\nSome minor suggestions:\nFor better clarity, Table 2 should indicate which numbers are accuracy and which are Rouge-L scores.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a new benchmark for testing the long-context capability of MLLMs, which involves inputs with a large number of image and text tokens. They provide extensive evaluation results of frontier MLLMs. While similar benchmarks exist for LLMs (like "needle in a haystack"), this is the first benchmark to test long-context capabilities in MLLMs. It includes both artificial and realistic scenarios.

Some reviewers raised concerns, such as:

- Some MLLMs specialized in long-context processing were missing in the evaluation.
- Missing discussion on whether the current low performance is a fundamental issue.
- Missing evaluation of closed LLMs and potential data contamination.

However, the authors provided satisfactory answers during the rebuttal.
Overall, all the reviewers were eventually satisfied, and we recommend acceptance.
Decision: Accept