MileBench: Benchmarking MLLMs in Long Context

Authors: dingjie  song
shunian  chen
guiming hardy chen
fei  yu
xiang  wan
benyou  wang
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 15:56:41

Abstract: Despite the rapid progression of Multimodal Large Language Models (MLLMs) and their impressive performance on various benchmarks, the applicability of these results to real-world tasks remains uncertain. 
This ambiguity primarily stems from the benchmarks' limited consideration for long-context and multi-image tasks, which are critical elements in real-world applications.
Existing benchmarks often focus on single-image and short-text samples, and when assessing multi-image tasks, they either limit the image count or focus on time-series captioning tasks, potentially masking MLLMs' performance challenges such as hallucination in long-context situations. 
To address these limitations, we introduce \textbf{\dataset}, a pioneering benchmark designed to rigorously test the \textbf{M}ult\textbf{I}modal \textbf{L}ong-cont\textbf{E}xt capabilities of MLLMs. 
This benchmark comprises a mix of text and images, long contexts, multiple tasks, and tasks requiring both comprehension and generation. 
We establish two distinct evaluation sets, diagnostic and realistic, to systematically assess MLLMs' long-context adaptation capacity and their ability to complete tasks in long-context scenarios.
Our experimental results, garnered from testing 19 models, revealed that while closed-source model GPT-4(Vision) outperforms others, most open-source MLLMs display inadequate performance in long-context situations. 
Hence, we strongly encourage an intensification of research efforts towards enhancing MLLMs' long-context capabilities, especially in scenarios involving multiple images.

Main paper: 
 introduction: 
 The recent swift development of Multimodal Large Language Models (MLLMs) (OpenAI, 2023; Anil et al., 2023; Liu et al., 2023a; Awadalla et al., 2023) has displayed outstanding performance across a diverse range of multimodal tasks (Yang et al., 2023; Wu et al., 2023) . Meanwhile, a surge of benchmarks for evaluating MLLM performance has emerged (Liu et al., 2023a; Fu et al., 2023; Ge et al., 2023; Li et al., 2023a) , offering insights into their general capabilities (Li et al., 2023b; Liu et al., 2023c; Yu et al., 2023) and task-specific capabilities (Liu et al., 2023d; Yue et al., 2023; Wang et al., 2024) .However, a critical aspect is often overlooked. Real-world applications frequently demand the processing of long contexts and multi-image tasks that include multi-round dialogues based on multiple images (Li et al., 2022) , action prediction tasks (Wu et al., 2021) , navigation tasks in 3D space (Krantz et al., 2020) , and understanding tasks with lengthy documents interspersed with images on Wiki pages (Hannan et al., 2020) .Despite this, existing benchmarks primarily focus on single-image and short-text samples (Liu et al., 2023a; Fu et al., 2023; Liu et al., 2023c; Li et al., 2023b) , thereby failing to fully capture the complexity and diversity of real-world scenarios. While some benchmarks evaluate multi-image tasks, they either have limited number of images provided per sample (e.g., SEED-Bench-2 (Li et al., 2023a) , DEMON (Li et al., 2023c) ) or only include time-series captioning tasks (e.g., Mementos (Wang et al., 2024) ), as shown in Figure 2 . In addition, this omission could potentially neglect the hallucination issue that MLLMs might exhibit in long-context situations (Huang et al., 2023) . Given the aforementioned shortcomings with existing benchmarks, we identify a pressing need for a more holistic evaluation that fully encapsulates the long-context and multi-image task demands prevalent in real-world applications.Addressing this need, we introduce MILEBENCH, the first benchmark specifically designed to test the MultImodal Long-contExt capabilities of MLLMs1 . To systematically assess the capabilities of MLLM in multimodal long contexts, our benchmark consists of two distinct evaluation sets, diagnostic evaluation and realistic evaluation. The former explores the long-context recall abilities of MLLMs, using needle-in-a-haystack and image retrieval tasks, while the latter stress-tests the model in a manner akin to real-world conditions using both temporal multi-image tasks and semantic multi-image tasks. To construct our evaluation sets, we gather 6,440 multimodal long-context samples from 21 pre-existing or self-constructed datasets, with an average of 15.2 images and 422.3 words each, as depicted in Figure 2 , and we categorize them into their respective subsets.After evaluating 22 models, the closed-source GPT-4o2 excelled in both diagnostic and realistic evaluations, achieving impressive scores of 99.4% and 60.3%, although it still falls short of a perfect 100% score. On the contrary, most open-source MLLMs struggled with long-context tasks as depicted in Figure 1 . Only Mantis and Qwen-VL-7B managed average scores of 47.5% and 37.2% in realistic and diagnostic evaluations respectively. These results underscore that there are "miles to go" towards fully-realized long-context MLLMs, prompting a call for increased research focus on such tasks, especially those involving numerous images. 
multi-image and long-context mllms: 
 Beyond training on single-image-text pairs, recent developments in MLLMs are also oriented towards handling multiple and interleaved image-text sequences (Awadalla et al., 2023; Li et al., 2023c) . However, these models have relatively limited contexts (i.e., up to 4K) compared to leading proprietary MLLMs such as GPT-4V (OpenAI, 2023) and Gemini (Anil et al., 2023; Reid et al., 2024) , which exhibit capabilities for long-context processing, supporting up to 128K and 10M tokens, respectively. However, there remains a notable gap in open-source MLLMs capable of long-context comprehension. Currently, the only opensource models equipped for long contexts are those designed for video, which are trained to process multiple frames, inherently managing multiple images and long contexts (Liu et al., 2024a; Zhang et al., 2023; Luo et al., 2023; Li et al., 2023d; e) . In this paper, we release an evaluation set specifically designed for multi-image and long-context MLLMs.Most of the MLLM benchmarks only evaluate multimodal tasks with a single image (Liu et al., 2023a; Fu et al., 2023; Liu et al., 2023c; Li et al., 2023b; Yu et al., 2023; Ge et al., 2023; Yue et al., 2023) . As a complementarity, SEED-Bench2 (Li et al., 2023a) and DEMON (Li et al., 2023c) test multimodal capabilities with multiple images but limit the evaluation to around three images, which is inadequate for a thorough multi-image comprehension assessment. Mementos (Wang et al., 2024) involves data samples with up to approximately 11 images, mainly focusing on temporal understanding and limited context scenarios. This focus overlooks the wide range of scenarios involving multiple images and long context. We provide an overview of existing MLLM benchmarks in Appendix A. To the best of our knowledge, MILEBENCH is the first comprehensive benchmark that evaluates MLLMs across both multi-image and long-context dimensions, catering to a broader spectrum of general scenarios. 
evaluation taxonomy: 
 MILEBENCH consists of two major components: Realistic Evaluation and Diagnostic Evaluation, as depicted in Figure 3 . Realistic Evaluation requires MLLMs to address tasks within multimodal long-context scenarios, emphasizing the models' proficiency in comprehending and reasoning across extended multimodal contexts. Conversely, Diagnostic Evaluation demands MLLMs to retrieve information from the provided context, highlighting the model's capability of long-range information retrieval and the elimination of distractors. The detailed taxonomy of MILEBENCH is illustrated in Table 4. 3 The realistic evaluation is designed to assess an MLLM's ability to comprehend, integrate, and infer information in a multimodal long context. We categorize the tasks into two main groups: Temporal Multi-Image tasks and Semantic Multi-Image tasks. Temporal T-14 Action Understanding and Prediction task involves interpreting and forecasting actions of objects or characters in sequential scenarios based on a series of images. This task is divided into three subtasks. Action Localization (Gao et al., 2017) assesses the model's ability to identify time actions within a sequence. Action Prediction (Wu et al., 2021) tests the model's capacity to predict a character's actions. Action Sequence (Wu et al., 2021) evaluates the model's understanding of the chronological order of a character's actions.T-2 Object and Scene Understanding task involves identifying and understanding objects within a sequence. It comprises four subtasks: Object Existence (Yi et al., 2020) tests the model's ability to detect and track an object's movement. Moving Attribute (Yi et al., 2020) assesses the model's understanding of moving object attributes. Object Interaction (STAR) evaluates the model's comprehension of inter-actions between people and objects in complex scenarios. Object Shuffle (Patraucean et al., 2023) gauges the model's ability to locate hidden objects amidst disturbances.T-3 Visual Navigation and Spatial Localization task tests the model's understanding of spatial and directional concepts through two subtasks. One is Egocentric Navigation (Krantz et al., 2020) , involves the model interpreting motion-related instructions and image sequences from a robot's perspective to predict the next action. Moving Direction (Yi et al., 2020) , assesses the model's ability to determine object movement direction, evaluating its understanding of spatial orientation. S-4 Dialogue task primarily involve the fusion of visual data and natural language dialogue understanding. This task needs the model to understand and process multimodal data, including both textual and visual information, while demonstrating consistency and complementarity in the task. The datasets involve Multimodal Dialogue (Li et al., 2022) for multimodal conversational question answering and Conversational Embodied Dialogue (Shridhar et al., 2020) for mapping from language commands and visuals to action sequences.S-5 Space Understanding task requires the model to perceive the spatial environment using multi-image information. It uses the nuscenes (Caesar et al., 2020) dataset, specifically designed for self-driving car technology. It contains sensor data for object detection and tracking, with 1000 annotated scenes for location and properties of objects. The diagnostic evaluation focuses on the MLLMs' capability to retrieve information without being distracted in a multimodal long context. We transform the tasks of "Needle in a Haystack" from NLP and "Image Retrieval" from CV into a multimodal format for assessment. This transition preserves the core of the conventional tasks while offering a more challenging and realistic measure of MLLMs' performance.Needle in a Haystack. The "Needle in a Haystack" task requires the model to find a preset password from a long context. This is widely used in diagnostic evaluations of long-context language models (Kuratov et al., 2024) . In this study, we reintroduce this novel task from a multimodal perspective to evaluate the context perceptual and retrieval abilities of MLLMs. Image Retrieval. Image Retrieval task (I-1) (Schall et al., 2022) requires the model to retrieve images from a set of candidates images given an anchor image (an example is shown in the lower right corner of Figure 3 ). In addition to perceptual and retrieval abilities, this task also necessitates that the MLLM possesses robust abilities in both object and conceptual recognition. We consider this a traditional computer vision task and see image retrieval as a "Needle in a Haystack" task with image modality queries.We have established a robust data collection process and meticulous review procedures to maintain the integrity and quality of our datasets.Data Collection. We collected the samples from two sources: (1) For most of the tasks, we selected and sampled 200 instances from the test sets of the pre-existing datasets for each task, giving priority to multi-image samples. For video data, we used the Katna (Keplerlab, 2021) to convert the video into a multi-image format by extracting one frame per second. The choice of these pre-existing open-source datasets was driven by their well-established reputation and the fact that they have been published in top-tier conferences and journals, ensuring their credibility and reliability. Please refer to Section 5 and Appendix D.3 for information on the data's licensing and data contamination issue. ( 2 1 . The task distribution is demonstrated in Figure 4 . A comprehensive breakdown of the datasets, tasks, and taxonomy can be found in Appendix B.1.For the open-source dataset comprised of the benchmark, we sample 10% of the data for manual verification. Our review team, composed entirely of authors, was assigned to scrutinize the precision of the sampled data, resulting in an Inter-annotator Agreement (IAA)5 of 95%, indicating a high level of consistency among reviewers. For the datasets we formulated independently, equivalent manual verification was carried out on the entirety of the dataset, yielding a similar IAA of 98%, thus ascertaining the data quality. Additionally, the error rate was found to be less than 1% for both datasets, affirming that these datasets maintain an exceptionally high quality and are virtually devoid of errors. 
experiment setup: 
 Evaluation Models. In this study, we conducted an evaluation of several models across three distinct categories that may handle multimodal long contexts, including five closedsource models (GPT-4V (OpenAI, 2023), GPT-4o, Gemini 1.0 (Anil et al., 2023) , Gemini 1.5 (Reid et al., 2024) , Claude 3 Opus 6), twelve open-source image models (Qwen-VL-Chat (Bai et al., 2023) , MiniGPT-v2 (Chen et al., 2023) , Cheetor (Li et al., 2023c) , Open flamingo (Awadalla et al., 2023) , LLaVA-1.5-7B/13B (Liu et al., 2023a) , LLaVA-1.6-7B/13B (Liu et al., 2024b) , ALLaVA-Longer (Chen et al., 2024) , Yi-VL (Young et al., 2024) , VILA (Lin et al., 2023) , Mantis (Jiang et al., 2024)) , and five open-source video models (Video-LLaMA-2 (Zhang et al., 2023) , Valley (Luo et al., 2023) , VideoChat2 (Li et al., 2023d) , LLaMA-VID (Li et al., 2023e) , LWM (Liu et al., 2024a) ). The details of the models and and their version information are in Appendix C.1. All models used greedy decoding to generate answers, with a designated generation length between 1 and 512. We conducted all experiments on NVIDIA A100 GPUs.Prompts and Metrics. To save costs, all evaluations were performed only in a zero-shot setting. The format of the prompts is detailed in the Appendix C.2. When the input length exceeds the maximum context length of the model, we keep the instruction, and truncate the interleaved image-text question from left so as to keep the question of a sample, as instruction and question are critical information and the importance of the last image is higher in many tasks, e.g. multimodal dialogue. Metrics for each dataset, as shown in Table 4 , are consistent with the original work for tasks built on previous datasets. For open-ended generation tasks, the popular n-gram-based metric ROUGE-L is adopted, and accuracy is the metric for multiple-choice and needle-in-a-haystack tasks.We present the results of our experiments in (5) Interestingly, the majority of open-source models scored zero in the Image Needle in a Haystack task. Upon inspection, we found that many of these models partially answered the needle numeric string without completely getting it right.This suggests that open-source models need to improve their ability to retrieve information from images, particularly their OCR capabilities. Detailed results from the realistic evaluation can be found in Appendix C.4. We also selected individual examples for error analysis, details of which can be found in Appendix C.3.In this section, we delve into a meticulous analysis of the results, focusing on two research questions that revolve around the MILEBENCH: "How do MLLMs perform given contexts with different lengths?" and "Do MLLMs also get Lost in the Middle in multimodal long contexts?" We also analyzed three important questions: "Does combined image help multi-image comprehension?", "How diverse and comprehensive is MILEBENCH?" and "Does data contamination issue exist in MILEBENCH?" Detailed analysis results are provided in Appendix D.1, Appendix D.2, and Appendix D.3, respectively.To investigate the performance of MLLMs with varying numbers of images, we divide our dataset into three levels: Few, Medium, and Many, based on the number of images per sample. The specific quantities for each level can be found in Table 1 . Figure 5 reports the average performance of the model on the three types of data with different numbers of images. It can be observed that as the number of images increases, the performance of most models significantly declines (as indicated by a steep slope in the curve), especially for the LLaVA-1.5 series models. This is likely because most models have only been trained on single image, resulting in insufficient generalization for multi-image test data. However, the performance of GPT-4V, GPT-4o, Gemini 1.5, Claude 3 Opus and Qwen-VL-Chat on the 0 5 0 0 1 0 0 0 1 5 0 0 2 0 0 0 2 5 0 0 3 0 0 0 3 5 0 0 4 0 0 0 4 5 0 0 5 0 0 0 5 5 0 0 6 0 0 0 6 5 0 0 7 0 0 0 7 5 0 0 8 0 0 0 8 5 0 0 9 0 0 0 9 5 0 0 1 0 0 0 0 1 0 5 0 0 1 1 0 0 0 1 1 5 0 0 1 2 0 0 0 1 2 5 0 0 1 3 0 0 0 1 3 5 0 0 1 4 0 0 0 1 4 5 0 0 1 5 0 0 0 1 5 5 0 0Token Limit 0 5 0 0 1 0 0 0 1 5 0 0 2 0 0 0 2 5 0 0 3 0 0 0 3 5 0 0 4 0 0 0 4 5 0 0 5 0 0 0 5 5 0 0 6 0 0 0 6 5 0 0 7 0 0 0 7 5 0 0 8 0 0 0 8 5 0 0 9 0 0 0 9 5 0 0 1 0 0 0 0 1 0 5 0 0 1 1 0 0 0 1 1 5 0 0 1 2 0 0 0 1 2 5 0 0 1 3 0 0 0 1 3 5 0 0 1 4 0 0 0 1 4 5 0 0 1 5 0 0 0 1 5 5 0 0Token Limit Liu et al. (2023b) pointed out that in needle-in-a-haystack tasks involving long texts, models may experience the "Lost in the Middle" phenomenon, where they struggle to find the needle located in the middle of the context. We investigated whether MLLMs would exhibit the "Lost in the Middle" phenomenon in multimodal contexts. We chose the two bestperforming models from closed-source and open-source models in the Needle in a Haystack task for analysis. As can be seen from the results in Figure 6 , MLLMs displayed varying behaviors. In multimodal long contexts, GPT-4V did not "get lost in the middle" and managed to complete the two tasks impressively with the scores 99.7% (N-1) and 99.1% (N-2). On the other hand, ignoring the scenarios where the data exceeds its maximum context length (8192 tokens or 32 images) and gets truncated, Qwen-VL-Chat showed a certain degree of "lost in the middle", particularly evident in the image needle task. This indicates that the "lost in the middle" phenomenon also exists in multimodal scenarios. However, a strong ability to manage long context can significantly reduce this risk. 
conclusion and future directions: 
 In this study, we introduced MILEBENCH, a pioneering benchmark designed to rigorously evaluate the multimodal long-context capabilities of MLLMs. We have established the diagnostic and realistic evaluation sets, designed to systematically assess the MLLMs' capacity for long-context adaptation and proficiency in task completion within these contexts. Despite some impressive performances, our experimental results underscore the urgent need for more focused research to enhance MLLMs' capabilities in these complex scenarios.Moving forward, we suggest two primary research directions: (1) Long-context MLLM:Given the ubiquity of mixed media content, there is a pressing need for models that can adeptly process multiple images in long-context scenarios.(2) Scaling MILEBENCH to Larger Contexts and Other Modalities: As real-world tasks continue to evolve, benchmarks should also adapt, incorporating larger contexts, complex task structures, and additional modalities to stimulate the development of more versatile MLLMs. These efforts will help equip MLLMs better for our increasingly multimodal world. 
 
 Appendix: The limitations of this study include that the results from closed-source MLLMs may vary over time, and there is a risk that some of the test data may be subject to leakage in the future.The dataset we're using is an aggregation of publicly accessible datasets licensed under the Creative Commons license (CC-BY) or other open-source licenses. We've meticulously adhered to all required legal procedures to incorporate this data into our research, recognizing the importance of transparency in data licensing for proper attribution and suitable data utilization. Our dataset also encompasses images derived from publicly accessible datasets and language data created through the GPT-4V API. While measures have been put in place to secure suitable content, we acknowledge the potential existence of problematic content. Should you come across any such content, we urge you to inform us immediately so we can make the necessary adjustments to sustain a dataset free from inappropriate content. We are unwavering in our commitment to maintain a high-quality, ethically responsible dataset and promise to uphold principles of privacy and transparency throughout our work.As shown in Table 3 , we enumerate the ten most extensively utilized MLLM benchmarks used recently. These benchmarks are compared across six dimensions: Visual Modality, Customized Question, Average Images, Max Images, Total Samples, and Question Type. It is noteworthy that our benchmark presents a greater challenge, particularly in the areas of Average Images and Max Images, in comparison to other benchmarks. Simultaneously, in terms of data modality, question types, and quantity, it is comparable to existing works. In Table 4 , we present a detailed taxonomy of the dataset, task composition, as well as the number of samples and metrics corresponding to each task. Please construct a multiple-choice question using the provided images and accompanying text. The question must be relevant to multiple images. The four answer choices should be as comparable as feasible, with only one correct answer. Present your response in the following structure: 'Question: <Question> Options: <Option 1><Option 2><Option 3><Option 4> Answer: <Answer>'. Within the MILEBENCH, three datasets were constructed by our authors. We employed a systematic approach for our task-specific data construction.• The Long Text with Images QA (WikiVQA) is derived from multimodal English Wiki documents. We collected documents that contain more than two images and have a text length ranging between 1000 and 8000 words. For each data entry, we extracted images and their associated captions from each document and leveraged GPT-4V to generate a question, options, and an answer, based on a prompt illustrated in Figure 10 . Subsequently, we structured each data entry of WikiVQA by using the original multimodal document as the context, accompanied by the question, options, and the corresponding answer. • For Image Retrieval, we used images from the GPR1200 (Schall et al., 2022) , which encompasses image data collated from a myriad of sources across 120 categories.For each data entry, we select two images from one category as anchor and positive samples, and then uniformly sample 4 to 62 images from other categories as negative samples. We provide basic information of benchmarked models in Table 5 , as well as brief introduction for each model to highlight their characteristics:• GPT-4V (OpenAI, 2023) and GPT-4o7 are developed by OpenAI and are deemed the most powerful vision-language models for comprehension and generation. We report testing GPT-4V (gpt-4-turbo-2024-04-09) on April 12, 2024 and GPT-4o (gpt-4o-2024-05-13) on May 13, 2024. • Gemini 1.0 (Anil et al., 2023) and Gemini 1.5 (Reid et al., 2024) are models developed by Google, demonstrating competitive ability among closed-source MLLMs. We report testing Gemini 1.0 (gemini-1.0-pro-vision-001) on March 20, 2024, and Gemini 1.5 (gemini-1.5-pro-latest) on April 12, 2024. Gemini 1.0 has a limitation of accepting up to 16 images as input.• Claude 3 Opus8 is a vision-language model recently released by Anthropic, impressing the community with its extra-long 200K context length. We report testing Claude 3 (claude-3-opus-20240229) on March 16, 2024. Claude 3 Opus has a limitation of accepting up to 20 images as input • LLaVA-v1.5 (Liu et al., 2023a ) introduces a simple yet effective modality alignment strategy, based on which many other models are developed. • MiniGPT-v2 (Chen et al., 2023) is a concurrent work with LLaVA-v1.5. It adopts a similar structure to the latter and is a grounding-enhanced model. • LLaVA-v1.6 (Liu et al., 2024b ) takes a step further on the basis of LLaVA-v1.5. It is able to process images with any resolution and releases more variants based on different LLMs. • ALLaVA-Longer (Chen et al., 2024) is a lite version of LLaVA with enhanced complex reasoning ability, even achieving competitive results with larger models. • Yi-VL (Young et al., 2024) is based on LLaVA architecture and adopts a 3-stage training process. • Cheetor (Li et al., 2023c) proposes to use a Visual Prompt Generator to capture residual visual details, which might be vital for model performance. • Qwen-VL-Chat (Bai et al., 2023 ) is a 7B model trained on billions of multimodal samples. • VILA (Lin et al., 2023) adopts a LLaVA-like structure but is trained with interleaved image-text data.• Mantis (Jiang et al., 2024) uses LLaMA39 as language model and is trained with multi-image data. • Open flamingo (Awadalla et al., 2023) the open-source implementation of Flamingo (Alayrac et al., 2022) , which serves as the foundation of subsequent multi-image and video comprehension models.• Video-LLaMA-2 (Zhang et al., 2023 ) is an MLLM that can process vision and audio data by adopting two distinct encoders and Q-Formers.• Valley (Luo et al., 2023) is a video comprehension model that uses a temporal modeling module to encode video inputs into embeddings, which are then concatenated with textual features to perform downstream tasks.• VideoChat2 (Li et al., 2023d) trains a Q-Former to extract visual features, and the LLM remains frozen with the added LoRA layer tuned.• LLaMA-VID (Li et al., 2023e ) conducts an extreme compression of images, representing an image with only 2 tokens.• LWM (Liu et al., 2024a ) is progressively trained with multimodal data to extend its context length. It is by far the only open-source MLLM that can handle up to 1M tokens as its inputs.In Figure 11 , we illustrate the prompt structure employed during our evaluation phase. We crafted a unified input prompt structure for each dataset, incorporating dataset-specific instructions. For a detailed view of the specific instructions utilized, please refer to Table 6 for temporal multi-image tasks, Table 7 for semantic multi-image tasks, Table 8 for diagnostic evaluation.Prompt for Multi-choice QA: Instruction: {Instruction} Question: {Interleaved image-text question} Choice List: {choices} Answer:Prompt for Open-ended QA: Instruction: {Instruction} Question: {Interleaved image-text question} Answer:Figure 11 : Prompts for evaluating models.Table 6 : Instructions for the datasets in temporal multi-image tasks.Analyze the provided visuals and determine the timing of the event in question. You must choose your answer from the Choice List.Analyze the provided visuals and forecast the individual's subsequent move. You must choose your answer from the Choice List.Based on the provided images, answer the question related to the sequence of action You must choose your answer from the Choice List.Based on the provided images, answer the question related to the existence of objects. You must choose your answer from the Choice List.Based on the provided images, answer the question related to the interaction of objects. You must choose your answer from the Choice List.Based on the provided images, answer the question related to the moving attribute. You must choose your answer from the Choice List.Based on the provided images, and find the hidden object. You must choose your answer from the Choice List.Egocentric Navigation Analyze the provided visuals and instructions, then determine the subsequent step. You must choose your answer from the Choice List.Based on the provided images, answer the question related to the moving direction. You must choose your answer from the Choice List.Based on the provided images, answer the question related to the counterfactual inference. You must choose your answer from the Choice List.Analyze the provided visuals and determine the change of the state in question. You must choose your answer from the Choice List.Analyze the given visuals and answer the question about the order of character. You must choose your answer from the Choice List.Analyze the provided visuals and determine the transition of the scene in question. You must choose your answer from the Choice List.Table 7 : Instructions for the datasets in semantic multi-image tasks.Webpage QA I will give you several images and a question, your job is to seek information in the slide and answer the question correctly. You must choose your answer from the Choice List.Textbook QA Provided with a series of diagrams from a textbook, your responsibility is to correctly answer the following question. You must choose your answer from the Choice List.Given a collection of relevant data, which includes images, text, and tables, your task is to respond accurately to the ensuing question. You must choose your answer from the Choice List.Analyze the given context and associated images, draw inferences from the combination of both, and provide responses to posed questions.Slide QA I will give you several slides and a question, your job is to seek information in the slide and answer the question correctly. You must choose your answer from the Choice List.OCR QA I will give you two pictures of the book cover. Please look at the pictures and answer a question You must choose your answer from the Choice List.Document QA I will give you some pictures, and each group of pictures will correspond to a question. Please answer it briefly. You must choose your answer from the Choice List.What's the difference between 2 images?Please give a editing Request to describe the transformation from the source image to the target image.Multimodal Dialogue Provided with a variety of pertinent information, including images, text, tables, and previous Q&A history, your role is to answer the upcoming question accurately.Give you a main goal, your job is to figure out what to do now by looking at current envirments. Your past views as well as decisions are also provided.Given six images taken from different cameras on a street view car, your task is to answer questions about the depicted scene. You must choose your answer from the Choice List.Table 8 : Instructions for the datasets in diagnostic evaluation.Answer user question concisely and directly based on the provided images.Answer user question concisely and directly based on the provided images.Given the anchor image and the candidate images, identify which candidate image is most visually similar to the anchor image. To investigate the multi-task characteristics of the realistic evaluation in our MILEBENCH, we analyzed the performance of all models across the nine tasks within this evaluation and calculated pairwise correlations between different tasks, as shown in Figure 13 . (1) We found that, aside from Task S-3 (Visual Relation Inference), tasks within the same category (either temporal multi-image or semantic multi-image) exhibited high correlation. Task S-3, being a challenging one, showed little variation in scores across models. (2) We also noted that Task T-3 (Visual Navigation and Spatial Localization) demonstrated lower correlation with other tasks, possibly due to its requirement of unique cognitive skills such as understanding the world from a firstperson perspective. These observations suggest that the realistic evaluation of MILEBENCH encompasses a broad range of task types, offering a more comprehensive assessment in the context of multi-image long-context scenarios. 19.9 14.6 35.0 6.5 36.0 37.5 46.5 14.5 28.5 13.9 47.0 37.0 52.5 LWM 13.3 14.9 13.5 6.5 3.5 17.5 15.0 17.5 11.0 10.9 9.0 1.5 1.0 We define "multimodal long contexts" as long text content integrated with two or more images, or content composed of multiple images.https://openai.com/index/hello-gpt-4oWe adapted the taxonomy from MVBench(Li et al., 2024) and DEMON(Li et al., 2023c) to suit our multimodal long-context setting.T, S, N, I denotes Temporal Multi-image Tasks, Semantic Multi-image Tasks, Needle in a Haystack Tasks, and Image Retrieval Tasks, respectively. The numerical value is the index of the task within the set of tasks.A degree of similarity among the annotations made by different annotators on the same data.https://www.anthropic.com/news/claude-3-familyhttps://openai.com/index/hello-gpt-4o/https://www.anthropic.com/news/claude-3-familyhttps://github.com/meta-llama/llama3 


Decision: The authors propose a new benchmark for testing the long-context capability of MLLMs, which involves inputs with a large number of image and text tokens. They provide extensive evaluation results of frontier MLLMs. While similar benchmarks exist for LLMs (like "needle in a haystack"), this is the first benchmark to test long-context capabilities in MLLMs. It includes both artificial and realistic scenarios.

Some reviewers raised concerns, such as:

- Some MLLMs specialized in long-context processing were missing in the evaluation.
- Missing discussion on whether the current low performance is a fundamental issue.
- Missing evaluation of closed LLMs and potential data contamination.

However, the authors provided satisfactory answers during the rebuttal.
Overall, all the reviewers were eventually satisfied, and we recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper presents a new benchmark, with a focus on multi-image scenarios. The benchmark mainly evaluates the long-context performance on the M-LLMs. 

The paper provides a comprehensive evaluation of several existing models, and shows that long-context capability is demanding for improvements.
strength_weakness: strength_weakness: - The testing samples of the proposed dataset are mostly manually borrowed from existing datasets. There is no reliable pipeline to further collect and scale up. 

- It remains difficult to automatically evaluate these models which have free-form text outputs. The experiments only consider ROUGE-L and retrieval accuracy as the main metric. This may limit the long-context output scenarios.
paper_summary: paper_summary: The paper introduces MILEBENCH, a benchmark designed to assess the capabilities of Multimodal Large Language Models (MLLMs) in handling long-context scenarios involving multiple images and texts. This benchmark is motivated by the limitations of existing benchmarks that predominantly focus on short texts and single images, which do not adequately represent real-world multimodal tasks that require handling complex, lengthy contexts with multiple images. MILEBENCH includes two types of evaluations: diagnostic and realistic, aiming to thoroughly assess the performance of MLLMs in complex scenarios that mimic real-world applications. The benchmark tests 19 different models, highlighting that while the proprietary model GPT-4(Vision) excels, most open-source models struggle with these tasks.
strength_weakness: strength_weakness: 1. Majority of the curated benchmark comes from existing dataset, which may have data leakage and lead to unfairness and bias. Moreover, which part of the existing dataset did you extract the data from, training set, validation set, or test set? 
2. While MILEBENCH incorporates a variety of tasks and sources, there is little discussion on the potential biases inherent in these sources or how they might affect the benchmark's generalizability.
questions: questions: Please check the "Reasons To Reject"
paper_summary: paper_summary: This paper focuses on evaluating the long-context understanding capabilities of multimodal large language models. The authors introduce MILEBENCH, a comprehensive benchmark encompassing multiple dimensions of multimodal long-context understanding, such as Temporal Multi-image Understanding, Semantic Multi-image Understanding, Needle in a Haystack, and Image Retrieval. Additionally, the paper provides a brief evaluation of various multimodal models, ranging from closed-source to open-source, trained on images and videos, presenting a comparative analysis of their performance.
strength_weakness: strength_weakness: 1. This paper collects data from publicly available datasets/benchmarks, yielding only 200 samples for each task. However, some of the pre-existing datasets used herein (e.g., START [1]) encompass a vast collection of video clips and annotations potentially suitable for directly assessing multi-image capabilities. It's not clear whether these datasets alone are inadequate. The authors can provide substantial evidence to justify the necessity of the proposed benchmarks.
2. Recent developments have seen certain models displaying capabilities in multiple images understanding (e.g., MM1 [2], mPLUG-Owl2 [3], MMICL [4]). These models should be included in the evaluation to provide a comprehensive understanding of their capabilities and how training strategy/training data/model designing affect the multimodal long-context understanding capabilities.
3. The experiment results indicate that some models struggle to follow instructions and perform poorly on several samples. It begs the question of whether these models inherently lack the multimodal long-context understanding or if their performance is hindered by unfamiliar patterns that might be improved with training on a few of data samples.  In essence, the authors are encouraged to delve deeper into discussing which fundamental abilities, which preventing the models from effectively tackling multimodal long-context problems, are absent.

[1] Wu, B., Yu, S., Chen, Z., Tenenbaum, J. B., & Gan, C. (2021, August). Star: A benchmark for situated reasoning in real-world videos. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track (Round 2).
[2] McKinzie, B., Gan, Z., Fauconnier, J. P., Dodge, S., Zhang, B., Dufter, P., ... & Yang, Y. (2024). Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611.
[3] Ye, Q., Xu, H., Ye, J., Yan, M., Liu, H., Qian, Q., ... & Zhou, J. (2023). mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04257.
[4] Zhao, H., Cai, Z., Si, S., Ma, X., An, K., Chen, L., ... & Chang, B. (2023). Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv preprint arXiv:2309.07915.
questions: questions: Refer to the Reasons To Reject.
paper_summary: paper_summary: The paper presents a new benchmark dataset for multimodal (image+text) LLMs, called MileBench.

The novelty of this benchmark is in its focus - testing multimodal models in long contexts with many images. The collected dataset is a subset of existing datasets and synthetically generated examples. The examples from the existing datasets were picked in a way which agrees with the benchmark's focus. The resulting dataset consists of around 6k examples categorized in 12 different tasks.

The authors also measured the performance of various models on this benchmark and report the results. Notably, the closed-source models emerged as clear winners on most of the tasks.

Overall, the paper is well written and presents a relevant and potentially useful contribution.
strength_weakness: strength_weakness: * It is not very clear which datasets have been used for the data collection. References are given to the tasks, so I am assuming the data comes from there, but this should be listed explicitly, perhaps in the appendix.
* Inadequate attention given to possible data contamination in discussion about the results (see my question below).
questions: questions: * Have you tried using different metrics than ROUGE-L? I think for open ended generation tasks it might make more sense to use a LLM-based metric instead. I realize that selecting a good metric is secondary to this paper's focus, but by introducing a benchmark like this, you basically dictate which metric to use if someone else decides to use the benchmark to compare with other models.

* Given that the collected benchmark is a subset of existing datasets, is it possible that the success of the closed-source models is determined by a data leakage? (https://aclanthology.org/2024.eacl-long.5/) Why are these models omitted in the analysis in section 4.3.3?

Some minor suggestions:
For better clarity, Table 2 should indicate which numbers are accuracy and which are Rouge-L scores.

Comments: title: Thanks for the response
Thanks for the response
Response
Official Comment for Response
comment: Thanks for your response. I would maintain the rating.
Thanks for your response. I would maintain the rating.
Thanks for pointing out that the data sources are actually already listed in the appendix - it's my bad I didn't notice previously. Since I presented is originally as a reason to reject, I am raising the score to 7.

I acknowledge the rest of the answers, however they don't particularly ease my concerns about insufficient data leakage discussion in the paper.
Thanks for the response, I would like to keep my score.
I have no further questions at this point and would like to keep my initial rating.