MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs

## Abstract

Abstract We introduce MIA-Bench, a new benchmark designed to evaluate multimodal large language models (MLLMs) on their ability to strictly adhere to complex instructions. Our benchmark comprises a diverse set of 400 image-prompt pairs, each crafted to challenge the models’ compliance with layered instructions in generating accurate responses that satisfy specific requested patterns. Evaluation results from a wide array of state-of-the-art MLLMs reveal significant variations in performance, highlighting areas for improvement in instruction fidelity. Additionally, we create extra training data and explore supervised fine-tuning to enhance the models’ ability to strictly follow instructions without compromising performance on other tasks. We hope this benchmark not only serves as a tool for measuring MLLM adherence to instructions, but also guides future developments in MLLM training methods.

## 1 Introduction

Figure: Figure 2: Comparison of various multimodal LLM benchmarks. (Left) Fixed-form visual question answering, often features short answers or multi-choice formats, such as MMMU []. This format is popular due to its ease of evaluation. (Middle) Open-ended responses, such as LLaVA-Bench (in the Wild) []. (Right) The proposed MIA-Bench, which also uses open-ended responses but focuses on evaluating precise adherence to complex instructions within the prompt.
Refer to caption: extracted/5755763/figures/comparison_with_mmmu_and_llava_itw.png

The rapid advancement of Multimodal Large Language Models (MLLMs) [, , , , , , , , ] has been a defining feature of recent AI research, showcasing increased model capabilities to comprehend and respond to visual inputs, often termed as multimodal “instruction following”.

To measure the progress of instruction following, many multimodal benchmarks have been developed, which can be roughly divided into two broad categories: ($i$) fixed-form visual question answering (VQA), often with short answers or using a multi-choice QA format; and ($ii$) free-form conversations with open-ended responses. Many current benchmarks have adopted the first format, including VQAv2 [], TextVQA [], ScienceQA [], MME [], MMBench [], SEED-Bench [], MathVista [], and MMMU []. These benchmarks are popular due to their ease of use in evaluating metrics and presenting model comparisons.

However, as visual assistant models, the ability to engage users in free-form conversations is also crucial. Benchmarks in this format include LLaVA-Bench [], MM-Vet [], VisIT-Bench [], InfiMM-Eval [], and the most recent Vibe-Eval [] and LLaVA-Bench-Wilder []. Typically, the free-form model responses are evaluated using external models as the judge.
These benchmarks are closer to daily-life visual chat scenarios; however, the type of “instruction following” examined in these benchmarks usually gauges a model’s ability to perform tasks in a broad, often loosely defined manner. Yet, the precise adherence to complex instructions within prompts – a critical aspect for evaluating LLMs [, , ] – remains less explored in the context of multimodal LLMs.

To this end, we introduce MIA-Bench,(^1^11Abbreviation for Multimodal Instruction Adherence Benchmark.)a new benchmark specifically designed for evaluating strict “instruction adherence”.
Our instruction adherence metric measures the precision with which MLLMs can execute layered and compositional instructions.
This involves not only recognizing the content of the instructions, but also meticulously executing the detailed demands without deviation (*e.g.*, answering in a given number of sentences, including specific elements, etc.).
By establishing this stricter criterion, our benchmark aims to push the boundaries of model precision and reliability in practical applications, ensuring that outputs not only align with the general intent of the instructions, but also match the exact specifications provided. An example from MIA-Bench is provided in Figure 1, and its comparison with previous MLLM benchmarks is illustrated in Figure 2.

MIA-Bench consists of 400 meticulously created image-prompt pairs, and encompasses diverse image contents including animals, food, landmarks, sport, art, landscape, text, etc.
to cover a broad spectrum of real-world scenarios. In constructing this benchmark, we sought not only to evaluate the current capabilities of state-of-the-art MLLMs, but also to push the boundaries of what these models can achieve when rigorously tested against structured and layered instructions. The final prompts are of various complexity levels, and compositional in nature, with five base instruction categories, which are tailored to probe the models’ linguistic dexterity, grammatical accuracy, and descriptive fidelity. For example, the prompt in Figure 1 is composed of five base categories, including description, mention, grammar, length limit, and genre.

We evaluate a wide array of MLLMs on the proposed benchmark, ranging from closed-source models (*e.g.*, GPT-4o [], Gemini Pro [], Claude-3 [], Reka []) to open-source ones (*e.g.*, LLaVA-NeXT [], Intern-VL-Chat-1.5 [], CogVLM2 [], Phi-3-Vision []). Our investigations reveal notable variations in model performance, highlighting great opportunities for improvement.

To address these challenges, we further propose to generate training data tailored for supervised fine-tuning (SFT), where we aim to refine the models’ abilities to process and comply with multifaceted instructions. Results from our SFT experiments indicate a promising enhancement in the models’ performance to strictly adhere to instructions, without hurting performance on other benchmarks.

Our contributions are summarized as follows. ($i$) We construct MIA-Bench, a new benchmark to comprehensively evaluate MLLMs on their capability to strictly adhere to instructions. ($ii$) We provide a detailed analysis of popular MLLMs, and suggest training methods for enhanced instruction following. For this purpose, we created training data and conducted experiments for additional supervised fine-tuning. MIA-Bench will be open-sourced,
and we hope this benchmark can serve as a useful
resource to stimulate further research on multimodal instruction adherence.

## 2 MIA-Bench

Figure: Figure 3: Examples from MIA-Bench, with detailed information on the instruction composition, base instruction weight and type.
Refer to caption: extracted/5755763/figures/instruction_example_5.png

MIA-Bench consists of 400 image-prompt pairs, with examples shown in Figure 3. The images are collected from diverse sources, including COCO 2017 validation set [], SBU [], TextVQA [], and Flickr. Images in the Flickr subset are photos of a variety of themes, including animals, art, architectures, text, food, math, etc. Images from the other three sources are randomly sampled from each corresponding source.
Figure 4 shows the top 15 image content categories and the distribution of the 8 sub-instruction categories in MIA-Bench. The image content is labeled using GPT-4v. For each image, we manually write diverse and challenging instructions that contain multiple sub-instructions.

Figure: Figure 4: The top-15 image content categories and the distribution of the 8 sub-instruction categories in MIA-Bench.
Refer to caption: x2.png

When constructing the instructions, we follow three principles, detailed below.

- •
Correctness. The instruction needs to be answerable by humans. For example, asking about objects that do not exist in the image makes the prompt unanswerable.
- •
No answer leakage. The instruction should not contain the answer to itself. ‘What color is the green object?’ is an example of answer leakage.
- •
Image-dependent. MMStar [] pointed out that on some multimodal benchmarks, MLLMs can generate correct answers without accessing images half of the time. Multi-modal capabilities are necessary to correctly answer MIA-Bench prompts.

### 2.1 Instruction Categories

In this paper, we use instruction to refer to the entire textual input, which in MIA-Bench can generally be viewed as a composition of multiple individual requests or constraints. We refer to these individual components as sub-instruction. Instructions in MIA-Bench are of diverse complexity, and sub-instructions contained are of multiple categories, summarized in Figure 4.

The sub-instructions in MIA-Bench can be categorized into description, length limit, mention, genre, grammar, math, perspective, and OCR, detailed below.

- •
‘description’ refers to describing a certain part of the image, with the exception of text-rich parts of the image, which falls under the ‘OCR’;
- •
‘length limit’ refers to the limitation of response length (*e.g.,* in exactly two sentences, using exactly 60 words);
- •
‘mention’ refers to mentioning or not mentioning certain objects or entities (*e.g.*, highlighting two similarities and one difference, comparing and contrasting the condition of the buildings with the activity on the street);
- •
‘genre’ refers to requests for a specific written form (*e.g.*, write a poem, write a narrative, with at least one pun included, all while weaving in a subtle theme of change);
- •
‘grammar’ refers to grammatical requirements (*e.g.*, use present tense, use capitalized letters, use integers);
- •
‘math’ refers to requirements to come up with a solution to math problems, or to identify errors in solutions to math problems, or to generate a valid math problem given table, charts, etc.;
- •
‘perspective’ refers to requirements specifying the viewpoint of an object or person in the image. This requires MLLMs to correctly identify what can or cannot be seen from the specified position, and understand the spatial relationship of objects in its surrounding with itself (*e.g.*, imagine you are the lady in the image, describe what you can see without turning your head around);
- •
‘OCR’ refers to requirements related to understanding OCR information in text-rich images such as menus, tickets, bills, etc. For example, given a photo of a ticket, the sub-instruction asking about the price printed on the ticket falls into this category.

Figure: Figure 5: The most frequently used verbs and co-occurring nouns in MIA-Bench.
Refer to caption: extracted/5755763/figures/words.png

Figure 5 shows the most frequently used verbs and co-occurring nouns in MIA-Bench.
To guarantee the diversity of prompts, when writing the instructions, we contribute instructions of various levels of complexity: basic, intermediate, advanced, creative, and complex.
The basic category is the simplest; the instructions normally only contain one or two sub-instructions, such as “What is the color of the cat?”, or “Describe the sofa in two words.”. The intermediate category consists of instructions that contain three or more sub-instructions, but are in general easy for MLLMs to follow.
The advanced category contains instructions that are challenging and contain three or more sub-instructions. The creative category contains instructions that instruct MLLMs to generate creative pieces of text, such as poems.
The complex category is a combination of the previous two categories; the instructions in this category are the most complicated as they usually contain multiple challenging sub-instructions. While we found these categories useful to elicit a diverse instruction set, we also found that practical examples were often difficult to categorize objectively. As a result, we only used these categories for data collection, but are not reporting per-category results.

### 2.2 Response Evaluation Method

We adopt GPT-4o [] to score MLLMs’ responses on each instruction and return a total score using the following prompt:

[Uncaptioned image]: x3.png

Each response is graded by first assessing how well it follows each sub-instruction, then computing the total score. Figure 6 shows an example of how responses from different MLLMs are evaluated and scored. Each sub-instruction in an instruction is assigned a maximum score ranging from 1 to 10; sum of the weight of all sub-instructions in an instruction is 10. For the example in Figure 6, there are 4 sub-instructions (denoted from S1 to S4); the first is worth 4 points and the rest is worth 2 points each. The response from GPT-4o partially follows the first sub-instruction which requires the response to be from the perspective of the dog, as the dog should not be able to see the car behind the man without turning around. The dog should be able to see the guitar, thus GPT-4o gets 2 points out of 4 for the first sub-instruction. It successfully follows the other 3 sub-instructions, achieving full score for them. Thus, the final score GPT-4o reaches is 8 out of 10. We always assign larger weight (6 if there are two sub-instructions, 4 if there are three or more sub-instructions) to the sub-instruction in the description category unless this category is absent in some cases, as usually a major part of the response is addressing this sub-instruction. For each MLLM, we compute the average score it gets on all 400 responses, and represent the ratio of the average score divided by 10 using percentage. We also compute the average score for each instruction category.

Figure: Figure 6: An example with responses from four MLLMs and their evaluation scores.
Refer to caption: x4.png

## 3 Experiments

In this section, we first present results of different MLLMs on MIA-Bench in Section 3.1, with additional supervised fine-tuning exploration in Section 3.2.

### 3.1 Benchmark Results

**Table 1: Evaluation results of a wide array of MLLMs on MIA-Bench.**
| Model | Meta-Avg | Description | Len-Limit | Genre | Grammar | Mention | Math | Perspective | OCR |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Open Source |  |  |  |  |  |  |  |  |  |
| Fuyu-8b [] | 24.52 | 52.06 | 24.52 | 17.06 | 17.18 | 36.43 | 22.62 | 66.67 | 33.09 |
| Kosmos-2 [] | 26.06 | 50.95 | 38.52 | 11.55 | 19.78 | 28.70 | 17.26 | 50.83 | 41.88 |
| InstructBLIP-13b [] | 38.16 | 50.54 | 39.57 | 29.34 | 38.43 | 42.28 | 12.50 | 50.00 | 30.42 |
| Sphinx [] | 50.99 | 75.33 | 53.51 | 60.45 | 48.28 | 57.75 | 47.41 | 70.00 | 61.04 |
| Idefics-2-8b [] | 51.42 | 59.37 | 62.73 | 48.07 | 64.09 | 46.20 | 46.51 | 48.33 | 61.97 |
| Yi-VL-34b [] | 53.90 | 74.89 | 52.05 | 59.09 | 55.91 | 57.25 | 54.17 | 41.85 | 70.09 |
| mPLUG-Owl2 [] | 57.86 | 75.01 | 65.25 | 63.39 | 60.26 | 57.70 | 57.22 | 65.00 | 62.08 |
| CogVLM-Chat [] | 58.95 | 60.42 | 57.86 | 67.94 | 60.55 | 62.92 | 36.67 | 60.83 | 61.87 |
| ShareGPT4V [] | 59.41 | 81.08 | 63.49 | 63.88 | 58.46 | 62.49 | 52.98 | 82.50 | 72.29 |
| DeepSeek-VL-7b-chat [] | 60.96 | 86.31 | 63.26 | 72.11 | 54.79 | 63.75 | 67.39 | 74.17 | 77.85 |
| LLaVA-1.5-7b [] | 62.18 | 78.00 | 68.60 | 63.95 | 64.18 | 65.89 | 47.31 | 86.67 | 60.75 |
| LLaVA-NeXT-7b-vicuna [] | 62.27 | 79.21 | 68.01 | 65.63 | 60.95 | 63.33 | 46.67 | 90.00 | 65.54 |
| Qwen-VL-Chat [] | 63.09 | 80.51 | 74.22 | 66.95 | 63.11 | 63.01 | 45.00 | 75.83 | 66.01 |
| LLaVA-1.5-13b [] | 63.55 | 80.98 | 70.15 | 64.54 | 59.30 | 67.42 | 45.11 | 69.17 | 76.28 |
| XComposer2-7b [] | 67.71 | 83.47 | 76.16 | 73.66 | 67.69 | 67.01 | 48.61 | 77.50 | 68.06 |
| LLaVA-NeXT-13b-vicuna [] | 69.16 | 86.75 | 69.88 | 82.07 | 64.77 | 74.99 | 48.56 | 77.50 | 75.83 |
| CogVLM2 [] | 73.43 | 87.60 | 74.52 | 83.47 | 71.97 | 77.01 | 71.53 | 90.83 | 87.16 |
| InternVL-Chat-v1.5 [] | 75.42 | 89.13 | 78.21 | 79.92 | 78.16 | 77.54 | 76.11 | 87.50 | 80.92 |
| LLaVA-NeXT-34b [] | 75.61 | 88.02 | 83.50 | 86.58 | 71.57 | 75.83 | 68.06 | 87.50 | 80.26 |
| Phi-3-vision [] | 76.02 | 84.90 | 84.46 | 86.52 | 67.93 | 74.70 | 78.16 | 74.17 | 83.96 |
| MiniCPM-Llama3-v2.5 [] | 76.27 | 84.12 | 79.44 | 80.33 | 81.25 | 76.99 | 64.08 | 81.67 | 76.59 |
| LLaVA-NeXT-110b [] | 79.84 | 86.99 | 84.86 | 82.49 | 79.04 | 80.10 | 71.94 | 80.83 | 75.45 |
| Proprietary |  |  |  |  |  |  |  |  |  |
| Gemini-1.0-Pro [] | 70.63 | 82.77 | 72.83 | 78.76 | 76.91 | 71.67 | 81.45 | 89.29 | 84.11 |
| Reka-Core [] | 76.95 | 91.05 | 79.91 | 85.16 | 78.98 | 82.08 | 82.53 | 77.50 | 81.08 |
| Claude-3-Haiku [] | 78.25 | 86.86 | 77.53 | 90.27 | 73.41 | 82.62 | 82.22 | 57.50 | 86.49 |
| Claude-3-Sonnet [] | 79.44 | 88.06 | 82.71 | 90.54 | 79.60 | 82.05 | 82.22 | 76.67 | 84.43 |
| Claude-3-Opus [] | 84.50 | 90.50 | 86.03 | 91.19 | 83.82 | 85.49 | 85.92 | 65.00 | 86.84 |
| GPT-4v [] | 86.11 | 90.03 | 87.61 | 94.59 | 80.12 | 89.37 | 85.63 | 59.17 | 85.26 |
| GPT-4o [] | 88.58 | 90.82 | 92.73 | 94.29 | 85.70 | 90.66 | 87.07 | 92.50 | 86.54 |

In total, we have evaluated 29 popular MLLMs on MIA-Bench. Results are reported in Table 1. Observations are summarized as follows.

- •
Overall, the best performance was achieved by GPT-4o [], with a score 88.58, showcasing its superiority across different categories of instruction adherence.
- •
The ability to describe content accurately was best exhibited by Reka []. Other models like Claude-3-Opus [], GPT-4v [] and GPT-4o also achieved scores higher than 90. This suggests that these models are good at generating coherent and contextually appropriate text.
- •
In the genre category, the highest proficiency was shown by GPT-4v and GPT-4o with scores above 94, suggesting an exceptional grasp of language nuances. Among open-source models, Phi-3-Vision [] and LLaVA-NeXT-34b [] show strong performance with scores of 86.52 and 86.58, respectively. The lowest score on this metric was by Kosmos-2 [], with a mere 11.55, pointing to difficulties in understanding or generating linguistically complex sentences.
- •
GPT-4o excelled in grammar with a score of 85.70, which indicates superior ability in syntax correctness and sentence structuring that matches specific requirements in the instruction. Among the open-source models, MiniCPM-Llama3-V-2.5 [] is notable with a score of 81.25. Contrastingly, Fuyu-8b [] scored the lowest with 17.18, reflecting major challenges in grammar adherence.
- •
GPT-4o also showed the best performance with a score of 92.73 in respecting prescribed length limits, which is crucial for tasks requiring concise and precise answers. Among open-source models, LLaVA-NeXT-110b [] stands out with a score of 84.86.
- •
Results from LLaVA series also suggest a strong correlation between LLM size and MIA-Bench performance across metrics.

**Table 2: Meta ranking of five state-of-the-art MLLMs on existing multimodal benchmarks compared with their ranking on MIA-Bench.**
| Model | MME | MMMU | MMB | MMVet | HallB | Math | Meta | MIA | MIA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Vista | Ranking | Ranking |  |  |  |  |  |  |  |
| GPT-4v [] | 1926.6 | 56.8 | 77/74.4 | 67.6 | 46.5 | 49.9 | 2 | 86.11 | 1 |
| Gemini-1.0-Pro [] | 1933.4 | 47.9 | 73.6/74.3 | 64.3 | 45.2 | 45.2 | 4 | 70.63 | 5 |
| Claude-3-Opus [] | 1586.8 | 59.4 | 63.3/59.2 | 58.1 | 37.8 | 50.5 | 5 | 84.50 | 2 |
| InternVL-Chat-V1-5 [] | 2187.8 | 45.2 | 82.2/82 | 62.8 | 49.3 | 53.5 | 1 | 75.42 | 4 |
| LLaVA-NeXT-34b [] | 2028 | 51.1 | 81.1/79 | 48.9 | 47.6 | 47.7 | 3 | 75.61 | 3 |

Correlation with other benchmarks.
In Table 2, we compare the ranking of 5 state-of-the-art MLLMs on MIA-Bench as well as their meta ranking on MME [], MMMU [] , MMBench [], MMVet [], HallusionBench [], and MathVista [] (meta ranking is computed by averaging rankings across these benchmarks). Our findings reveal a discrepancy between the two sets of rankings. Notably, InternVL-Chat-V1.5 [], which holds the highest meta-ranking among the five MLLMs on the other benchmarks, ranks the lowest on MIA-Bench. Conversely, Claude-3-Opus, which has the lowest meta-ranking, secures the second position on the MIA-Bench. This indicates that excelling in tasks evaluated by existing benchmarks does not necessarily translate to superior instruction adherence capability assessed by MIA-Bench.

Correlation with LLM backbone performance.
To determine if the performance on MIA-Bench is attributable solely to the underlying LLMs, we also evaluate several MLLMs on IFEval [], a benchmark that assesses the instruction adherence capability of LLMs, and compare their ranking with that on MIA-Bench. This comparison is shown in Appendix, which shows that the instruction adherence capabilities of MLLMs do not consistently align with their LLMs’ adherence capability.

Other external models as the judge.
Since the evaluation uses GPT-4o as the judge, it is natural to conjecture that GPT-4o may favorably score its own responses. To alleviate this concern, we use Claude-3, a strong performer in Table 1, to evaluate responses from GPT-4o and itself, and compare their scores with each other. The prompt used to grade responses is the same as the one used in GPT-4o grading. We find that even using Claude-3 Opus to score its own and GPT-4o’s responses, GPT-4o still achieves a superior score. When scored by Claude-3-Opus, GPT-4o achieves 89.84 score in contrast to Claude-3-Opus’ 85.89. Based on this observation, we use GPT-4o for evaluation by default, and observe that results from multiple runs may have around 1% variation.

### 3.2 Supervised Fine-Tuning (SFT)

The performance of small-scale models such as LLaVA-NeXT-13b is less ideal on MIA-Bench. In this section, we study the use of supervised fine-tuning to enhance model performance.

Additional SFT data construction.
First, we randomly sample 1000 images from COCO 2017 training set, and use GPT-4v to generate five instructions for each image, using the prompt below.

**Table 3: Detailed results on MIA-Bench before and after (denoted by *) supervised fine-tuning on additional constructed diverse instruction-tuning data. We re-ran the baseline.**
| Model | Total Score | Description | Length Limit | Genres | Grammar | Mention | Math | Perspective | OCR |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaVA-NeXT-13b [] | 69.16 | 86.75 | 69.88 | 82.07 | 64.77 | 74.99 | 48.56 | 77.50 | 75.83 |
| LLaVA-NeXT-13b* | 78.85 | 86.90 | 86.80 | 88.02 | 71.34 | 81.01 | 60.87 | 84.17 | 72.65 |

**Table 4: Results on MIA-Bench and other major multimodal benchmarks before and after (denoted by *) supervised fine-tuning on additional diverse instruction-tuning data. We re-ran the baseline.**
| Model | MMBench | TextVQA | VQA2 | LLaVA-itw | POPE | VizWiz | MathVista | MIA-Bench |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaVA-NeXT-13b [] | 70.6 | 64.26 | 82.80 | 85.8 | 87.7 | 60.41 | 33.0 | 69.16 |
| LLaVA-NeXT-13b* | 68.6 | 63.20 | 82.58 | 83.4 | 86.9 | 59.72 | 32.0 | 78.85 |

[Uncaptioned image]: x5.png

We then manually process the generated instructions. The cleaned data for SFT consists of 5000 image-prompt pairs.

Then, we use GPT-4v to generate responses to the constructed prompts. To evaluate the quality of these responses, we sampled 100 responses and manually checked if they adhere to the instructions. We find that 90% of the responses successfully followed all instructions in the prompt, serving as a proper ground-truth response for model training. Examples of this additional training data is provided in the Appendix.

Results.
Using LLaVA-NeXT-13b as the backbone, we train the model for 1 epoch on the constructed SFT data.
Results on MIA-Bench and other benchmarks are summarized in Table 4, with detailed results on MIA-Bench reported in Table 3. The performance on MIA-Bench has been significantly improved by around 10 points, at the cost of minor regressions across other benchmarks. Examples are shown in Figure 7 to compare responses from LLaVA-NeXT-13b before and after SFT.

Figure: Figure 7: Examples of LLaVA-NeXT-13b responses before and after supervised fine-tuning on additional diverse instruction-tuning data.
Refer to caption: x6.png

## 4 Related Work

Multimodal LLMs and Benchmarks.
Multimodal Large Language Models (MLLMs) have recently emerged as a significant research focus.
LLaVA [] and MiniGPT-4 [] pioneered visual instruction tuning, and the past
year has witnessed a boom of open-source MLLMs based on this concept. Prominent examples include InstructBLIP [], mPLUG-Owl(-2/Doc) [, , ], Qwen-VL [], CogVLM [], SPHINX(-X) [, ], InternLM-XComposer2-VL [], InternVL(-1.5) [, ], VILA [], MM1 [], Mini-Gemini [], Idefics2 [], Phi-3-vision [], to name a few. There is also a rich body of literature on enabling MLLMs for referring and grounding [, , , , , , , ], image generation and editing [, , ], *etc*.

Various benchmarks have been proposed to evaluate the performance of MLLMs across different dimensions. Benchmarks like VQAv2 [], TextVQA [], ScienceQA [], MME [], MMbench [], SEED-Bench [], MathVista [], and MMMU []
aim to assess comprehensive multimodal understanding abilities. Additionally, there are benchmarks that specifically study model hallucination, including POPE [], MHalDetect [], GAVIE [], HallusionBench [], and MAD-Bench []. Many of these benchmarks have gained popularity within the community due to their use of multiple-choice evaluations. However, they do not accurately reflect the common use cases for MLLMs, where user interactions are typically open-ended. To address this, benchmarks like LLaVA-Bench [], MM-Vet [], and Vibe-Eval [] have been proposed. Our MIA-Bench also falls into this category; however, we focus on studying the exact instruction adherence of MLLMs, a metric that previous benchmarks have only loosely measured.

Instruction Following Benchmarks for LLMs.
Several benchmarks have been proposed to measure the instruction adherence ability of LLMs. Instruction-Following Eval (IFEval) [] is a benchmark for assessing LLMs’ adherence ability to the given instructions. Its approach emphasizes verifiable instructions, which enhance objectivity and reproducibility in evaluations. IFEval creates 541 prompts spanning 25 instruction types, revealing a significant performance gap in instruction adherence ability between GPT-4 [] and PaLM-2 []. This demonstrates the benchmark’s ability to effectively differentiate between models in adherence ability.
On the other hand, InfoBench [] introduces a new metric called Decomposed Requirements Following Ratio (DRFR) for assessing the instruction-adherence capabilities of LLMs. DRFR dissects complex instructions into simpler sub-instructions, allowing for a granular evaluation of compliance with various task aspects. InfoBench contains 500 diverse instructions consisting of 2,250 decomposed questions in multiple constraint categories. The evaluation of advanced LLMs using this framework highlights their strengths and areas for improvement, especially in complex instruction adherence scenarios. Compared with these previous work, we are the first known effort that specifically focuses on benchmarking the instruction adherence ability of multimodal LLMs.

## 5 Conclusion

This paper introduces MIA-Bench, a benchmark designed to evaluate the ability of MLLMs to strictly adhere to complex instructions within prompts. Through the analysis of 400 image-prompt pairs from diverse sources, our findings highlight variability in model performance and much room for improvement, underscoring a critical need for enhanced training methods to improve instruction compliance.
We further explored supervised fine-tuning (SFT) using LLaVA-NeXT as the backbone, which yielded promising results. Going forward, future research can expand on both SFT and alignment methods such as RLHF [, ] and DPO [, ], enhancing MLLMs to achieve higher accuracy and reliability in practical applications across diverse instructional contexts.

## Limitation

In designing the instructions for our benchmark, we incorporated a wide range of categories to enhance the diversity of sub-instructions. Nonetheless, the real world presents an infinite variety of instructions, many of which may pose significant challenges for MLLMs.

## Appendix A Appendix

### A.1 Comparison between MIA-Bench and IFEval

Figure: Figure 8: Comparison between MIA-Bench and IFEval.
Refer to caption: extracted/5755763/figures/ifeval2.png

### A.2 Examples of diverse instruction-tuning data used for SFT

Figure: Figure 9: Examples of diverse instruction-tuning data used for SFT.
Refer to caption: x7.png

### A.3 Examples of how MLLMs respond to instructions in MIA-Bench

Figure: Figure 10: An example of how MLLMs respond to instructions in MIA-Bench.
Refer to caption: x8.png

Figure: Figure 11: Another example of how MLLMs respond to instructions in MIA-Bench.
Refer to caption: x9.png

Figure: Figure 12: The third example of how MLLMs respond to instructions in MIA-Bench.
Refer to caption: x10.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: please refer to the weakness part.
soundness: 3
strengths: (1) MIA-Bench fills a gap in existing benchmarks by focusing on the ability of multimodal large language models (MLLMs) to adhere to complex instructions, a crucial yet previously underexplored area of evaluation. This benchmark’s design reveals potential deficiencies in models when following strict, multi-layered instructions, supporting the practical deployment of multimodal models in complex, instruction-based tasks.
(2) The MIA-Bench dataset consists of 400 image-instruction pairs covering various instruction types, such as description, length limit, genre, grammar, and OCR. With data from diverse sources, it reflects the variety found in real-world scenarios. This diverse set of instructions enhances the benchmark’s comprehensiveness and real-world applicability.
(3) The paper provides a systematic evaluation of 29 popular MLLMs, analyzing their performance across different instruction categories. This large-scale comparison offers researchers a detailed performance reference and targeted insights for future model improvements.
summary: This paper introduces MIA-Bench, a benchmark designed to evaluate multimodal large language models (MLLMs) on strict adherence to complex instructions. Based on a dataset of 400 image-instruction pairs, MIA-Bench assesses the performance of 29 popular MLLMs, highlighting that current MLLMs struggle with precise instruction adherence. In addition, the paper also explored the supervised fine-tuning (SFT) method based on the LLaVA-NeXT model and achieved positive results, demonstrating the potential effectiveness of this method in improving model instruction adherence.
weaknesses: (1) Although the SFT experiments in the paper demonstrate improved performance in adhering to complex instructions, they may lack in terms of model generalization. The results may be specific to the LLaVA-NeXT model, with no validation of applicability to other models, leaving it unproven whether SFT is equally effective for enhancing complex instruction adherence in MLLMs of different architectures and sizes.
(2) The differences between MIA-Bench and other benchmarks are indeed striking, and the authors believe that this may indicate that MIA-Bench's unique design focuses more on evaluating the model's strict instruction-following ability. However, the authors' explanation does not completely rule out the possibility that MIA-Bench itself may have design biases. This explanation is based on speculative experimental results and has not been verified in depth by sufficient experiments.
(3) Although the authors introduced Claude-3-Opus for comparison to mitigate scoring bias, the two models may exhibit similar biases in evaluation. Therefore, using Claude-3-Opus as the sole comparison tool may be insufficient to fully reveal potential scoring biases in GPT-4o.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: I do not have any further questions. Please refer to the "weaknesses" section.
soundness: 3
strengths: 1. Evaluating the capability of LMMs to adhere to complex instructions is worth studying.
2. Table 2 demonstrates that MIA-Bench is not highly correlated with existing benchmarks.
3. Figure 8 illustrates that MIA-Bench is vision-centric, which is not strongly correlated with the performance of the LLM backbone.
summary: This paper proposes MIA-Bench, which mainly focuses on the instruction-following capability of current Large Multimodal Models (LMMs) under complex and compositional instructions. Unlike previous close-ended benchmarks (e.g., multiple choice for MMBench) and open-ended benchmarks (e.g., LLaVA-Bench), MIA-Bench aims to evaluate precise adherence to complex instructions. As for evaluation, the authors leverage GPT-4o as the judge model. The authors have evaluated many representative LMMs on the proposed MIA-Bench.
weaknesses: In the context of evaluating open-ended freeform responses like MIA-Bench, it is crucial to account for the inherent variability introduced by the judge model. For example, the performance of MM-Vet has been observed to fluctuate by up to 10 points when assessed with different versions of GPT. 

This variability raises several important considerations:

1. **Standard Deviation Reporting:** When evaluating Large Multimodal Models (LMMs) using the same judge model (e.g., GPT-4o) across multiple trials, it is essential to report the standard deviation of the performance metrics.
2. **Detailed Generation Configuration:** The specific generation parameters of the judge model, such as top-p, top-k, temperature, and num_beams, should be explicitly documented.
3. **Impact of Generation Configuration:** It is necessary to investigate whether the above generation configuration of the judge model has a substantial impact on the performance metrics.
4. **Performance Across Different Judge Models:** Detailed performance metrics should be reported for various judge models. Specifically, the performance of LMMs should be evaluated using different versions of judge models, such as Claude-3-Opus, GPT-4o-20240806, GPT-4o-20240513, GPT-4v-20240409, GPT-4-1106-preview, or even open-sourced models (e.g., Qwen2.5 and LLaMA-3.1). This comprehensive approach will help in identifying any model-specific biases and in providing a more reliable assessment of the LMMs.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The 400 instructions in MIA Bench are manually annotated. It is important to ensure the reasonableness of the sub-instructions, such as whether the length limitation is appropriate. Additionally, has there been any manual verification of the scoring results from GPT-4o to confirm their accuracy and reasonableness?

2. The instructions in MIA-Bench are composed of multiple sub-instructions. How is the number of sub-instructions per instruction determined? Is there a difficulty grading for the instructions, such that a higher number of sub-instructions indicates a greater level of difficulty?
soundness: 3
strengths: 1. The paper proposes a benchmark for evaluating the abilities of MLLMs on following compositional instructions, covering a variety of categories and tasks, and providing guidance for MLLMs in following complex composite instructions in the future. 

2. The paper proposes a set of instructions that can effectively enhance the ability of MLLMs to follow layered and compositional instructions.
summary: To evaluate and guide the instruction-following capabilities of multimodal models, MIA-Bench introduces a set of benchmarks for multimodal instruction. MIA-Bench primarily measures the abilities of MLLMs on following layered and compositional multimodal instructions and provides a set of multimodal instructions to enhance model performance in these areas.
weaknesses: 1. Since the images in MIA-Bench are sourced from widely used datasets such as COCO 2017, SBU, TextVQA, and Flickr, I believe you should prioritize evaluating whether the current open-source models exhibit any data leakage issues on these datasets to demonstrate that MIA-Bench does not suffer from data contamination.

2.  I notice that the experiments in Table 3 were conducted on LLaVA-Next-13b. Is this done after fine-tuning on a preference-aligned model? If so, I think this continued SFT setup leading to a performance improvement in MIA-Bench, while other benchmarks exhibit varying degrees of decline, is quite intuitive. Could you please provide additional experiments on the impact of mixing generated instructions with original instructions on model performance? Furthermore, I notice that there are only 5000 generated instructions here; how would the model performance on MIA-Bench be affected if the number of instructions were increased?

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: - In MIA-Bench, are instructions all related to the corresponding image content or randomly picked from a instruction bank?
- Instruction following abilities highly depends on LLMs used in the MLLM models. It would be better to group the performances in Table 1 and Table 2 according to the LLMs' size.
soundness: 2
strengths: + The paper presents a new benchmark that rigorously tests multimodal large language models' ability to follow complex instructions, addressing a previously underexplored area.

+ This paper offers a comprehensive set of 400 diverse image-prompt pairs that challenge models with layered instructions, enhancing the assessment of their linguistic and descriptive capabilities.

+ Experiment demonstrates performance improvements in instruction adherence through supervised fine-tuning.
summary: The paper introduces MIA-Bench, a benchmark for evaluating the ability of multimodal large language models (MLLMs) to follow complex instructions. The benchmark consists of 400 image-prompt pairs that test the models' compliance with layered instructions to generate accurate responses. The evaluation of various state-of-the-art MLLMs shows significant performance variations, indicating areas for improvement in instruction adherence. The paper also discusses the creation of additional training data and supervised fine-tuning to enhance the models' instruction-following capabilities without affecting their performance on other tasks.
weaknesses: - Complex instruction flowing ability might not be the main interests for MLLM users currently, especially when it comes to writting style constraints such as length and genre. Such abilities are mostly evaluated on LLMs.

- The performance of models on MIA-Bench does not necessarily correlate with their performance on other benchmarks, suggesting that excelling in this specific task may not translate to generalized improvements across different multimodal tasks.

Meta-review: This paper introduces a new MIA-Bench benchmark specifically designed for MLLM's complex instruction following ability study. The authors' major contribution is a new dataset containing 400 image-instruction pairs, where the instructions are written by humans with high quality. Besides, the authors provided a comprehensive evaluation of the existing MLLMs and provided SFT positive results. The weaknesses and concerns are the model generalization issue, the MIA-Bench's design biases, the diverse model judgment issue, and the potential data leakage issue. After rebuttal, the authors addressed most of these concerns, and all reviewers agreed with the acceptance. Meanwhile, the value of the new benchmark with 400 high-quality image-instruction pairs is highlighted. Therefore, the final recommendation is accept.

Decision: Accept (Poster)