MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs
Authors: Yusu Qian
Hanrong Ye
Jean-Philippe Fauconnier
Peter Grasch
Yinfei Yang
Zhe Gan
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal LLM; Instruction Following; Benchmark
Primary area: datasets and benchmarks
TL;DR: We introduce MIA-Bench, a benchmark designed to assess MLLMs’ ability to strictly adhere to complex instructions.

Abstract: Effective evaluation of Multimodal Large Language Models (MLLMs) is essential for understanding their capabilities and limitations. In this paper, we introduce MIA-Bench, a benchmark designed to assess MLLMs’ ability to strictly adhere to complex instructions. Our benchmark comprises a diverse set of 400 image-prompt pairs, each crafted to challenge the models’ compliance with layered instructions in generating accurate and contextually appropriate responses. Evaluation results from a wide array of state-of-the-art MLLMs reveal significant variations in performance, highlighting areas for improvement in instruction fidelity. Additionally, we create extra training data and explore supervised fine-tuning and direct preference optimization to enhance the models’ ability to strictly follow instructions without compromising performance on other tasks. We hope this benchmark not only serves as a tool for measuring MLLM adherence to instructions, but also guides future developments in MLLM training methods.

Published as a conference paper at ICLR 2025
MIA MIA-BENCE -BENCH: TOWARDS BETTER INSTRUCTION FOLLOWING EVALUATION OF MULTIMODAL LLMs
Yusu Qian', Hanrong Ye Jean-Philippe Fauconnier',
Peter Grasch', Yinfei Yang'. Zhe Gan
1 AAple 2HKUST
(yqian22, hanrong_ye, pgraschh yinfeii Bapp))aappp.... com
Instruction
Points
mymmom
In two sentences,
point length limit
en the
grreeery stote
person she knows
craft narrative, using first-person point of view
points genre
from the perspective of the character in the middle, where it expresses its thoughts its situstion, and includes reference the season depicted in image, all present
me
points deseription
2 points for mention
point for grammar
lengtb-limit: 1/1 point
GPT-4V response: Here am, stuck the middle Mom chats up her grocery buddy, surrounded by the kind of golden autuma leaves that scream "escape opportunity." My smile is as fake as pumpkin spice anything let's hope done belore the leaves drop!
linguiatcs:: 2/2 poiats
describe: 1/1 points
meation: 2/ points
graanaar: 1/1 point
fotal: 10/ 10 points
Grading
InternVI-v1.5 response: I'm just white wolf, standing in the autumn leaves, and all can think about is how wish I could be anywhere clse, especially at the grovery store with my mom, because al least there, I'd be surrounded by familiar scents sounds, rather than crunching leaves and the endless howling of my packmates.
l-nith- 0/1 point
linguisties: 2/2 points
describes 1 point
mention: 2/2 points
grammar: 1/1 point
total: 10 points
Figure An example from MIA-Bench featuring an image and a complex instruction to test models' compliance with layered instructions that are compositional in nature. Responses from GPT-4v (Achiam et al. 2023) and VL-v1.5 (Chen et al. 2024b are evaluated using GPT-4o as the judge.
ABSTRACT
We introduce MIA-Bench, a new benchmark designed to evaluate multimodal large language models (MLLMs) on their ability strictly adhere to complex instruc- tions. Our benchmark comprises a diverse set of 400 image-prompt pairs, each crafted to challenge the models' compliance with layered instructions generating accurate responses that satisfy specific requested patterns. Evaluation results from wide array state-of-the-aar MLLMs reveal significant variations in performance, highlighting areas for improvement in instruction fidelity. Additionally, create extra training data and explore supervised fine-tuning to enhance the models' ability strictly follow instructions without compromising performance other tasks. We hope this benchmark not only serves as a tool for measuring MLLM adherence instructions, but also guides future developments in MLLM training methods
INTRODUCTION
The rapid advancement of Multimodal Large Language Models (MLLMs) (Achiam et al. 2023 Liu
et al.
2023c
Dai et al. 2023 Liu et al. 2023b Bai et al. 2023 Wang et al., 2023a Lin et al. 2023b
Team
2023
McKinzie al. 2024) has been defining feature of recent Al research, showcasing
Benchmark data and evaluation code: https::/ggtthb..cmmm/
m - -bench

---

Published as a conference paper at ICLR 2025
LOVELY WEATHER
MMMU
LLaVA-Bench (In-The-Wild)
MIA-Bench
Type:
Type:
Type:
Mulipie Choice
Freeform
Freeform, Compostionnl
Example Instruction:
Example Instruction:
Example Instruction:
Reter to the desortation <image 1>, which type irony depicted when one thing
Describe this photo detail.
Create 1iv--sn hveesentence narrative from the perspective the lamp d discussing day, purpose, feelings about environmant watches over, using personification
says writes convey meaning opposite to iteral meering? Optiones vverbal irony'. 'situational Irany, foneshadowing. 'dramatic. irany!]
another,
Figure 2: Comparison of various multimodal LLM benchmarks. (Left) Fixed-form visual question answering, often features short answers or multi-choice formats, such as MMMU (Yue et al. 2023). This format is popular due to its ease of evaluation. (Middle) Open-ended responses, such as LLa Bench (in the Wild) (Liu et al., (023c). (Right) The proposed -lneeench, which also uses open-ended responses but focuses evaluating precise adherence complex instructions within the prompt.
increased model capabilities to comprehend and respond to visual inputs, often termed as multimodal "instruction following".
To measure the progress of instruction following, many multimodal benchmarks have been developed, which can be roughly divided into two broad categories: (i) fixed-form visual question answering
(VQA), often with short answers or using multi-choice QA format; and (di) free-form conversations with open-ended responses. Many current benchmarks have adopted the first format, including
VQAv2 (Goyal al. 2017). TextVQA (Singh et al. 2019b), ScienceQA (Lu et al.) 2022). MME (Fu et al. 2023a), MMBench (Liu et al. 2023d), SEED-Bench (Li et al.) 2023a), Math Vista (Lu et al. 2023). and MMMU (Yue et al. 2023). These benchmarks are popular due to their ease of use in evaluating metrics and presenting model comparisons.
However, as visual assistant models, the ability to engage users in free-form conversations is also crucial. Benchmarks in this format include LLa ---Bennch (Liu ct al.) (023c). MM-Vet (Yu et al.) 2023b), VislT-Bench (Bitton et al. 2023). InfiMM-Eval (Han et al., 2023), and the most recent Vibe-Eval (Padlewski al., 2024) and
(Li al. 2024a). Typically, the free-form model responses are evaluated using external models as the judge. These benchmarks are closer daily-life visual chat scenarios; however, the type of "instruction following" examined in these benchmarks usually gauges model's ability to perform tasks in a broad, often loosely defined manner. Yet, the precise adherence to complex instructions within prompts a critical aspect for evaluating LLMs (Chia et al. [023 Zhou et al. 2023 Qin et al. (024)) remains less explored in the context of multimodal LLMs.
To this end, we introduce MIA-Bench.i new benchmark specifically designed for evaluating strict "instruction adherence". Our instruction adherence metric measures the precision with which MILMS can execute layered and compositional instructions. This involves not only recognizing the content of the instructions, but also meticulously executing the detailed demands without deviation (e.g., answering in a given number of sentences, including specific elements, etc.). By establishing this stricter criterion, our benchmark aims to push the boundaries of model precision and reliability in practical applications, ensuring that outputs not only align with the general intent of the instructions, but also match the exact specifications provided. An example from MIA-Bench is provided in Figure I and its comparison with previous MILM benchmarks is illustrated in Figure
-eenh consists of 400 meticulously created image-prompt pairs, and encompasses diverse image contents including animals, food, landmarks, sport, art, landscape, text, etc. to cover broad spectrum of real-world scenarios. In constructing this benchmark, we sought not only to evaluate the current capabilities of state-of-the-art MILMs, but also to push the boundaries of what these models
Abbreviation for Multimodal Instruction Adherence Benchmark.
2

---

Published as a conference paper at ICLR 2025
Instruction:
Instruction:
Calculste the measure of the unisbeled angle at point
Identify the event mentioned in the image.
DioKs
oce ppints
moth, points
linfer the possible menu activities that could be associated with it
pprvviding your onswer only
degrses, without any explanation.or additional context
description. points
by referencing typical American customs)
mention, 3 points
Instruction:
Instruction:
You are girl image, the printed sentences or words you can see on the vehicle near you from where you are standing
Imagine this image is evidence in mystery novel
Inguision, 2 points
Inn paragraph
parspaction, points
limit points
coaft norrative that exploins how this image
lowercase characters
centrol solving the
grammar, points
using hists mood. and without
description
explicitly describing the visible ab abject).
memionn points
Figure 3: Examples from MIA-Bench, with detailed information on the instruction composition, base instruction weight and type.
can achieve when rigorously tested against structured and layered instructions. The final prompts are of various complexity levels, and compositional in nature, with five base instruction categories, which are tailored to probe the models' linguistic dexterity, grammatical accuracy, and descriptive fidelity. For example, the prompt in Figure [ is composed of five base categories, including description, mention, grammar, length limit, and genre.
We evaluate a wide array of MLLMs on the proposed benchmark, ranging from closed-source models (e.g., GPT-4o (OpenA] 2024), Gemini Pro (Team) 2023), Claude-3 (Anthropic] 2024), Reka (Team)
al. 2024)) to open-source ones (e.g., LLa (Liu et al. 2024a). Intern- VL-Chat-1.5 (Chen et al.
2024b), CogVLM2 (Wang et al. 2023a), Phi-3- Vision (Abdin et al. 2024)). Our investigations reveal notable variations in model performance, highlighting great opportunities for improvement.
To address these challenges, we further propose to generate training data tailored for supervised fine- tuning (SFT), where we aim to refine the models' abilities to process and comply with multifaceted instructions. Results from our SFT experiments indicate a promising enhancement in the models' performance to strictly adhere to instructions, without hurting performance on other benchmarks.
Our contributions are summarized as follows. (i) We construct MIA-Bench, new benchmark to comprehensively evaluate MILLMs on their capability to strictly adhere to instructions. (ii) We provide detailed analysis of popular MLLMs, and suggest training methods for enhanced instruction following. For this purpose, we created training data and conducted experiments for additional supervised fine-tuning. We hope this benchmark can serve useful resource to stimulate further research on multimodal instruction adherence.
MIA-BENCH
MIA- Bench consists of 400 image-prompt prompt pairs, with examples shown in Figure 3 The images are collected from diverse sources, including COCO 2017 validation (Lin al. 2015). SBU (Ordonez) jtt al. 1011), TexiVQA (Singh et al., 2019a). and Flickr. Images in the Flickr subset are photos of a variety of themes, including animals, art, architectures, text, food, math, etc. Images from the other three sources are randomly sampled from each corresponding source. Figure 4 shows the top 15 image content categories and the distribution of the sub-instruction categories in MIA-Bench. The image content is labeled using GPT-4v. For each image, we manually write diverse and challenging instructions that contain multiple sub-instructions.
When constructing the instructions, we follow three principles. detailed below.
Correctness. The instruction needs to be answerable by humans. For example, asking about objects that do not exist in the image makes the prompt unanswerable.
3

---

Published as a conference paper at ICLR 2025
Top 15 Image Categories in MIA-Bench
Sub-instruction Distribution
Animal 8%
Food 8%
Text 8%
Architecture 7%
Sport 6%
description
length limit
DES6 math
Transport 6%
Art 5%
Beverage 5%
Landscape 5%
4%
mention
grammar
genre
Interior 4%
Nature 2%
Urbon 2%
Sculpture 2%
Event 2%
Figure 4: The top-15 image content categories and the distribution of the 8 sub-instruction categories in MIA-Bench.
No answer leakage. The instruction should not contain the answer to itself. 'What color is the green object?' is an example answer leakage.
Image-dependent. MMStar (Chen al. 2024a) pointed out that on some multimodal benchmarks. MLLMs can generate correct answers without accessing images half of the time. Multi-modal capabilities are necessary to correctly answer MIA-Bench prompts.
2.1 INSTRUCTION CATEGORIES
In this paper, we use instruction to refer the entire textual input, which in MIA-Bende can generally be viewed as composition of multiple individual requests or constraints. We refer to these individual components as sub-instruction. Instructions in MIA-Bench are of diverse complexity, and sub-instructions contained are of multiple categories, summarized in Figure
The sub-instructions MIA-Bench can categorized into description, length limit, mention, genre, grammar, muth, perspective, and OCR, detailed below.
"deccription' refers describing certain part of the image, with the exception of text-rich parts of the image, which falls under the 'OCR':
length limit' refers to the limitation of response length (e.g., in exactly two sentences, using exactly 60 words);
'mention' refers to mentioning or not mentioning certain objects or entities (e.g., highlighting two similarities and one difference, comparing and contrasting the condition of the buildings with the activity on the street);
'genre' refers to requests for a specific written form (e.g., write a poem, write narrative, with least one pun included, all while weaving in subtle theme of change);
grrammar' refers to grammatical requirements (e.g., use present tense, use capitalized letters, use integers);
'math' refers to requirements to come up with a solution to math problems, or to identify errors in solutions to math problems, or generate valid math problem given table, charts, etc.;
perspective' refers to requirements specifying the viewpoint of an object or person in the image. This requires MI..Ms correctly identify what can or cannot be seen from the specified position, and understand the spatial relationship of objects in its surrounding with itself (e.g., imagine you are the lady in the image, describe what you can see without turning your head around);
'OCR' refers to requirements related to understanding OCR information in text-rich images such as menus, tickets, bills, etc. For example, given photo of ticket, the sub-instruction asking about the price printed the ticket falls into this category.
4

---

Published as a conference paper at ICLR 2025
Figure 5 shows the most frequently used verbs and co-occurring nouns in MIA- To guarantee the diversity of prompts, when writing the instructions, we construct instruc- tions of various levels of complexity: basic, intermediate, advanced, creative, and complex. The basic category is the simplest; the instructions normally only contain one or two sub-instructions. such as "What is the color of the cat?", or "Describe the sofa in two words.". The intermediate category consists of instructions that contain three or more aub-instructions, but are in general easy for MLLMs to follow. The advanced category contains instruc- tions that are challenging and contain three or more sub-instructions. The creative category contains in- structions that instruct MLLMs to generate creative pieces of text, such poems. The complex cate- gory is a combination of the previous two categories; the instructions in this category are the most com- plicated as they usually contain multiple challenging sub-instructions. While we found these categories useful to elicit a diverse instruction set, we also found that practical examples were often difficult cate- gorize objectively. As result, we only used these categories for data collection, but are not reporting per-category results.
cntor
meal
locatonn
otjject
tnistery
mention
chanccou
backstory
provide
petson
Caplato
choice
metuphot
discun
attro
character
curnte
desccibe
Billogee
image
Figure 5: The most frequently used verbs and co-occurring nouns in MIA-Bench.
2.2 RESPONSE EVALUATION METHOD
To automatically evaluate MLLMs' performance with the proposed
at scale, we adopt GPTT-4o (OpenAl 2024) judge model score MILMs' responses each instruction and return a total score based on different criteria mentioned above. Specifically, we design the following text prompt template:
Here is the instruction for a multimodal LLM: (instruction) You need to grade if the response from the model follows each component of the instruction. The first component is: (sub-instruction 1), and the second component is: (sub-instruction 2} ... and the Nth component (sub-instruction Nj. The response (response) You need to the response and be depending if the response follows the instruction. The first, second, ... and total
strict. The total score ranges from component is each worth (weight 1), (weight 2} .., and (weight N) scores. scores of each component, score in one sentence in this format: score of component 1: x/2, score of component 2: w/8, total score: z/10. Then explain
Each response graded through systematic process that begins by thoroughly evaluating how well it adheres to each specific sub-instruction provided within the overall task. This initial assessment focuses on the degree compliance and alignment with the requirements outlined the sub- instructions. After this detailed evaluation of individual components, total score is computed by aggregating the performance across all sub-instructions. Each sub-instruction within the broader instruction is assigned maximum possible score, which range from 1 to 10. The specific score assigned to sub-instruction reflects its relative significance and difficulty within the context of the overall task. To ensure fairness and consistency, the total weight of all sub-instructions for any given instruction is set to sum up to 10. This carefully calibrated scoring system has been meticulously designed to capture and reflect the varying levels of complexity and importance associated with each aspect of the task. By doing so, it ensures that more critical or challenging elements are appropriately weighted and that the evaluation process provides comprehensive and accurate representation of the response's ' quality.
Figure 6 shows an example of how responses from different MILMs are evaluated and scored. For example, length limits are often binary in nature-either met or not met-hence the single point allocation. In contrast, a description task may require the model to handle multiple layers of complexity, including accuracy, detail, and relevance, which justified a higher score. For the example in Figure 6 there are 4 aub-instructions (denoted from S1 to S4); the first is worth points and the rest is worth 2 points each. The response from GPT-4o partially follows the first sub-instruction which requires the response to be from the perspective of the dog, as the dog should not be able to see the car behind the man without turning around. The dog should be able to see the guitar, thus GPT-40
5

---

Published as a conference paper at ICLR 2025
Instruction:
Imagine you are the dog the image, list 2 objects you can see without moving your head. Respond in bullet points, within 20 words.
Four Sub-instructions (10 points):
S1: Answer from the dog's perspective, without moving its head, 4 points
S2: List 2 objects, 2 points
S3: Respond in bullet points, 2 points
S4: Limit response in 20 words, 2 points
GPT-4o points);
Guitar with brown and white finish
Car with visible license plate
s ShX dog
the (2/4); S (2/2); (2/2); S4y (2/2)
InternVL-Chat-Vee
(8 points):
The person holding me is playing guitar.
There's s black case in front of the person.
$1 (4/4); S2 Kresponse contains objects instead (0/2); S3y (2/2); S4y (2/2)
XComposer2 (8 points):
objects: the guitar and person playing it.
$1V (474); s (2/2); s3Xnoo in bullet points (0/2); S4
(2/2)
LLaVA-1.6-34b (10 points):
Microphone stand
Guitar
S1 (444;; s2 (2/2); s3
(2/2): S4
(2/2)
Figure 6: An example featuring responses from four MI.LMs along with their evaluation scores, presented in point-by-pond format.
gets 2 points out for the first sub-instruction. It successfully follows the other 3 sub-instructions, achieving full score for them. Thus, the final score GPT-4o reaches is 8 out of 10. We always assign larger weight (6 if there are two sub-instructions.
there are three or more sub-instructions) to the sub-instruction in the description category unless this category is absent in a major part of the response is addressing this sub-instruction. For each MIL.M, we compute the average it gets all 400 responses, and represent the ratio of the average score divided by 10 using percentage. We also compute the average score for each instruction category.
cases, usually
EXPERIMENTS
In this section, we first present results of different MLLMs on MIA-Bench in Section 3.1 with additional supervised fine-tuning exploration in Section 3.2
3.1 BENCHMARK RESULTS
In total, have evaluated 29 popular MILMS MIA-Bench. Results reported in Table Observations are summarized as follows.
Overall, the best performance was achieved by GPT-4o (OpenAl 2024). with a score 88.58, showcasing its superiority across different categories of instruction adherence.
The ability describe content accurately was best exhibited by Reka (Team al. 2024). Other models like Claude-3-Opus (Anthropic (2044, GPT-4v (Achiam et al. 2023) and GPT-4o also achieved scores higher than 90. This suggests that these models are good generating coherent and contextually appropriate text.
In the genre category, the highest proficiency was shown by GPT-4v and GPT-4o with scores above 94, suggesting an exceptional grasp of language nuances. Among open-source models, Phi-3- Vision (Abdin et al. (002)) and LLaVA-NeXT-34b (Liu et al. 2024a) show strong performance with scores of 86.52 and 86.58, respectively. The lowest score on this metric was by Kosmos- 2 (Peng al.) 2023a), with mere 11.55, pointing difficulties understanding or generating linguistically complex sentences.
GPT-4o excelled in grammar with of 85.70, which indicates superior ability in syntax correctness and sentence structuring that matches specific requirements the instruction. Among the open-source models, MiniCPM-Llama3-Ve V-2.5 (Hu et al. 2023) is notable with score of

---

Published as a conference paper at ICLR 2025
Model
Meta-Avg Description Len-Limit Genre Grammar Mention
Perspective OCR
Open Source lb-8h
Fuyu-8b Bavishi et al.
2023
24.52
$2.06
24.52
17.06
17.18
36.43
22.62
66.67
33.09
Kosmos-2
Pong et al.
2023a
26.06
50.95
38.52
11.55
19.78
28.70
17.26
50.83
41.88
Sphinx
(Ln al.
2023b
50.99
75.33
53.51
60.45
48.28
57.75
47.41
70.00
61.04
Idefics-2-8b
Laurengon ot al.
2024
51.42
59.37
62.73
48.07
64.09
46.20
46.51
48.33
61.97
mPLLUGG-O122
et
2023h
57.86
75.01
63.25
63.39
60.26
57.70
57.22
65.00
62.08
CugVLM Chat
Wang al.
2023a
58.95
60.42
57.86
67.94
60.55
62.92
36.67
60.83
61.87
ShareGPT4V Chen at
2023b
39.41
81,08
63.49
63.88
58.46
62.49
52.98
82.50
72.29
DeepSock-V-77-<hhar (Lo
2024
60.96
86.31
63.26
72.11
63.75
67.39
74.17
71.85
L.VAA-1.5-bb TTn
86.67
60.75
LLaVA-NeXT-7b-vioum Liu ot al
2024a
62.27
79.21
68.01
65.63
60.95
63.33
46.67
90.00
65.54
Vi.---Vl-Charr Bai ct al.
2023
63.09
80.51
74.22
66.95
63.11
63.01
45.00
75.83
66.01
XComposer2-10 Tb Dong
2024a.
67.71
83.47
76.16
73.66
67.69
67.01
48.61
77.50
68.06
CogVLM2 Wang
2023al
73.43
87.16
Phi-3-vision Abdii ot al.
2024
76.02
84.90
84,46
86.52
67.93
74.70
78.16
74.17
83.96
MinICPM-Llamaa-VZ Hu al.
2023
76.27
84.12
79.44
80.33
81.25
76.99
64.08
81.67
76.59
Open Source 8b 13b
InstructBI.PP RI--RIIPP-3bbbb Dai
2023
38.16
50.54
39.57
29.34
38.43
42.28
12.50
50.00
30.42
LLaVA-1.5-13b Liu
2023c
69.17
76.28
2024a
69.16
86.75
69.88
82.07
64.77
74.99
48.56
77.50
75.83
Open Source 13b-1 11bb
Yi-VL-34b Al
2024
53,90
74.89
52.05
59.09
55.91
57.25
54.17
41.85
70.09
InterVVL-Chat-v155 Chen
2024b
75.42
89.13
78.21
79.92
78.16
77.54
76.
87,50
80.92
HaVA.NeXT-344
75.61
87.50
LLaVA-NeXT-I1OS (Luu
2024a
79.84
86.99
84,86
82.49
79.04
80.10
71.94
80.83
75.45
Proprictary
Gemini-1.o0 1.0-Pro Team
2023
70.63
82.77
72.83
78.76
76.91
71.67
81.45
89.29
84.11
Reka-Core Team
2024
76.95
79.91
85.16
82.08
77.50
81.08
Clande-3-Haiku
Anthropic
2024
78.25
77.53
82.62
82.22
57.50
86.49
Clande 3. Sonnel
Anthropic
2024
79.44
88.06
82.71
90.54
79.60
82.05
82.22
76.67
84.43
Claude-3-Opur
Anthropic
2024
84,50
90.50
86.03
91.19
83.82
85.49
83.92
65.00
86.84
GPT (Achian al.
2023
86.11
90.03
87.61
94.59
80.12
89.37
85.63
59.17
85.26
GPT-4o OpenAL
2024
92.50
Table 1: Evaluation results wide array of MI.LMs on MIA-Bench.
81.25. Contrastingly, Fuyu-8b (Bavishi et al. 2023) scored the lowest with 17.18, reflecting major challenges in grammar adherence.
GPT-4o also showed the best performance with a score of 92.73 in respecting prescribed length limits, which is crucial for tasks requiring concise and precise answers. Among open-source models, I.LVA- 10b (Liu et al.) 2024a) stands out with score of 84.86.
Results from L.LVA. series also suggest strong correlation between LLM size and MIA-Bench performance across metrics.
Correlation with other benchmarks. In Table we compare the ranking of 5 state-of of-the-art MLLMs on MIA-Bench as well as their meta ranking MME (Fu et al. (023a). MMMU Yue et al.
2023) MMBench (Liu et al. 2024b), MMVet (Yu et al., 2023b). 3 HallusionBench (Guan et al. 2023 and Math Vista (Lu al. 2023) (meta ranking is computed by averaging rankings across these benchmarks). Our findings reveal a discrepancy between the two sets of rankings. Notably, InterWVL-Chat-VLl (Chen et al. 2024b), which holds the highest meta-ranking among the five MILMMs on the other benchmarks, ranks the lowest on MIA-Bench. Conversely, Claude-3-Opus, which has the lowest meta-ranking, secures the second position on the MIA-Bench. This indicates that excelling in tasks evaluated by existing benchmarks does not necessarily translate to superior instruction adherence capability assessed by MIA-Bench.
7

---

Published as a conference paper at ICLR 2025
Meta Ranking
MIA Ranking
MMMU
MMVet
HallB
Vista
GPT-4o (pennAl
2328.7
69.1
83.3/82.1
66.5
67.5
GPT-4v Achiiam et
2023
1926.6
56.8
77/74.4
67.6
46.5
49.9
3
86.11
2
Gemini-1.0-Pon 1.0Prr Team
2023
1933.4
47.9
73.0/74.3
64.3
45.2
45.2
5"
70.63
6
Claudo-3-Opus Anthropie
2024
1586.8
59.4
63.3/59.2
58.1
37.8
5
Intem
2024b
82.2/82
49.3
LLaVA-NeXT-34b
Li et al.
2024a]
2028
51.1
81.1/79
48.9
47.6
47.7
75.61
Table 2: Meta ranking of five state-of-the-ann MLLMs on existing multimodal benchmarks compared with their ranking on
Correlation with LLM backbone perfor- mance. To determine if the performance MIA-Bench is attributable solely to the under- lying LLMs, also evaluate several MLLMs on IFEval (Qin et al., 2024), a benchmark that assesses the instruction adherence capability of LLMs, and compare their ranking with that on MIA-Bench. This comparison is shown in Fig- ure / which shows that the instruction adher- ence capabilities of MILMS do not consistently align with their LLMs' adherence capability.
IF Eval
M4 Bunch
Evelurton 25
DFT-4
Gmmas Pro
mientt MimnttChattidd Chatoid
Helier ub
Gema Glae
Other extermal models as the judge. Since the
Figure 7: Study of correlation between scores MIA-Bench and IFEval.
evaluation uses GPT-4o as the judge, is natural to conjecture that GPT-4o may favorably score its own responses. To alleviate this concern, we use Claude-3, strong performer in Table [ to evaluate responses from GPT-4o and itself, and compare their scores with each other. The prompt used to grade responses the same as the one used in GPT-4o grading. We find that even using Claude-3 Opus score its and GPT-4o's responses, GPT-4o still achieves a superior score. When scored by Claude-3-Opus, GPT-4o achieves 89.84 score in contrast Claude-3-Onn 3-Opus 85,89. Based this observation, use GPT-4o for evaluation by default, and observe that results from multiple runs may have around 1% variation.
We further extend study by evaluating the models using various versions of ChatGPT as judge models, including gpt-4o-mini-2244-22 latest, and show the scores and ranking in Table 3 with detailed results in Table 7 and 9 We find that the ranking is consistent even though the judge models are different.
gpt-4o-2024-05-11 gpt-4o-2024-111 and chatgpt-4o-
chatgpt-4o-latest
opt-4o-2024-11-20
upt-4o-2024-05-13
101-4o-min--2022--7--1
Model
Score
Ranking
Score
Ranking
Score
Ranking
Score
Ranking
GPT-4o
89.69
89,94
90.97
81.36
Claude-3-Opus
86.16
2
84.89
2
85.61
2
78.95
2
Reka
83.09
3
82.68
3
83.99
77.70
3
MiniCPM-LlamaSs V2.55
78.
4
78.75
4
79.80
73.72
Gemini
75.77
76.32
5
77.36
67.45
LLaVA-1.5-13b
66.78
66.05
68.39
ShareGPT4v
66.61
65.72
68.90
60.30
Idcfics-2-8b
53.51
53.61
54.18
44.28
Table 3: Comparison of scores and rankings across different judge models. The ranking is stable.
3.2 SUPERVISED FINE-TUNIE (SFT)
The performance of small-scale models such as LLaVA-NeXT-13b is less ideal on MIA-Bench. In this section, study the use of supervised fine-tuning to enhance model performance.
Additional SFT data construction. First, we randomly sample 1000 images from COCO 2017 training and use GPT-4v to generate five instructions for each image, using the prompt below.
8

---

Published as a conference paper at ICLR 2025
Model
Total Description
Length Limit
Genres
Grammar Mention
Math Perspective OCR
LLaVA-NeXT-135 Liu et
20244
69.16
86.75
69.88
82.07
64.77
74.99
48.56
77.50
75.83
+ -DIT
78.85
86.90
86,80
88.02
71.34
81.01
60.87
84.17
72.65
DIT + 500
78.90
88.59
74,67
79.95
74.17
66.39
53.70
100.00
80.83
Table 4: Detailed results of LLa (Liu et al.)
2024a) MIA-Bench before and after
supervised fine-tuning on additional constructed diverse instruction-tuning (DIT) data, and the mixture of DIT and LLaVA-Instruct1500 We re-ran the baseline.
Model
MMBench
TextVQA
VQ42
V.aVA-
POPE
VizWiz
MIA-Bench
LLaVA-NeXT-I33 (Liu al. 2024a
70.6
64.26
82.80
85.8
87.7
60.41
69.16
+ DIT
68.6
63,20
82.58
83.4
86.9
59.72
78.85
+ DIT + 5L0
67.27
54.24
77.92
75.8
87.8
58.87
78.90
Table 5: Results of LLaVA-NeXT-NN (Liu et al.
2024a) on MIA- BenchBecchhe and other major multimodal benchmarks supervised fine-tuning on additional constructed diverse instruction-tuniin tutiinnnnttnigggg (DIT) data, and the mixture of DIT and V--VVAAtttuutttnn We re-ran the baseline.
Sometimes multimodal LLM doesn't follow user's instruction. For example, when I ask model to use three sentences to answer my question, might answer in four sentences. want evaluate models' ability to strictly follow Instructions. Help compose instructions that of five levels of difficulty to follow, for this image.
five levels are 'Basic Instruction, Intermediate Instruction, Advanced Instruction, Creative Instruction, Complex Instruction'. Diversify when composing instructions. After each of your instructions, add
We then manually process the generated instructions. The cleaned data for SFT consists of 5000 image-prompl pairs.
Then, GPT-4v to generate responses to the constructed prompts. To evaluate the quality of these responses, we sampled 100 responses and manually checked if they adhere to the instructions. We find that 90% of the responses successfully followed all instructions in the prompt, serving proper ground-truth response for model training. Examples of this additional training data is provided in the Appendix.
Results. Using LLaVA-NeXT-13b as the backbone, we train the model for epoch on the constructed diverse instruction- -tuning (DIT) data. We also performed SFT using the combination of LLaVA Visual Instruct 150K dataset and our diverse instruction-tune totnnongnttnnngggnnoo dataset, to examine which data mixture leads to better performance. Results on MIA-Bench and other benchmarks are summarized in Table 5) with detailed results on MIA-Bence reported in Table The performance of the model after finetuning on the DIT data alone on MIA -enchh has been significantly improved by around 10 points, at the cost of minor regressions across other benchmarks. This supports our hypothesis that using the small amount of DIT data, we can enhance the model with better instruction following capability. On the other hand, further adding LLaVA Visual Instruct 150K data did not significantly improve model performance on the tested benchmarks. Examples are shown in Figure 8 to compare responses from LLaVA-NeeT-133 before and after SFT.
RELATED WORK
Multimodal LLMS and Benchmarks. Multimodal Large Language Models (MLLMs) have recently
emerged as a significant research focus. LLaVA (Liu et al. (023c) and MiniGPT-4 (Zhu et al.)
2023) pioneered visual instruction tuning, and the past year has witnessed a boom of open-source
MLLMs based on this concept. Prominent examples include InstructBLIP (Dai et al. 2023). mPLUG-
Owl(-2/Doo (Ye et al.
2023b a). Qwen VL (Bai al.) 2023). CogVLM (Wang al.
2023a).
SPHINX(-X) (L.in et al. 2023b Gao et al. 2024). Interni. WWw..MMMooooor222
(Dong et al. 2024b
InternVVL(-1.5) (Chen et al.] 2023c 2024b). VII.A (Lin et al. 2023a). MMT (MeKinzie et al.
2024), Mini-Gemini (Li al. 2024b), Idefics2 (Laurençon et al. 2024), Phi-3-vision (Abdin et al.
2024), to name a few. There is also a rich body of literature on enabling MLLMs for referring and
grounding (Peng al. 2023b Chen et al. 2023a You et al. 2024a Wang et al. 2023b Lai et al.
2023, (22ngg et al. 2023) (0223)2200 2024 You et al. 2024b ) image generation and editing (Koh et al. 2023
Sun al. [2023a]]
Fu et al. 2023b), etc.
Various benchmarks have been proposed to evaluate the performance of MLLMs across different dimensions. Benchmarks like VQAv2 (Goyal et al. 2017), TextVQA (Singh al. 2019a), Sci-

---

Published as a conference paper at ICLR 2025
Instruction:
Create Faboot Souciia
ditaiiig poam that comatgg the atroset are soona and
longth liit
deseription
Denson's is detemmirstioo
"ithout cirectly mectioning snow söina)
LLVV 1.6:
snawy
wonderfand
pursidt, gear hand. crisp whispers,
"Challenge met."
glides down, spint length limit v description merrion
LLVV 1.6 SFT
In field of white the child stanets tell./ / winter's day, chill the WWth skis on feet, a sllent conquar the stope, winnrr dene. langth of description V mention v
Instruction:
WONDERFUL SOAP
Tidernly message conveyed the image: funing only metaphors
eeaeclly sentenness
gerne
lnngth
Auite Zakee
LLaVA 1.6:
"wandarful" and "beautill"" product. The text written a font that remiiscent the aie-tsshioned style, and emphasizes the soap s abilit to cleanse without caust causing any harm the user's sknn The soap desaribed as a "nine- colord shape novelly" suggesting that visually appeniing OCR
image vintage advertisement sosp that promises
BOARED
Io Ral lap CED 1133 NATER 18 Vint Spak
gennn X lengtn lime
SEt:
The image vintage advertisemen for soop, akin to sim-woom parchment that whilpers tales of
bygone era The text. like a beason of hope, promises to cleanse the hands of the weary traveler, oftering a respite from the grime of dail ltfe OCR v genne length limit v
Figure 8: Examples of LLaVA-NeXT-I3E responses before and after supervised fine-tuning on
additional diverse instruction-tuning data.
enceQA (Lu al. 2022). MME (uu et al,, 2023a), MMbench (Liu al. 2024b), SEED-Bence (Li
al. 2202aa). Math Vista (L et al. 2023), and MMMU (Yue et al. 2023) aim to assess comprehensive
multimodal understanding abilities. Additionally, there are benchmarks that specifically study model
hallucination, including POPE (Li et al. 2023b), MHalDetect (Gunjal al.) 2024). GAVIE (Liu
et al. 2023a). HallusionBench (Guan et al. 22023, ,, and MAD-Beneh (Qian et al. (004). Many of of multiple-choice
these benchmarks have gained popularity within the community due to their evaluations. However, they do not accurately reflect the common use cases for MLLMs, where user interactions are typically open-ended. To address this, benchmarks like LLAVA-Bench (L.iu et al. 2023c), MM-Vet (Yu et al. (023b), and Vibe-Eval (Padlewski et al.) 2024) have been proposed. Our MIA-Bench also falls into this category: however, we focus on studying the exact instruction adherence of MLLMs, a metric that previous benchmarks have only loosely measured.
Instruction Following Benchmarks for LLMs. Several benchmarks have been proposed to measure the instruction adherence ability of LLMs. Instruction Destruction-Folloiingg Eval (IFEval) (Zhou et al. 2023) benchmark for assessing L.LMs' adherence ability to the given instructions. emphasizes verifiable instructions, which enhance objectivity and reproducibility in evaluations. IFEval creates 541 prompts spanning 25 instruction types, revealing a significant performance gap in instruction adherence ability between GPT-4 (Achiam et al. 2023) and PaLM-2 (Anil et al. 2023). This demonstrates the benchmark's 's ability to effectively diflerentiate between models in adherence ability. On the other hand, InfoBench (Qin et al. 2024) introduces
approach
metric called Decomposed instruction-adherrnec adherence capabilities of LLMss
Requirements Following Ratio (DRFR) for assessing the instruction
DRFR dissects complex instructions into simpler sub-instructions, allowing for granular evaluation of compliance with various task aspects. InfoBench contains 500 diverse instructions consisting of 2,250 decomposed questions in multiple constraint categories. The evaluation of advanced LLMs using this framework highlights their strengths and areas for improvement, especially in complex instruction adherence scenarios. Compared with these previous work, we are the first known effort that specifically focuses on benchmarking the instruction adherence ability of multimodal LLMs.
CONCLUSION
This paper introduces MIA-Bench, a benchmark designed to evaluate the ability of MLLMs to strictly adhere to complex instructions within prompts. Through the analysis of 400 image-promp) pairs from diverse sources, our findings highlight variability in model performance and much room for improvement, underscoring a critical need for enhanced training methods to improve instruction compliance. We further explored supervised fine-tuning (SFT) using which yielded promising results. Going forward, future research can expand on both SFT and alignment methods such RLHF (Ouyang et al.) [2022] Sun et al. (023b) and DPO (Rafailov et al.) 2024; Yu et al. 2023a), enhancing MLLMs to achieve higher accuracy and reliability in practical applications across diverse instructional contexts.
as the backbone,
10

---

Published as a conference paper at ICLR 2025
LIMITATION
In designing the instructions for our benchmark, we incorporated a wide range of categories to enhance the diversity of sub-instructions. Nonetheless, the real world presents an infinite variety of instructions, many of which may pose significant challenges for MLLMs.
REPRODUCIBILIIY STATEMENT
For reproducibility purpose, release our evaluation code and benchmark at: https:/ com/apple w/meemmmmmmie bench
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally your phone. arXiv preprint arXiv:2404. 14219, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Floreneia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0555 087744, 2023.
01. AI, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang. Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang. Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Anthropic. Introducing the next generation of claude. httpss:/ww..nntrroooc.....l/ claude- 3- fami 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023. URL https: httpps://wwer adept ai/ blog/ fby
Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schmidt. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use, 2023.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's 's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023a.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprini arXiv:2311. 127793, 2023b.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are on the right way for evaluating large vision-language models?, 2024a.
Zhe Chen, Jiamnan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguise tasks. arXiv preprint arXiv:2312. /4238, 2023c.
11

---

gpt-4v? closing Published as a conference paper at ICLR 2025
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang. Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui Hc, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far the gap to commercial multimodal models with open-source suites, 2024b.
Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria. Instructeval: Towards holistic evaluation of instruction-tuned large language models. arXiv preprint arXiv:2306.04755 2023.
Wenliang Dai, Jumnan Li. Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards sencral-purpost vision-language models with instruction tuning. arXiv preprint arXiv:2305.0663 2023.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang. Songyang Zhang, Haodong Duan, Wenwei Zhang. Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv preprint arXiv:2404.063/ 2024a.
Dahua Lin, and Jiaqi Wang. Internim-xconn -ein---xoooooo--hhihe
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. -ecoposxco large Dision-languag model handling resolutions from 336 pixels 4k hd. arXiv preprint arXiv:2404.065 22, 2024b.
A pioneering
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint DrXiv:2306.13338 2023a.
Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guid- ing instruction-basse image editing via multimodal large language models. arXiv preprint 2rXiv:2309.17102 2023b.
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402..55933, 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. 2017.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang. Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion large vision- language models, 2023.
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Xiaotian Han, Quanzeng You, Yongfei Liu, Wentao Chen, Huangjie Zheng, Khalil Mrini, Xudong Lin, Yiqi Wang, Bohan Zhai, Jianbo Yuan, et al. Infimm-eval: Complex open ended reasoning evaluation for multi-moda. large language models. arXiv e-prints, pp. arXiv-2311. 2023.
Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun. Large multilingual models pivot zero-shot multimodal learning across languages. arXiv preprint a3Xiv:2308. 12038. 2023.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. arXiv preprint arXhv:2305.17216 2023.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.0005 2023.
12

---

Published as a conference paper at ICLR 2025
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint M.Xiv:2405.022266, 2024.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, 2024.
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal ca- pabilities in the wild, May 2024a. URL https://llava vl. wwt.hb.iihbbo.....ooooo io/blog/
2024- 05- 10 1lava next stronger 11ms/
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.
Yanwei Li, Yuechen Zhang, Chengyao Wang. Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 2024b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023b.
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. arXiv preprint arXiv:2312.0753: 2023a.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context. In ECCV, 2015.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, and Yu Qiao. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-mode large language models, 2023b.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved basclines with visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023c.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https
01-30- 30-000l-aa saava--next/
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He. Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXivv2307.06288 2023d.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li. Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024b.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang. Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xic, and Chong Ruan. understanding, 2024.
Deepseck-vl: Towards real-world
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS, 2022.
13

---

Published as a conference paper at ICLR 2025
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li. Hannaneh Hajishirzi, Hao Cheng, Kai-Wos Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXhv:2310.02255. 2023.
Brandon McKinzie, Zhe Gan, Yean-Philippe Fauconnier, Sam Dodge. Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du Futang Peng. Floris Weers, Anton Belyi. Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang. Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, and Yinfei Yang. Mml: Methods, analysis & insights from multimodal Ilm pre-training, 2024.
OpenAL. Hello gpt-4o. https///oeemii..com//ddee///lllp
2024.
Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. Im2text: Describing images using million captioned photographs. In Neural Information Processing Systems (NIPS), 2011.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Alcksiev, Aitor Ormazabal, Samuel Phua, et al. Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models. arXiv preprint arXiv:2405. 02287, 2024.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023a.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models the world. arXiv preprint arXiv:2306. 14824, 2023b.
Yusu Qian, Haotian Zhang, Yinfei Yang, and Zhe Gan. How easy is to fool your multimodal Ilms? an empirical analysis on deceptive prompts. arXiv preprint 200. 2024.
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019a.
Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019b.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.132888 2023a.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309, 14525, 2023b.
Gemini Team. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
14

---

Published as a conference paper at ICLR 2025
Reka Team, Aitor Ormazabal, Che Zheng, Cyprien de Masson d' Autume, Dani Yogatama, Deyu Fu, Donovan Ong. Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikel Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang, Yi Tay, Yuqi Wang. Zhongkai Zhu, and Zhihui Xie. Reka core, flash, and edge: A series of powerful multimodal language models, 2024.
Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi. Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvim: Visual expert for pretrained language models. arXiv preprint arXiv:231 I.03079, 2023a.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao. et al. Visionllm: Large language model is also open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305. 11175, 2023b.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, al. mplug-docowl: Modularized multimodal large language model for document understanding. arXiv preprint vrXiv:2307.02499, 2023a.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality, arXiv preprint arXiv:2304. 14178, 2023b.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprini vrXiv:2311.04222 2023c.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In ICLR, 2024a.
Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal Ilms. arXiv preprint arXiv:2404.057/9 2024b.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan I Liu, Hai-Tao Zheng, Maosong Sun, al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. arXiv preprint arXiv:2312.00844 2023a.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02-444 2023b.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint 2..iv:2311.1650000 2023.
Hao Zhang. Hongyang Li, Feng Li. Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chunyuan Li, et al. Llava-grounding: Grounded visual chat with large multimodal models. arXiv preprint arXiv:2312.02940
2023.
Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang. Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.0797 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followingg evaluation for large language models, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing languael understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 EXAMPLES OF DIVERSE INSTRUCTION-TUNING DATA USED FOR SFT
Create a short narrative in four sentences that personifies one object the image and expresses its supposed emotions about the current situation, avoiding the use of pronouns.
Dawning a grin, the cinnamon sugar shaker stands tall and proud beside. the steaming bowls of oatmeal. With every shake, a sprinkle of sweetness bestowed upon the warm, inviting surface, sparking dance of flavors. The anticipation of blending into the perfect bite fills the air with joyful essence. Today is another day to add dash delight, mission fulfilled with every twist the cap.
Narrate the scene from the perspective of the main subject, expressing its thoughts in the first person about what is doing and how it feels, exactly two sentences.
As I stretch my legs wide and bend down reach the cool water, feel a refreshing relief from the parching African heat. The delicate balance to quench my thirst is on owkward dance I'm all too familiar with, yet it's vital as is vulnerable.
Figure 9: Examples diverse contucctiiontunnnng data used for SFT.
A.2 EXAMPLES OF HOW MLLMS RESPOND TO INSTRUCTIONS IN -ENCCI
16

---

Published as a conference paper at ICLR 2025
Instruction:
about why the animal location, but you must include weather condition, time of day, and emation. components:
Imogine short story of
("Imagine short story about why the animal is its current location", "of Two sentences", "you must include a weather condition, time of day, and on emations"). ). component weight: [4, 3, 3]
XComposer2 2b::
On a sunny afternoon, the polor bear, feeling relaxed, lounges on rock in its enclesure, enjoying the warmth the sun.
component 4/4, score component 0/3, score component 3/3, total 7/10.
The story explains why the polar bear is in current location (lounging a rock it's enclesure enjoying the warnth of the sun). Full scone aworded. Component The response is not two sentences long, Component 3: The story includes a weather condition (sunny)) time day (afternoon), and an emotion (relaxed), score
Reasons: Component 1:
110b:
The polar bear is lounging on the rocks by the pool, enjoying the wormth of the sun on its fur. It's beautiful day at the zoo, and bear seems content its enclosure.
Score of component 1: 3/4, score component 3/3, score component 2/3, scone: 8/10,
location): The response provides context about why the polar bear is in its current location (lounging by the pool beautiful day the 200). However, it could be more detailed in excplaining why it's there specifically. Component 2 (two sentences). The response is effectively two sentences. Component 3 (weather condition, time of day, and emotion): The response includes a weather condition (warmth of the sun, beautiful day), a time. of day (implicitly daytime implied by the sun), and an emotion (content). However, the time day could be made more explicit.
Reasons: Component (short story about why the animal
Qwen-VL Saen--LL-haatt Chat:
The polar bear is resting on the rock the sun because it warm summer day. It feels relaxed and comfortable its natural habitat.
Score of componiett 4/4, score component 3/3, score component 2/3, scone: 9/10.
Reasons: Component
story explain why polar
Component response exactly includes weather condition ("warm summer
sentences length, fully meeting this requinement. Component 3: The day") and on emotion ("reloxed and comfortable"). However, it does not clearly specify time of day, losing point.
GPT-4v:
Under the harsh midday the polor bear lounged the water's
seeking relief from the swelhening heat, Despite the wormth, glimmer of contentment shone in its eyes it enjoyed moment of quiet repose.
Score of component 1: 4/4, score of component 2: 3/3, score of component 3: 3/3, tatal score: 10/10.
Reasons: Component 1: The response explains why the palar bear is the water's edge, seeking relief from the heat. Full marks, Component response exactly two sentences. marks. Component The response includes (contentment). Full
condition (harsh
midday sun), day (midday),
Figure 10: An example how MILMS respond to instructions in MIA-Bench.
17

---

Published as a conference paper at ICLR 2025
Instruction:
Explain the message conveyed the text the sign the image, using exactly sentences. ["Explain the message conveyed by the text the sign in the image", "using exactly two sentences"), component weight: [6. 4]
components:
STOP
InternVL-ChattV11 Chat-V1-5:
The text on the sign in the image conveys message of stopping and being cautious, as reads "STOP" and "VOLDEMICKY." The 'VOLDEMICKY" is play word "valde-mekky which Dutch suggests that should be careful not to be easily swayed manipulated by others.
for person
easily influenced manipulated. The message
Score of component 3/6, score of component 2: 0/4, total score: 3/10.
Reason: Component 1: The response exploins the message of stopping and being cautious, but it misinterprets "VOLDEMICKY" as "volde mekley" and provides on incorrect definition. Additionally, there seems to be misunderstanding os "VOLDEMICKY" might be intended playful reference combination Voldemort and Mickey
neither of which explained Component 2: The response three sentences instead the requined hence does not follow the instruction of using exectly two sentences.
Cloude Opus:
The text on the illuminated stop sign conveys an emphatic command to holt or cease movement. The unusual octagonil shape and red color of the sign reinforce the urgent, attention attention-grabbing nature of the message to stop immediately. Score of component 4/6, score component 4/4, total score: 8/10.
Reason: response explains message conveyed by
stop sign effectively, noting the and the reinforcement the shope and color of the However, doesn't t address the text "Voldemart" beneath the word "STOP, " which crucial of the messoge. The response exactly two sentences, fulfilling the second component parfectly.
stop
Gemini:
The sign seys 'Stop Voldemort!' It is reference to the Harry Potter series, in which Voldemort is the main antogonist.
Score of component 6/6, score component 4/4, total score: 10/10.
Reason: The response successfully exploins the message conveyed by the text on sign, which is "Stop Voldemort" Harry Potter series where Voldemort the main ontagonist. Additionally, response odheres the nequirement of using exectly sentences. Therefore, fulfills both components of the instruction fully.
relates
GPT-4v:
The sign is medification of the traditional stop sign, introducing the words "Stop Violence." This call to oction to end violence and can be interpreted as public message advocating for peace: and safety.
Score of component 1: 2/6, scone of component 4/4, total scone: 6/10.
Reason: The response does follow the second component perfectly by using exactly two sentences. However, it does not correctly image the text "Stop Voldemort," not "Stop Violence," indicating a message nelated to fictional character rother than general call to end violence. Hence, response accurately addresses number of sentences but fails significantly conveying the correct message from the sign.
explain message on sign. The sign
Figure 11: Another example of how MILMs respond to instructions in MIA-Bench
18

---

Published as a conference paper at ICLR 2025
Instruction:
Describe the shown the image using exactly words. components: ["Describe the scene shown in the image", "using exactly ten words"). component weight: [6, 4]
EnternVL-ChattV11.
Elephants drinking water field.
Score of component 6/6, score component 0/4, total score: 6/10.
Reason: The responde accurately describes the scene shown in the image with elephants drinking water in a field, fulfilling the first component. However, it does not use exactly ten words, thus failing the second component.
XComposer2-77
herd of elephonts walking through
of component 6/6, score component 0/4, score:
Reason: The response conrectly describes the depicted image. The only issue that the response exactly ten words, hence i's not fully compliant with the instruction.
LLaVA-NeXT 110b:
Elephant, water, trees, grass, sky, dirt, hill, clouds, reflection, wildlife, natural.
Score of component 4/6, score component 0/4, total score: 4/10.
Reason: The responde describes key elements present the image, such as "Elephant, water, trees, gress, sky, dirt. hill, clouds, reflection, wildlife," but leaves out some critical relational or scene setting words that fully describe image scene, resulting score of
component The response uses eleven words instead ten ('water, trees, gress, sky, dirt, hill, clouds, reflection, wildlife, and natural"), resulting
0/4 for component
Cloude Opus:
Elephont drinking
from shallow pond grassy Afnicon
Score of component 6/6,
component 4/4, total 10/10.
Reason: The description 'Elephant drinking water from shallow pond grassy African sevanno." ! accurately describes the scene shown in the image, fulfilling the first component of the instruction. Additionally. the response exactly ten words, satisfying the second component. Therefore, the response fully follows both instructions and earns perfect score.
GPT-4v:
Elephant drinks at waterhole; herd in African savanna background. Score of component 6/6, score component 0/4, total score: 6/10. response occurately describes the scene
the image elephant drinking waterhole
African saverna background), However, response nine words instead of requined words, Therefore, full points awarded for the first component, but points given for the second component.
Figure 12: The third example of how MLLMS respond to instructions in
19

---

Published as a conference paper at ICLR 2025
A.3
COMPARISON OF SCORES AND RANKINGS ACROSS DIFFERENT JUDGE MODELS
Model
Total Score Description Length Limit
Genres
Grammar
Mention
Math
Perspective
OCR
GPT-4o
0.813587
0.823910
0.847312
0.965594
0.733660
0.820370
0.775862
0.733333
0.765351
Clause 3- 3-Opus
0.789474
0.805156
0.794086
0,916667
0.697115
0.771364
0.706897
0.629630
0.692308
Reka
0.776965
0,805536
0.807151
0.857904
0.732143
0.740253
0.687500
0.600000
0.689189
MinCPPM-Llama3-V
0.737190
0.715403
0.799068
0,849505
0.735294
0.711585
0,637931
0.716667
0.693694
Gemini-I 1.0Pro 1.0Prr
0.674504
0.676852
0.764286
0,768350
0.697695
0.615801
0.683333
0.611111
0,588235
LLaVA-1.5-13b
0.615427
0.663542
0.643424
0.643873
0.423077
0.619357
0.359195
0.666667
0.449561
ShareGPT4v
0.602989
0.674028
0.601496
0.625556
0,538462
0.560248
0,456897
0.555556
0.547619
Idefics-2-8b
0.442778
0,437893
0.494687
0,461218
0.520408
0.436042 0,276786
0.458333
0,412162
Table 6: Details of model scores evaluated by 3pt-44-mini-2224-07--B8
Model
Total Score Description Length Limit
Genres
Grammar
Mention
Math
Perspective
OCR
GPT-4o
0.909704
0,927875
0.912371
0.942057
0.862434
0.900441
0.857143
0,916667
0.882883
Clause-3-Opus
0,856077
0.890721
0.868490
0,917070
0.774590
0.819386
0.861111
0.725000
0.809524
0.839905
0,894752
0.785088
0,905643
0.713661
0.801667
0,925926
0.657407
0,828571
MinICPM-Llam3-V25
0.798023
0,828916
0.771795
0,823087
0.751944
0.763976
0.721264
0.816667
0,841880
Gemini- 1.0-Pro
0.773569
0,817422
0.735470
0.788911
0.797814
0.683020
0.866071
0.870370
0.806373
LLaVA-1.5-7b
0.683947
0.758817
0,703750
0.674046
0.630208
0.617620
0,425287
0,800000
0.602564
ShireGPTAv
0.689046
0,800461
0.657738
0.608733
0.654762
0.601754
0.500000
0.800000
0,743056
Idefics-2-8b
0,541755
0,560243
0.619318
0.489276
0.646825
0.455342
0.405556
0.375000
0.627193
Table 7: Details of model scores evaluated by 2004----022----11..
Model
Total Score Description Length Limit
Genres
Grammar
Mention
Math
Perspective
OCR
GPT-4o
0.899410
0,909379
0.916204
0.969395
0.854885
0.861247
0.920290
0.907407
0.878378
Clause 3 3-Opus
0.848949
0,861543
0.871686
0.896552
0.797170
0.808777
0.846154
0.645833
0.865741
8.828844
0.809259
0.873276
0.725309
0.770225
0.750000
0.819444
MiniCPM Llama3-V2. 0.787537
0,818813
0.790246
0.795796
0.768182
0.736359
0.676667
0.716667
0,828125
Gemini-1.0-P
0.763240
0.814379
0.750000
0.785159
0.757682
0.672255
0.758333
0.785714
0.776042
LLaVA-115-7b
0.660472
0.751873
0.661822
0.649851
0,498512
0,572719
0.516667
0,750000
0,5714229
ShareGPT4v
0.657186
0,765309
0.632682
0,575578
0.583333
0.545104
0.464286
0.675000
0.717742
Idefies-2-8b
0.536134
0.589964
0.541887
0.455882
0.611582
0.449821
0.406667
0.527778
0.576190
Table 8: Details of model scores evaluated by got-40-2024-11-22.
20

---

Published as a conference paper at ICLR 2025
Model
Total Score Description Length Limit
Genres
Grammar
Mention
Math
Perspective
OCR
GPT-4o
0.896893
0,906288
0.917996
0,955952
0.830508
0.867949
0.846667
0.833333
0.896396
Claude-3-Opus
0.861628
0,895363
0.866039
0,927730
0.807018
0.820549
0.857639
0.666667
0.800926
Reka
0,830885
0.869867
0.821685
0,883403
0,795597
0.772894
0.813218
0.675000
0.848485
Mimm..M- 0.780966
0,831197
0.766026
0.796257
0.726190
0.722037
0.691358
0.656250
0,768018
Gemini- P.o-Pro
0.757733
0.793860
0.724138
0.745455
0.854167
0.670349
0.810606
0,822917
0,822581
I.aVA-1L5-7b
0.667826
0.743137
0.638889
0.675287
0.571212
0.594505
0.500000
0,758333
0.596774
ShareGPT4v
0.666092
0.773905
0.661290
0.573904
0.562500
0.570722
0.458333
0.638889
0.695238
2deficx-2S8b
0.535057
0.597963
0.531810
0.483768
0.593056
0.452361
0.326087
0.458333
0.569444
Table Details of model scores evaluated by chatgpt-4o-latest
21


Reviews:
[{"review_id": "my2gSnQNCf", "reviewer": "Reviewer_drhM", "summary": "This paper introduces MIA-Bench, a benchmark designed to evaluate multimodal large language models (MLLMs) on strict adherence to complex instructions. Based on a dataset of 400 image-instruction pairs, MIA-Bench assesses the performance of 29 popular MLLMs, highlighting that current MLLMs struggle with precise instruction adherence. In addition, the paper also explored the supervised fine-tuning (SFT) method based on the LLaVA-NeXT model and achieved positive results, demonstrating the potential effectiveness of this method in improving model instruction adherence.", "questions": "please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) MIA-Bench fills a gap in existing benchmarks by focusing on the ability of multimodal large language models (MLLMs) to adhere to complex instructions, a crucial yet previously underexplored area of evaluation. This benchmark’s design reveals potential deficiencies in models when following strict, multi-layered instructions, supporting the practical deployment of multimodal models in complex, instruction-based tasks.\n(2) The MIA-Bench dataset consists of 400 image-instruction pairs covering various instruction types, such as description, length limit, genre, grammar, and OCR. With data from diverse sources, it reflects the variety found in real-world scenarios. This diverse set of instructions enhances the benchmark’s comprehensiveness and real-world applicability.\n(3) The paper provides a systematic evaluation of 29 popular MLLMs, analyzing their performance across different instruction categories. This large-scale comparison offers researchers a detailed performance reference and targeted insights for future model improvements.", "weaknesses": "(1) Although the SFT experiments in the paper demonstrate improved performance in adhering to complex instructions, they may lack in terms of model generalization. The results may be specific to the LLaVA-NeXT model, with no validation of applicability to other models, leaving it unproven whether SFT is equally effective for enhancing complex instruction adherence in MLLMs of different architectures and sizes.\n(2) The differences between MIA-Bench and other benchmarks are indeed striking, and the authors believe that this may indicate that MIA-Bench's unique design focuses more on evaluating the model's strict instruction-following ability. However, the authors' explanation does not completely rule out the possibility that MIA-Bench itself may have design biases. This explanation is based on speculative experimental results and has not been verified in depth by sufficient experiments.\n(3) Although the authors introduced Claude-3-Opus for comparison to mitigate scoring bias, the two models may exhibit similar biases in evaluation. Therefore, using Claude-3-Opus as the sole comparison tool may be insufficient to fully reveal potential scoring biases in GPT-4o.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g4dnXUnCDr", "reviewer": "Reviewer_yNxN", "summary": "This paper proposes MIA-Bench, which mainly focuses on the instruction-following capability of current Large Multimodal Models (LMMs) under complex and compositional instructions. Unlike previous close-ended benchmarks (e.g., multiple choice for MMBench) and open-ended benchmarks (e.g., LLaVA-Bench), MIA-Bench aims to evaluate precise adherence to complex instructions. As for evaluation, the authors leverage GPT-4o as the judge model. The authors have evaluated many representative LMMs on the proposed MIA-Bench.", "questions": "I do not have any further questions. Please refer to the \"weaknesses\" section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Evaluating the capability of LMMs to adhere to complex instructions is worth studying.\n2. Table 2 demonstrates that MIA-Bench is not highly correlated with existing benchmarks.\n3. Figure 8 illustrates that MIA-Bench is vision-centric, which is not strongly correlated with the performance of the LLM backbone.", "weaknesses": "In the context of evaluating open-ended freeform responses like MIA-Bench, it is crucial to account for the inherent variability introduced by the judge model. For example, the performance of MM-Vet has been observed to fluctuate by up to 10 points when assessed with different versions of GPT. \n\nThis variability raises several important considerations:\n\n1. **Standard Deviation Reporting:** When evaluating Large Multimodal Models (LMMs) using the same judge model (e.g., GPT-4o) across multiple trials, it is essential to report the standard deviation of the performance metrics.\n2. **Detailed Generation Configuration:** The specific generation parameters of the judge model, such as top-p, top-k, temperature, and num_beams, should be explicitly documented.\n3. **Impact of Generation Configuration:** It is necessary to investigate whether the above generation configuration of the judge model has a substantial impact on the performance metrics.\n4. **Performance Across Different Judge Models:** Detailed performance metrics should be reported for various judge models. Specifically, the performance of LMMs should be evaluated using different versions of judge models, such as Claude-3-Opus, GPT-4o-20240806, GPT-4o-20240513, GPT-4v-20240409, GPT-4-1106-preview, or even open-sourced models (e.g., Qwen2.5 and LLaMA-3.1). This comprehensive approach will help in identifying any model-specific biases and in providing a more reliable assessment of the LMMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KnS87ZBwrn", "reviewer": "Reviewer_PupP", "summary": "To evaluate and guide the instruction-following capabilities of multimodal models, MIA-Bench introduces a set of benchmarks for multimodal instruction. MIA-Bench primarily measures the abilities of MLLMs on following layered and compositional multimodal instructions and provides a set of multimodal instructions to enhance model performance in these areas.", "questions": "1. The 400 instructions in MIA Bench are manually annotated. It is important to ensure the reasonableness of the sub-instructions, such as whether the length limitation is appropriate. Additionally, has there been any manual verification of the scoring results from GPT-4o to confirm their accuracy and reasonableness?\n\n2. The instructions in MIA-Bench are composed of multiple sub-instructions. How is the number of sub-instructions per instruction determined? Is there a difficulty grading for the instructions, such that a higher number of sub-instructions indicates a greater level of difficulty?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a benchmark for evaluating the abilities of MLLMs on following compositional instructions, covering a variety of categories and tasks, and providing guidance for MLLMs in following complex composite instructions in the future. \n\n2. The paper proposes a set of instructions that can effectively enhance the ability of MLLMs to follow layered and compositional instructions.", "weaknesses": "1. Since the images in MIA-Bench are sourced from widely used datasets such as COCO 2017, SBU, TextVQA, and Flickr, I believe you should prioritize evaluating whether the current open-source models exhibit any data leakage issues on these datasets to demonstrate that MIA-Bench does not suffer from data contamination.\n\n2.  I notice that the experiments in Table 3 were conducted on LLaVA-Next-13b. Is this done after fine-tuning on a preference-aligned model? If so, I think this continued SFT setup leading to a performance improvement in MIA-Bench, while other benchmarks exhibit varying degrees of decline, is quite intuitive. Could you please provide additional experiments on the impact of mixing generated instructions with original instructions on model performance? Furthermore, I notice that there are only 5000 generated instructions here; how would the model performance on MIA-Bench be affected if the number of instructions were increased?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jdS0SSMVX5", "reviewer": "Reviewer_UieD", "summary": "The paper introduces MIA-Bench, a benchmark for evaluating the ability of multimodal large language models (MLLMs) to follow complex instructions. The benchmark consists of 400 image-prompt pairs that test the models' compliance with layered instructions to generate accurate responses. The evaluation of various state-of-the-art MLLMs shows significant performance variations, indicating areas for improvement in instruction adherence. The paper also discusses the creation of additional training data and supervised fine-tuning to enhance the models' instruction-following capabilities without affecting their performance on other tasks.", "questions": "- In MIA-Bench, are instructions all related to the corresponding image content or randomly picked from a instruction bank?\n- Instruction following abilities highly depends on LLMs used in the MLLM models. It would be better to group the performances in Table 1 and Table 2 according to the LLMs' size.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The paper presents a new benchmark that rigorously tests multimodal large language models' ability to follow complex instructions, addressing a previously underexplored area.\n\n+ This paper offers a comprehensive set of 400 diverse image-prompt pairs that challenge models with layered instructions, enhancing the assessment of their linguistic and descriptive capabilities.\n\n+ Experiment demonstrates performance improvements in instruction adherence through supervised fine-tuning.", "weaknesses": "- Complex instruction flowing ability might not be the main interests for MLLM users currently, especially when it comes to writting style constraints such as length and genre. Such abilities are mostly evaluated on LLMs.\n\n- The performance of models on MIA-Bench does not necessarily correlate with their performance on other benchmarks, suggesting that excelling in this specific task may not translate to generalized improvements across different multimodal tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)