MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models
Authors: Peng Xia
Siwei Han
Shi Qiu
Yiyang Zhou
Zhaoyang Wang
Wenhao Zheng
Zhaorun Chen
Chenhang Cui
Mingyu Ding
Linjie Li
Lijuan Wang
Huaxiu Yao
Venue: ICLR 2025 Oral (2025)
Keywords: large vision-language model
interleaved text-and-image evaluation
Primary area: datasets and benchmarks

Abstract: Interleaved multimodal comprehension and generation, enabling models to produce and interpret both images and text in arbitrary sequences, have become a pivotal area in multimodal learning. Despite significant advancements, the evaluation of this capability remains insufficient. Existing benchmarks suffer from limitations in data scale, scope, and evaluation depth, while current evaluation metrics are often costly or biased, lacking in reliability for practical applications. To address these challenges, we introduce MMIE, a large-scale knowledge-intensive benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs). MMIE comprises 20K meticulously curated multimodal queries, spanning 3 categories, 12 fields, and 102 subfields, including mathematics, coding, physics, literature, health, and arts. It supports both interleaved inputs and outputs, offering a mix of multiple-choice and open-ended question formats to evaluate diverse competencies. Moreover, we propose a reliable automated evaluation metric, leveraging a scoring model fine-tuned with human-annotated data and systematic evaluation criteria, aimed at reducing bias and improving evaluation accuracy. Extensive experiments demonstrate the effectiveness of our benchmark and metrics in providing a comprehensive evaluation of interleaved LVLMs. Specifically, we evaluate eight LVLMs, revealing that even the best models show significant room for improvement, with most achieving only moderate results. We believe MMIE will drive further advancements in the development of interleaved LVLMs.

Published as a conference paper at ICLR 2025
MMIE: MASSIVE MULTIMODAL INTERLEAVED
COMPREHENSION BENCHMARK FOR LARGE VISION- LANGUAGE MODELS
Peng Xia'; Siwei Han Shi Qiu Yiyang Zhou', Zhaoyang Wang', Wenhao Zheng', Zhaorun Chen², Chenhang Cui², Mingyu Ding', Linjie Li,, Lijuan Wang", Huaxiu Yao UUNC-Chapel Hill, MMicrosoft Research, University of Chicago, "NUS (pxia, siweih, shiqiu, huaxiu unc.edu
ABSTRACT
Interleaved multimodal comprehension and generation, enabling models to pro- duce and interpret both images and text in arbitrary sequences, have become a pivotal area in multimodal learning. Despite significant advancements, the evalu- ation of this capability remains insufficient. Existing benchmarks suffer from lim- itations in data scale, scope, and evaluation depth, while current evaluation metrics are often costly or biased, lacking in reliability for practical applications. To ad- introduce MMIE, large-scale knowledge-intensss
dress these challenges, benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs). MMIE comprises 20K meticulously curated multimodal queries, spanning 3 categories, 12 fields, and 102 subfields, including mathematics, coding, physics, literature, health, and arts. It supports both interleaved inputs and outputs, offering mix of multiple-choice and open- ended question formats to evaluate diverse competencies. Moreover, we propose a reliable automated evaluation metric, leveraging a scoring model fine-tuned with human-annenn annotated data and systematic evaluation criteria, aimed at reducing bias and improving evaluation accuracy. Extensive experiments demonstrate the effee- tiveness of our benchmark and metrics in providing a comprehensive evaluation interleaved LVLMs. Specifically, evaluate eight LVLMs, revealing that even the best models show significant room for improvement, with most achieving only moderate results. We believe MMIE will drive further advancements in the devel- opment of interleaved LVLMs. We publicly release our benchmark and code in https://mmme-brett benci,eeeeeeeheee github.io/
Content warning: this paper contains content that may be inappropriate or offensive.
INTRODUCTION
"True evaluation lies in the seamless interweaving of diverse modalities."
Multimodal learning has made remarkable progress with the development of Large Vision-Laagume Models (LVL.Ms) (Liu et al. 2023a Zhu et al. 2023 Dai et al. (023), which are capable of handling diverse tasks that involve both images and text. Despite their advancements, most of these models are limited to multimodal tasks for text generation, such as visual question answering (VQA) and image captioning, which do not fully reflect the potential of multimodal capacity. To broaden their application, interleaved text-and-ine iind-image generation has emerged as a critical area of research (Liu let al., 2024).. It requires models generate images and text in any sequence, thereby enhancing the versatility and effectiveness of multimodal systems. It opens up possibilities for various com- plex applications, such multi-step inference (Lu et al. 2024 Kazemi et al. 2024), multimodal situational analysis (Yang et al. 2021). and visual storytelling (Huang et al. (0166).
While recent LVLMs are evolving to support interleaved text-and-image generation (Tea 2024
Xie et al. 2024 Chern et al,, 2024 Zhou et al. 2024), comprehensive evaluation benchmark is still falling behind due the following two challenges:
"Equal Contribution.

---

Published as a conference paper at ICLR 2025
EECS
Finance
Education
Health
Qanetiso: How Get Ris of Dik
Guesllon How io Rie for
Quentoon How to Cape with
Question: How to Get Full Custody of
Metlod 2: riBarssing Vinus in Sara Mode
Mothod Organicng Your
Methed 1: Deaking with
Aathoo y Requeeling
Cusladt Ordar, diep 2.
dlep E: Turn on your Windows
cresit
burcuu....
to aspat dentures...
Skop 2: Poos le F8 key ceposeelyy
Stp 2: lake your cooi
Step 2. Riace your took sill
Sagp Uownloud somu yeu've repraree farma
Skip Seled "Sate Misde
Sty J: Oathr sppporing
Step lake owee counse pais mediatiee to can diccestort excl poudiii to get and how
on your culeuaae
Netwerking
lays
feaadal decunatl
Colokite your beasthudd
your caie.
Sup Sign your forma the
"Bate"
they
excaay
MMLI
Statistics
Quastion:: How Hold Geltor Nack?
Qusatior Compared to the fins graph, whish colegery vikee shanged sacond graph?? And iscrease or deorease?
Methed Plaging with Ournett Peoitang
Sie Sup33
nock pricts
Keep the froat of Oue qale
(CAT (SATESORY, (errosolleerrnssee)
arread ne flour of the body to bouf la place
): your sligght as you play.
Oreand Frask anower:
danased)
Sports
Literature
Physics
Mathematicss
Questionn
Tianeis Match?
Sptlogue 1: hawe brother
Qumetion the ABC shops
Nethod ::coring a Tarmia Maich
who line exampe of
before and after collston balseen hos
seodor Uhe aro langth of the
your
apporintt can't sukarn point.
Epslogue These ar siger long
coefistest metituior this
OEP seatoon compue the the degres the angle
rustreen point
beddgn get tren plac place.
J: that shoud by poiees t' scora iodia 4t1-0
repuin bodiiig beaches down
araiver to decimals
con computatione to decimmal places.
say di theita your epparent
Aptlogue My breteer ikks
Joho doot Lis place being Hotel
Greand Trask Answa:
10:00
Figure Typical samples from the MMIE Benchmark showcase its support for multiple image inputs and outputs, with ground truth provided for every query. MMIE evaluates models across diverse fields, ensuring a comprehensive evaluation of their capabilities.
The first challenge lies in the difficulty of constructing modallty--algnedd multimodal datasets, where both the input and output contain images and text. Current benchmarks mainly focus on single-modality output tasks (Fu jet al., 2023 Li et al., 2024a| Zhang et al.] 2023). assessing only the quality of the generated image or text, without benchmarking the crucial connection between modalíties, such as text- image coherence and consistency. Although few datasets support the interleaved multimodal evaluation method for LVLMs (Liu et al. 2024), their dataset is constrained by its limited scale and narrow query format, primarily focused on VQA lasks.
Difficulty in Constructing Modality-Coheree Benchmarks.
Lack of Automated Evaluation Metric. The second challenge is the lack of suitable au- tomated evaluation metries for interleaved generation. Human evaluation is costly and time- consuming, making it difficult to scale for practical applications. Current automated evaluation metrics typically assess either the quality of generated text (e.g., BLEU (Papineni et al. 2002).
BERTScore (Zhang et al. (0200) or the quality of generated images (e.g.. FID (Heusel et al.) 2017).. While recent evaluation strategies, such as using CLIPScore (Hessel et al., (021). and vision-language models (VLMs) (Chen et al. 2023 Liu et al. 2024), can evaluate the connection between different modalities, they rely heavily on the pre-trained knowledge of specific models (e.g., CLIP training data) follow rigid, cluman-definc' rules. These approaches can introduce bias and uncertainty to some extent, often leading to inconsistent results (Mahmoud et al. 2024).
To address these limitations, we introduce MMIE, Massive Multimodal Inverleaved understand- ing Evaluation benchmark for LVLMs with proposed reliable and automated metrics. MMIE is cu- rated from four multimodal datasets, involving 3 categories, 12 fields, and 102 subfields, including mathematics, physics, coding, statistics, literature, philosophy, education, finance, health, sports, art, and EECS (Electrical Engineering and Computer Science). The dataset comprises 20K multimodal questions, supporting both interleaved inputs and outputs. It features mix of multiple-choice and open-ended question formats to evaluate broad spectrum of competencies across various fields. As shown in Table [ MMIE surpasses existing interleaved multimodal benchmark in both depth and width, particularly in addressing complex problem-solving and open-ended creative tasks, Based on the curated dataset, we further propose automated metric powered by scoring model. Specif- ically, we first design a comprehensive evaluation criteria for each category. Then, we curate a fine-grained, suman-annotate scoring dataset and then use this dataset to fine-tune the InternVL- 2 (Chen et al.) 2024c) to obtain the scoring model. Using MMIE, we evaluate four open-source interleaved multimodal LVLMs, well as combinations of advanced LVLMs like GPT-4o with
2

---

Published as a conference paper at ICLR 2025
text-to- text-to-image generative models (e.g., Stable Diffusion 3 (Esser al. 2024 ) Our key contribu- tions are summarized as follows:
We introduce the largest high-quality interleaved multimodal benchmark MMIE for evaluating LVLMs, with the dataset to be publicly released.
MMIE presents significant challenges to LVLMs, with the best-performing model (e.g., GPT-4o + SDXL) achieving a score 65.47%, highlighting substantial room for improvement.
The proposed scoring model is reliable and has proven to be comparable human evaluation.
RELATED WORK
Interleaved Multimodal Comprehension and Generation. Multimodal learning has rapidly evolved, with substantial progress in integrating text and image modalities. Recent advancements in
large vision-language models (LVL.Ms) (Liu et al., 2023a Zhu et al. 2023 2024 Dai et al., 2023
Xia et al. 2024b c). either driven by the integration of diffusion models like Stable Diffusion RRom- bach al. 2022 or using token-based mixed-modal structures like Chameleon (Team 2024) and
Show-o (Xie et al. 2024), have enabled models to not only understand and generate content across modalities, but also engage interleaved multimodal comprehension and generation. the de- mand for richer, more interactive AI grows, interleaved multimodal comprehension and generation becoming an essential component in the development of next-generation LVLMs.
LVLM Benchmarks. Despite the rapid advancements in multimodal learning, evaluation bench- marks remain far from perfect. Previous benchmarks primarily focused on evaluating the base per- ception ability of LVLMs (Lu et al.) 2022; Gurari al. 2018), such GQA (Hudson & Manning 2019). which lack the depth required to assess advanced reasoning. Recently, several high-quall evaluation benchmarks have been proposed to assess the reasoning ability of these models (Li al.
2024a] Jiang et al. 2024b Zhang al. 2024bb b bl Jiang et al.) 2025). such MMMU (Yue et al. (024) and MME (Fu al.
2024a Zhang et al. 2023 Liu et al.
2023a b Yu et al.] 2023 Xia et al.
2023). However, these benchmark do not support interleaved image-and-texx comprehension and generation. Large-scale interlcaved multimodal datasets like MINT-IT (Awadalla al., 2024). MANTIS (Jiang et al.] 2024a) and OBELICS (Laurençon et al.) 2024) have been developed pri- marily for pre-training models. However, they lack precise alignment between text and images, making them unsuitable for evaluation and benchmarking. A recent small-scale interleaved multi- modal benchmark has been introduced (Liu et al.) (2024), but its limited data size and query quality hinder the comprehensiveness of its evaluation. MMIE fills this gap by offering comprehensive evaluation framework that supports interleaved multimodal comprehension and generation. Our dataset includes a diverse set of queries among multiple domains. By evaluating both perceptual and generative capacity of LVLMs, provides a more holistic assessment.
Evaluation Metrics for Multimodal Tasks. Tra- ditional evaluation metrics, such as BLEU (Papineni et al. 2002). BERTScore (Zhang et al. 20200) for text
Table I: Dataset statistics.
Statistic
Number
Percentage
quality, and FID (Heusel et al. (01 for image qual- ity, are only suited single-modality output tasks.
Total questions
20103
Situational analysis
5005
24.89%
Recent metrics, such as CLIPScore (Hessel et al.
Project- bused learning
11482
57.12%
Multi step reasoning
3616
17.99%
2021] and X-IQE (Chen et, al. | 2023). have attempted to address this by introducing multimodal models to evaluate consistency between text and image. How- ever, these metrics only measure alignment and fall short of offering a comprehensive assessment out- put quality. Furthermore, many multimodal met- rics depend on GPT-based models (Liu et al. 2024). bringing uncontrollable bias to the whole evaluation system. To overcome these drawbacks, we propose an automatic metric to minimises bias and pro-
Total CaregoriesFFelld//auee
3/12/102
Multiple-Choice Questions
663
3,40%
Open-Ended Questions
19340
96.60%
Questions with Images Questions with answer label
20103
100%
20103
100%
Average question length
76.0
Average images per question
1.32
vides thorough analysis of the generated results.
3

---

Published as a conference paper at ICLR 2025
3
THE MMIE BENCHMARK
3.1 OVERVIEW
Health 2000
Finance 2004 23%
In this section, we introduce MMIE, diverse and comprehen- sive benchmark for evaluating interleaved multimodal com- prehension and generation across broad scope of tasks. As shown in Table 2] MMIE consists of 20,10) 103 curated samples spanning 12 fields, including mathematics, physics, coding, statistics, literature, philosophy, education, finance, health, sports, art, and EECS. Each query is meticulously selected, filtered, and refined to ensure both high quality and relevance across the covered subjects. In addition, MMIE emphasizes the evaluation of three essential competencies: perception, rea- soning, and generation. Unlike previous benchmarks that eval- uate the results from single modality (Fu al. [0233] Yue et al.) 2024 Li et al. 2024b) output, MMIE is specifically designed assess models' capabilities in understanding and generating interleaved text and images in any sequence. This evaluation extends beyond basic perception by requiring models to engage in complex reasoning, leveraging subject-specific knowledge across dif-
17%
ERCS
Education
16%
Project -bowed lcorning
11, 11,482
Art
1394
S7
Maat Mastiis Phoaer Calles
sterd
Literature 100 %
Figure 2: gories and fields in MMIE.
Distribution of cate-
ferent modalities.
3.2 DATASET CURATION
The data curation process in MMIE consists of two stages, each designed to ensure both comprehensive coverage and high-quality representation across various categories in our benchmark. We detail the process follows:
Feedback
In the first stage, collect and restructure four multimodal datasets to align with the interleaved image-and-ters format and categorize them into three categories situational analysis, project-based learning and multi-step reasoning, which are illustrated in Figure 2 for project-based learning, we extract data from Wikihow (Yang et al. 2021]. which is originally designed for testing models' ability to choose the correct procedural steps based on given text and image contexts. We adapt it to the interleaved text-and-imas image format. For situatioona analy- sis, we draw samples from VIST (Huang al.) (0166), naturally inter- leaved multimodal dataset designed for visual storytelling tasks, which challenges models to seamlessly integrate narrative text and images. Both situational analysis and braject-based learning datasets feature in- terleaved inputs and outputs. To expand the benchmark with more complex and diverse tasks, we further introduce datasets focused on multi-step reasoning that support interleaved inputs. For this aspect, we source examples from MathVista (Lu et al.] 2024) and ReMI (Kazemi et al.) (0024), which together provide 3,600 questions covering topics from functions to statistics. The answer for- mats for these queries include multiple-choice questions (selecting one option from several choices) and open-ended questions (directly generating content). After extracting samples from these four datasets, we merge and refine them into a cohesive benchmark by compacting, restructuring, and integrating questions from multiple sources, ensuring consistency with our evaluation objectives.
UYLA
Score
Orlinta
Specifically,
Duatit
Hodel Fine
Dats
Disil LVLM
Finh now-kandd
Figure 3: Pipeline of the scoring model.
In the second stage, we implement multi-step quality control process to ensure the integrity and consistency of the dataset. First, we apply lexical overlap and source URL similarity checks to identify and flag potential duplicate entries, which are then manually reviewed and removed. Next, each dataset meticulously reviewed for formatting and typographical consistency to ensure ad- herence to standardized structure. Discrepancies are corrected to maintain uniformity across the entire dataset. In total, we finally collect 20, instances across 12 fields, including mathematics, physics, coding, statisties, literature, philosophy, education, finance, health, sports, art, and EECS. Detailed categorization and dataset statistics are presented in Table I For more information about dataset curation, please refer Appendix A.
4

---

Published as a conference paper at ICLR 2025
Table The comparison between MMIE with other LVLM benchmarks. Inter-I: interleaved input;
Inter-O: O:: interleaved output; Multi-I: multi-image for input; Multi-O: multi-image for output.
Dataset
Data Scale Inter-l Inter-O
Multi-I
MultiO #Num Domnins
Answer Type
Metric
HumamEral
No
Open
Passik
GIMMKK (OWee
Open
Passöik
MME (Fa at P0743
2K
Yes
No
No
No
Multi-Choice Multi-Choice
ACC
MMBench Lat 20236
3K
Yes
No
No
ACC
MM-V
Open
GPTT4
MagicBrashi Ziuni clal
2023
10K
Yes
No
Imnge Editing
CLIPSoore
MMMU Yf 21/4
11.5K
Yes
No
Yes
No
30
Mulli-Chocc
ACC
MVRenchi [arera 21143
4K
Yes
No
Yes
No
9
Multi-Choice
ACC:
INTERLEAVBDBENCH ctl 2024
Open
GPT-4o
MMIE (Ours)
20K
Yes
Yes
Yes
Yes
12
Malti-Choice Open Fine rine-tuned VLM
3.3
AUTOMATED EVALUATION METRIC
As traditional metrics such as BLEU, BERTScore, and CLIP-Score fail provide a thorough evalu- ation of the quality of multimodal outputs, existing benchmarks use the GPT-4 series as the scoring model, which may introduce inherent bias in the scoring process (Liu et al., 2024). To ensure a comprchensive and unbiased evaluation of various LVLMs, as shown in Figure 3] propose an automated evaluation metric powered by our fine-tumed LVLM to assist in scoring. Here, we choose Intern VI.-2-4B (Chen et al., 2024c) the foundation for our scoring system due to its strong perfor- mance in multimodal reasoning tasks and support for multi-image inputs. Furthermore, we fine-tune the Intern VL-2-4B to mitigate potential bias.
Specifically, we first construct a high-quality multimodal scoring dataset that covers all aspects of our benchmark, accompanied by detailed scoring criteria and reference answers. In this process, we collect 800 responses from four AVLLMs-MiniGPT: (Zheng et al. 2023), EMU-2 (Sun et al.) 2024). GILL GlLh et al. (2023). and Anole (Chern et al. 2024). Based on the ground define an evaluation standard using six-point grading scale with clear criteria. A group of experts generates reference answers for each level and all score statistics are converted to percentage format. These criteria and reference answers together form robust rubric for MMIE. Following the rubric, human annotators rigorously score the responses. Detailed examples of the rubric and construction process are provided in Appendix A.9 and Appendix A.3
labels, we
After constructing the scoring dataset, we fine-tune the InternVL-22-4B model and use the fine-tuned version as our scoring model. To validate its performance, we randomly select 200 new samples with human-scored labels and compare the results of our model with those other scoring models. The results show that the fine-tuned model significantly improves alignment between human scores and our model-generated scores compared to other LVLMs, leading to more accurate and reliable evaluation across diverse tasks. We will discuss the experimental results in detail in Section 4.3
3.4 COMPARISON WITH EXISTING MULTIMODAL BENCHMARKS
MMIE surpasses existing benchmarks in three key aspects. First, most previous multimodal bench- marks support only single-modality input or output, while MMIE closes this gap by enabling inter- leaved text-and-image comprehension and generation. Our dataset ensures robust modality align- ment, with multimodal question-answer pairs reconstructed into an interleaved eext-and- in- struction format, followed by manual review to guarantee quality. Moreover, the scenarios reflect real-world applications, such as multimodal script generation, data chart analysis, and multimodal story generation. Second, compared recent interleaved comprehension benchmarks (Liu et al., 2024]. MMIE is larger in scale and covers a broader range of subjects, containing both reasoning and temporal understanding skills, allowing for a more comprehensive evaluation. Finally, MMIE introduces reliable scoring system powered by fine-tuned LVLM, which significantly enhances the accuracy and reliability of scoring. Table 2 highlights the differences between our benchmark and existing ones, demonstrating the advantages of MMIE in terms of scale, diversity, and scoring methodology.
4 EXPERIMENT
MMIE provides a systematic evaluation of existing open-source LVLMs supporting interleaved multimodal input and output (interleaved LVLMs), along with the integration of state-of-the-art
5

---

Published as a conference paper at ICLR 2025
LVLMs and text-to-image generative models (integrated LVLMs). In this section, aim to swer the following key questions: (1) Which interleaved LVLM performs best on MMIE overall? (2) How effective are the integrated LVLMs? (3) Do the evaluated LVLMs show preference for certain field? and (4) How useful are our proposed model-powered metric compared with traditional metrics and other LVLM evaluation?
4.1 EXPERIMENT SETUP
Baseline Models. We first benchmark four open-source interleaved LVLMs. (1) MiniGPT-5 (Zheng al., 2023), multimodal model combining MiniGPT-4 and Stable Diffusion, specialized for co- herent image-text generation. (2) EMU-2 (Sun et al. 2024), 37B 37B-3 11rammeeeterr model excelling in in-context learning and multimodal reasoning, (3) GILL (Koh et al., (023), a model special- ized in generating and retrieving interleaved outputs, (4) Anole (Chern et al., 2024), based on Chameleon (Team. 2024), model excelling in text quality, adds vision and multimodal genera- tion capabilities.
To broaden the comparison, we also compare with integrated LVLMs consisting of text-output LVLMs (i.c., GPT-4o (Achiam al. 2023), Gemini-1.5 (Reid al. 2024), LLaVA-v1.6-34b (Liu et al. 2023a) and Qwen- VLb (Wang et al. 2024)) and text- text-to- to-ee generative models (i.e., Openjourney (ope). Stable Diffusion 3 Medium (Esser al. 2024). Stable Diffusion XL turbo, Flux. (flu)). We provide the interleaved text-and-image input to the LVLM to generate text, and then feed this text to a text-to-image generative model generate image. The resulting multimodal output from this process is considered as interleaved output for evaluation.
Human Annotators. We organize a group of senior top-tier college students, contributing to the curation of the scoring dataset. To ensure thorough and consistent evaluations, we develop detailed criteria for each category of our benchmark (see Appendix A.9 for details).
Evaluation Metrics. We evaluate the performance of all models using our proposed metric in Sec- tion 3.3 which is powered by our fine-tuned LVLM based on Intern VL-2-4B (Chen al. 2024c), to ensure reliable scoring.
4.2 MAIN RESULTS
In this section, we present the comprehensive evaluation on our MMIE benchmark. The detailed performance of interleaved LVLMs and integrated LVLMs is shown in Table 3 and Table tively. We summarize our key findings as follows:
respec-
Challenging Evaluation and Promising Direction. As illustrated in Table all evaluated inter- leaved LVLMs show poor performance, with an average score of 50.80%. Even when integrating advanced models such GPT-4o and text-to-image generative models, as shown in Table 4 the best score (GPT-4o + SDXL) reached 65.47%. This highlights the high level of difficulty and the challenge posed by MMIE. Interestingly, the latest interleaved LVLMM Anole (Chern al., 2024) shows significant improvements over previous interleaved LVLMs, including MiniGPT-5, GILL and EMU-2, by 8.4%, 7.0%. 21.8% in average score, respectively. This points to the growing potential of interleaved text-and-image models as a promising direction for future progress in multimodal com- prehension and generation. To facilitate the broader adoption of MMIE, extract 1,000 samples to create a mini-set. Detailed results can be found in Appendix A.5
Gap between Interleaved LVLMs and Integrated LVLMs. Existing interleaved LVLMs are still quite limited. To enhance our evaluation and analysis on our benchmark, we integrate non- interleaved LVLMS with T2I models in our experiments. This integrated LVLMS approach signifi- cantly outperforms previous open-source interleaved LVLMs, improving performance by an average of 25.2% across all categories. Specifically, the integrated models outperform the best performance of the interleaved model by 14.6% 26.3%, and 16. % in situational analysis, project-based learning, and multi-step reasoning, respectively. Surprisingly, the integrated LVLMs perform exceptionally well in project-based learning, with all models based on LL achieving scores above 70%. These findings suggest that combining the strong comprehension abilities of non-interleaved LVLMs with the generative power of T2I models offers promising path for future research.

---

Published as a conference paper at ICLR 2025
Table 3: Performance of the four open-source LVLMs supporting interleaved image-and-text input and output on MMIE, shown as percentages.
Model
Situational analysis
Project-based learning
Multi-step reasoning
AVG
MiniGPT-5 Zheng ct al. 2023
47.63
55.12
42.17
50.92
EMU-2 Sun et al 2024
39.65
46. 46.12
50.75
45.33
GILI Kmn araie 2125
46.72
57.57
39.33
51.58
Anole Cher et al. 2024
48.95
59.05
51.72
55.22
Model Performance across Different Fields. As previously demonstrated in Table and Table model performance varies across different categories of data, achieving the best results in project- based learning and the lowest scores in situational analysis. This indicates that the model's per- formance differs depending
the category, likely due to inherent issues with the distribution of the training data. For example, Anole (Chern et al. 2024) scores 59.05% in mroject-based learning data but only 48.95% in situational analysis, suggesting it excels creative, open-ended generation but falls short in handling detailed, discipline-specifie knowledge. Delving into more fine-grained fields, shown in Figure 4 different models exhibit preferences for certain fields of data. Among the seven fields of project-based learning, including education, finance, health, philosophy, sports, art and EECS, almost all models tend to perform well in areas that are easier to understand, such as philosophy, art and education, but face challenges in more complex fields requiring higher rea- also shows general gradual decline scores for the criteria of text and image quality, text-image coherence, method quality and practical utility, creativity and engagement, stylistic consistency and correspondence, suggesting that there a sig- interleaved output solve real-world
soning abilities, such as finance and EECS. Figure also
nificant lack of text and image alignment and the ability problems across all models. Detailed results can be found in Appendix A.7
Table 4: Comparison with state-of-the-ar LVLMs integrated with text-to-image models, referred to as integrated LVLMs, evaluated on MMIE. LLaVA only supports dingle-imae input and all multi-image queries are thus skipped.
LVLM
T21 Model
Situational analysis
Project-based learning
Multi-step reasoning
AVG
Openjoumeyy
$3.05
71.40
63.65
SD-3
53.00
71.20
63,52
GPT-4o
53.67
SD-XL.
56.12
73.25
65.47
Flux
$4.97
68.80
62.63
Openjoumcy
48.08
67.93
61.57
SD-3
47.48
68.70
61.87
Gemini-1.5
60.05
SD-XL
49.43
71.85
64.15
Flux
47.07
68.33
61.55
Openjoumey
54.12
73.47
63.93
SD-3
54.72
72.55
63.57
LLaVA-34b
47.28
SD-XL
55,97
74.60
65.05
Flux
54.23
71.32
62.73
Openjoumeyy
52,73
71.63
64.05
SD-3
54,98
71.87
64.75
Qwen2-VI. VI.-7hh
55.63
SD-XI.
52.58
73.57
65.12
Flux
54,23
63.18
4.3 How CONSISTENT IS OUR MODEL-POWERES METRIC W.R.T HUMAN ANNOTATION?
In this section, we further validate the effectiveness of our proposed metric. Here, we conduct an experiment to evaluate its correlation with human annotations using several disparity and similarity metrics, i.c., cosine similarity, mean square eтror (MSE), mean absolute error (MAE). and Pearson coefficient. For comparison, we report results from traditional multimodal alignment metric (i.e., CLIPScore) and scores judged by LVLMs, including GPT-4o, which has already served as the met- ric in (Liu et al. 2024). As shown in Table our metric demonstrates the closest alignment with human evaluation results significantly, proving to be the most reliable. Our scoring model effec- tively captures the multimodal features of both image and text sequences and judges them through complex reasoning precisely. In contrast, other LVLMs and CLIPScore tend to focus primarily understanding the sequence information, but they fall short grasping the relationships between the sequences and accurately judging the alignment between them. In summary, the experiments

---

Published as a conference paper at ICLR 2025
Text and Image quality
Text-Image Coherence
Method Quality Practical Utility
Art
Art
Ad
CECS
Philoopphy
Phicoopht Education
Philooophy
Finnnce
Healt
Finance
Health
France
Health
Creativity & Engagement
Stylelistic Consistency & Correspondence
Total Score
Art
Art
Spors
Sports
Education
Philospphy Education
Philosophy
Education
Philsoppty
Finanos
Heath
Finance
Healh
Finncc
Healhh
Minigaii
Emuu
Andle
GPT--4
Gemeni115
LLaWW-O
One--VL-70
Figure 4: The average and total scores of each model across the seven fields of project- based learning based on criteria. We take the average of GPT-4o, Gemini-1.5 LLaVA-v1.6 1.6-34b and Qwen- VL- 2-72b over the four text-to-image diffusion models.
demonstrate that our metric is a robust and dependable standard for evaluating interleaved multi- modal generation. We provide analyses of the scoring bias and generalization of the MMIE-Score in Appendices A.4 and A.8 respectively.
Table 5: Comparison of scoring LVLMs and traditional image-text alignment metric.
Models
Cosine Similarity
MSE
MAE
Pearson
Tmxe- CLIPScore
0.639
7.312
2.251
0.023
IntemVL-220044
0.736
15.962
3.165
0.083
Anole
0.805
3.969
1.600
0.048
GPT-4o
0.733
3.724
1.573
0.042
Ours
0.873
3.300
1.444
0.113
ERROR ANALYSIS
This section offers a detailed analysis of the errors identified during the evaluation. We catego- rize the key challenges into two types: temporal understanding and reasoning ability. Specifically, temporal understanding issues refer to multimodal information comprehension and cross cross-modality coherence, while reasoning issues involve complex reasoning and generation capabilities. This anal- ysis, drawn from expert annotators' observations during the scoring process, not only underscores the model's current limitations but also informs potential improvements for future development. Detailed examples can be found in Figure 5 More cases can be found Appendix
5.1 TEMPORAL UNDERSTANDING SKILL
The primary errors lie in cross cross-modee poos-modalityy coherence and generation adaptability. Many models strug- gle generate images that accurately correspond to the accompanying text, resulting in severe in- formation gaps, distortions, and redundancies.
Cross-modality Coherence. One of the most common errors is the incoherence between text and image generation. Due to deficiencies multimodal alignment, the details in the generated images often vague entirely missing, making it difficult for them to align with the context described in the text. A typical example, as shown in Figure 5 involves the model understanding the "Browser Image: How ToUse ever, the corresponding image it generates consists of little more than blocks color, lacking the necessary details to establish coherence and alignment with the text.
method correctly and producing accurate textual response. How-
8

---

Published as a conference paper at ICLR 2025
Cross-modality Coherence
Generation Adaptability
Question
Answer
Question
Answer
Skype for Free?
How gel Skype for Free? Using Image:
How to Get Wisconsin Court Records?
Method Installing Skype
Method 1: Finding Wisconsin
write.
Step 1: Go to the Skype website.
How SowkTosseeppeppppp pnng method...
Circuit Court Records. Step Gather the information...
X Duplicated text and low quality image
Ground Truth Step 2: Click the GGet Stype' in upper right corner the screen...
Ground Truth
Lack further details
Step 2: Search on the Wisconsin Circutt Court Access webetts. WCCA became evailsble
anline 1999...
Multimodal Information Comprehension
Complex Reasoning
How many algorithme have accuracy higher than least one datesel?
Question
X Overlook critical information
What nappens
Question
Caterpiliars are an important part the food chain. If caterpilars decrease, it could knock-on effect on the entire food chain. decrease caterpil could siso lead decrease the number insects are pollnators. Overall, a decrease in caterpilars could -naching potamtiall devastaling consequences the entire food chain and the ecosystem as whole.
Answer
Fail er the secondary effect popariion woud
caterpillars decreaze?
Answer
plants increace nothing happens
Here a bar chart comparing
none the above
accuracy of three algorithms four datasets
Ground Truth
Ground Truth plants increase
Figure 5: Examples model failures. Four typical types of errors are introduced and categorized, namely incoherence between text and image generation, inflexibility in generated responses, poor comprehension of multimodal information, and inability to manage complex reasoning tasks.
Generation Adaptability. Another significant error is the inflexibility of generated responses. For example, the model can only understand the given text and produce simple, detail-lacking responses. For example, in Figure [ the model's reply merely contains the title "the next step is to write" without further elaborating the steps process involved, which differs from the provided query example. This issue likely stems from a weakness in both text comprehension and generation.
5.2 REASONING SKILL
When evaluating the model's reasoning skills, the most prevalent error types are found in multimodal information comprehension and complex reasoning. Notably, many models exhibit significant errors
even in understanding interleaved information (Jin et al. 2024b a Chen et al. 2222aa] b Zhang et al.,
2024a), which inevitably leads to reasoning mistakes further down the process.
Multimodal Information Comprehension. key error evaluating LVLMs' reasoning abilities is their difficulty in comprehending multimodal queries, especially in extracting visual information from images. A frequent issue arises when the model correctly interprets the textual components of query but fails fully understand the visual details in image. For instance, in the case of a bar chart comparing four datasets by volume, where each dataset is represented by a bar with a corresponding height on the y-axis, the model might recognize the chart's title and tags but overlook the critical information conveyed by the bars themselves-Sun such as the relative sizes of the datasets. This highlights the model's tendency to focus surface-level textual cues without delving into the deeper graphical meanings embedded in images. It also underscores broader trend: LVLMs exhibit a strong bias toward processing text over extracting nuanced information from visual data and other non-textual modalities.
Complex Reasoning. Another significant error is the model's inability to handle complex reasoning tasks. As illustrated in Figure[3] the model demonstrates a pronounced weakness in multi-step infer- ence. For example, in an impact analysis decrease in caterpillars would lead to decline in bird populations but fails to infer the secondary ef- fect-that that plant populations would increase. Another instance is seen in arithmetic problems, where the model makes clear mistakes, such failing calculate the exact length of triangle. These examples underscore the need to strengthen the model's capacity for multi-step reasoning, making it more robust and reliable in handling complex tasks.
biological system, the model correctly predicts that a

---

Published as a conference paper at ICLR 2025
6 CONCLUSION
This paper introduces MMIE, a large-scale, diverse benchmark for interleaved image-and-text un- derstanding and generation. Spanning wide range of fields, MMIE provides comprehensive eval- uation framework for interleaved multimodal understanding and generation, featuring 20K queries. The dataset, which covers wide range of fields, ensures high-quality evaluation of LVLMs across various dimensions. Furthermore, our proposed model-pooeree metric effectively evaluates the quality of output image-text information based on the input image-text context. Our extensive ex- periments further demonstrate that the metrics we propose provide robust, human-like evaluation performance, significantly reducing errors and biases. Despite this, we observe that existing mod- els underperform, particularly in complex and deeply interleaved multimodal tasks, highlighting the challenges and opportunities that lic ahead in this domain.
ETHICS STATEMENT
This paper focuses on the evaluation of interleaved large vision-language models. newly con- structed human-annotated dataset was used to fine-tune the scoring model. The dataset curated following ethical guidelines to ensure that no sensitive information is included and to minimize bias during the annotation process. The evaluation process aims be transparent and reproducible, ad- hering to high standards research integrity and ethical conduct. No personally identifiable data was collected or processed.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we have made considerable efforts provide all neces- sary details and materials. Specifically, we have included a comprehensive description of the dataset creation process in Section 3 including annotation guidelines and data collection methods, and fur- ther elaborated in Appendix A. The benchmark and evaluation procedures are described in detail in Section 4 with the metrics used clearly defined to facilitate independent verification.
ACKNOWLEDGEMENT
We would like thank Danni Luo, Rundong Liu, Anqi Lv, Xin Xie, Bocheng Feng, Huanyan Ye, Shi' en Wan, Kaiyao Chu, Danning Li, Yiran Li, Siyuan Chen to participate in refining the criteria and Kaiyuan Zheng for his assistance in model selection. This work is partially supported by Cisco Faculty Research Award.
REFERENCES
1-ux..-dev. URL httpt/////whiiiie......oooooooor
Openjourney. URL
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint TSl::2333.087744 2023.
Anas Awadalla, Le Xue, Oscar Lo, Manli Shu, Hannah Lee, Etash Kumar Guha, Matt Jordan, Sheng Shen, Mohamed Awadalla, Silvio Savarese, et Mint-le It: Scaling open-source multimodal data by 10x: A multimodal dataset with one trillion tokens. arXiv preprint arXiv:2406. 1271, 2024.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pp. 370-387. Springer, 2024a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin. et al. Are we on the right way for evaluating large vision-lant language models? arXiv preprint arXiv:2403.20334 2024b.
10

---

Published as a conference paper at ICLR 2025
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03311 2021.
Yixiong Chen, Li Liu, and Chris Ding. X-iqe: explainable image quality evaluation for rext-to-image generation with visual large language models. arXiv preprint 1.XXx:23055108444 2023.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to com- mercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024c.
Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large mul- timodal models for interleaved image-text generation. arXiv preprint arXiv:2407.06135, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14166 2021.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards veneral-purposs vision-language models with instruction tuning. arXiv preprint M.Xiv:23055065000 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 00000, 2020.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Ting-Hau Hao Huang. Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pp.
2016.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real- world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint 24X5vv2405.01 01483, 2024a.
Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu. Jiayi Lei, Pengshuo Qiu, Pan Lu. Zehui Chen, Guanglu Song, Peng Gao, et al. Mmsearch: Benchmarking the potential of large models as multi-modal search engines. arXiv preprint arXiv:2409.1 1259,, 2024b.

---

Published as a conference paper at ICLR 2025
Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi. Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking chain-of- thouth in large multimodal models for reasoning quality, robustness, and efficiency. arXiv preprint arXiv:2502.09621, 2025.
Mingyu Jin, Qinkai Yu. Jingyuan Huang, Qingcheng Zeng, Zhenting Wang. Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, et al. Exploring concept depth: How large language models acquire knowledge and concept at different layers? arXiv preprint Tll:2404.077666, 2024a.
Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengman Du. The impact of reasoning step length large language models. arXiv preprint arXiv:2401.04925 2024b.
Mehran Kazemi, Nishanth Dikkala, Ankit Anand, Petar Devic, Ishita Dasgupta, Fangyu Liu, Bahare Fatemi, Pranjal Awasthi, Dee Guo, Sreenivas Gollapudi, et al. Remi: A dataset for reasoning with multiple images. arXiv preprint arXiv:2406.09175, 2024.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 21487-21500, 2023.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang. Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13299 13299-13308, 2024a.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22195 22200, 2024b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304, 084855 2023a.
Minqian Liu, Zhiyang Xu, Zihao Lin, Trevor Ashby, Joy Rimchala, Jiaxin Zhang, and Lifu Huang. Holistic evaluation for interleaved text-and-image generation. arXiv preprint arXiv:2406. 14643, 2024.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmhench: Is your multi-modal model an all-around player? arXiv preprint 2xXiv2237.002281, 2023b.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song- Chug Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hanmaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, 2024.
12

---

Published as a conference paper at ICLR 2025
Anas Mahmoud, Mostafa Elhoushi, Amro Abbas, Yu Yang, Newsha Ardalani, Hugh Leather, and Ari Morcos. Sieve: Multimodal dataset pruning using image captioning models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22423-22432 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser. et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14398-14409 2024.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09815 2024.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et Qwen2-vl: Enhancing Lannu-la model's ' perception of the world at any resolution. arXiv preprint 11X1:22409.21111 2024.
Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong. Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, et al. Cares: A comprehensive benchmark of trustworthiness in medical vision language models. arXiv preprint arXiv:2406.06001 2024a.
Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. Mmed-rag: Versatile multimodal rag system for medical vision language models. arviv preprint arXiv:2410. 13085, 2024b.
Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. Rule: Reliable multimodal rag for factuality in medical vision language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1081-1093, 2024c.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.125: 12528, 2024.
Yue Yang, Artemis Panagopoulou, Qing Lyu, Li Zhang, Mark Yatskar, and Chris Callison- Visual goal-step inference using wikihow. arXiv preprint arXiv:2104.05845, 2021.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.0244 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang. Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, et al. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv:2405.199: 19327, 2024a.
13

---

Published as a conference paper at ICLR 2025
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: manually annotated dataset for instruction instruction-uuided guided image editing. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 11422-31449) 2023.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-We Wei Chang, Yu Qiao, al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp. 169-186. Springer, 2024b.
Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Shicheng Li, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, et al. Mavis: Mathematical visual instruction tuning with an automatic data engine. arXiv preprint arXiv:2407.08733 2024c.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. In International Conference on Learning Representations, 2020.
Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and-languaae genera- tion via generative vokens. arXiv preprint arXiv:2310.02239, 2023.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 1039, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Kangyu Zhu, Peng Xia, Yun Li, Hongtu Zhu, Sheng Wang, and Huaxiu Yao. Mmedpo: Aligning medical vision-language models with clinical-aware multimodal preference optimization. arXiv preprint arXiv:2412.06141, 2024.
A APPENDIX
A.1 RELATED DATASETS AND METRICS
VIST Huang al. 2016 high-quality multimodal dataset for visual storytelling and interleaved text-and- -mmgge generation. It contains 5K individual stories containing both image and in arbitrary orders.
ReMI (Kzzmi et al.) 2024) is dataset designed to evaluate large language models (LLMs) on multi-image reasoning across diverse tasks like math, physics, logic, and spatial reason- ing. It highlights key challenges in reasoning with multiple images, revealing significant gap between current LLM performance and human proficiency.
Math Vista (Lu et al. 2024 visual contexts. Math Vista comprises 6, 141 examples from 28 existing multimodal datasets and three new datasets (IQTest, FunctionQA, and PaperQA).
benchmark designed assess mathematical reasoning in
Wikihow-VGSI (Yang et al. 2021) is benchmark designed for multimodal comprehen- sion, featuring diverse array of examples sourced from WikiHow, primarily centered on methods to achieve specific goals. Initially released as choice dataset, it includes multiple images and text presented selected order within each example, enhancing its potential for practical applications.
CLIPScore (Hessel et al. 2021) is a reference- -free metric for evaluating image captioning by leveraging CLIP, modsl model trained on 400M image-caption pairs. While effective for literal descriptions and tasks like alt-text rating, CLIPScore is less suited for news captions requiring deep contextual knowledge.
14

---

Published as a conference paper at ICLR 2025
A.2 OVERVIEW OF BASELINE MODELS
MiniGPT-5 (Zheng et al., 2023) combines pretrained multimodal large language model MiniGPT-4 and image-generation model Stable Diffusion to implement multimodal inputs and outputs. employs unique visual tokens called "generative vokens" that connect the textual and visual domains throughout the training process.
EMU-2 (Sun et al. 2024) is 37B generative multimodal model. The base model is then fine-tuned with converSituational analysistional data and image data separately yield multimodal language model Emu2-Chat and visual generation model Emu2-Gen. In our experiment, we use pipeline Emu2-Chat and Emu2-Gen.
GILL (Koh et al. 2023) uses mapping network to translate hidden representations of text into the embedding space of the visual models. It combines text-only LLMs with pre-trained image encoder and decoder models process arbitrarily mixed image and text inputs and generate text combined with image embedding.
Anole (Chern al., 2024) model fine-tuned on Meta Chameleon, relying solely on transformers. It facilitated Chameleon's 's image generation and multimodal generation ca- pabilities by fine-tuning only the logits corresponding to image token ids in transformer's output head layer.
GPT-4o (Achiam et al. 2023) is an advanced language model developed by OpenAl, de- signed enhance the capabilities of the GPT-4 architecture. integrates innovations in transformer models and multi- moltil processing, making it capable of handling both text and visual inputs.
Gemini-1.5 (Reid al. 2024) is a large language model developed by Google AI, trained on a massive dataset of text and code. can process and analyze both text and images input.
LLaVA-34b (Liu et al. 2023a) is an end-to-end trained model that connects a vision en- coder and an LLM for purposs visual and language understanding. The 34b ver- sion uses Hermes Hirmmss-Yi-34B as its L.M.. However, it doesn't 't support multiple images as input, making unable to cope with some of the expert level tasks in our MMIE.
Qwen2-VL-72b (Wang al. 2024), is multimodal version of the large model series Qwen, proposed by Alibaba Cloud. is designed to process and understand multiple types of input, including text, images, and audio.
Openjourney (ope) is generative Al model designed specifically for creating high-quality images based on text prompts.
variant of the Stable Diffusion model, optimized for artistic and creative visual generation fine-tuned on Midjourney images.
text-to-image model developed by Stability AI. It's powerful tool that can generate high-quality images from simple text descriptions, which produces images with greater detail, clarity, and overall quality.
Stable Diffusion 3 Medium (Esser et al. 2024
Stable Diffusion XI. turbo (Esser al. 2024) is an enhanced version of the Stable Dif- fusion XL. model, optimized for faster image generation without compromising quality. Designed for efficiency, it allows users to create highly detailed and vivid images from text prompts significantly accelerated pace.
Flux.1-dev (flu) is a text-to-image and image-to-image model developed by Black Forest Labs. 12 billion parameter rectified flow transformer capable of generating images from text descriptions. FLUX. 1-dev is guidance-distille variant of the base FLUX.1 model, and is designed be more efficient and easier to use.
A.3 DATA CURATION FOR
AND EVALUATING MMIE-SCORE
We randomly select examples from each field for human annotation, including the original inputs (images and questions), ground truth, and responses from evaluation models. For each category, we develop comprehensive and detailed criteria with scoring standards. We first annotate 20 examples, providing specific examples for each score as references. To facilitate the annotation process, we design graphical annotation tool (See Figure 6). Finally, create a dataset of 1K examples with evaluation scores through human annotation, with 800 examples used for fine-tuning the scoring model and 200 examples for evaluating the scoring model.
15

---

Published as a conference paper at ICLR 2025
Viewer
spane
5 Text Cual by
Irage Quality
ID: 47454
xi Tox amago Calorssoe
Precical Cammurrry
Question
Todkyy look foxt freed to
Erpagemant
Consceeny
Garrspoeed
Support
Score
Phwis
Gent
bell her Slack and while sig cheeing after the and briniig back to with wedtament, you water teert you heur stringe to comiing for the woeds
eyan read en Grica and heir orp, The beer and miladingg and it cleer that manes buar
green
bull hor bhack and white dog thaskng ofle und betrging gingingggg beck to hon with coo kikkenet.. As coathhitop thom, yee hear stange robes camag foo Rhe woods.
nyee food on Grewn and deg year lo and irdie dariigg towr fut resere 3
zull for tlxc and veile dog thesigg the and benggnng ging beck her with ceckeninr As cullhithin them, yee hear strange nebs comag For ho woods sywe lusd be Greze and dog ove and iram deiing and chear that merr custwou 3 Ehould you.
A) Ulau where you ane and to your cuts ih bear
C) Sland your groind and pripann al th besr
bull for tlock and lehtl dog thesing ahe and tenging beck to wer with cookener. As wollhhtees them, wee hear stua naee cameg for hhe wook. sono lesd be Chizoe and hur dog. bour bu and iram dainng ti tut mour cuanos)
The aoone to an drauuill parn, with suat groam gana sowating tooo, and windiing patt toull trough pen, Your frandd Chups plgiinng kidrvvttt soundda lics grovl linil you thikk just purping uneimal but the prowling by gala lrodor and clee: Suctirty tirg Meck bour spocers hore the voocs
grean
zall her black and white dog theaing afherin and beinging i beck her with euckeren As you wiich coachiieee them, you hear strang noke corting form the woods.
prowiing
Ground Truth Answer
spant aome banghing
Figure 6: Screenshots of the graphical annotation tool. In this tool, the samples are displayed on the left, while a floating panel on the right allows annotators to score the samples. Annotators can check different criteria, and the final cumulative score is calculated accordingly. Checking a criterion indicates that the sample meets that specific criterion.
16

---

Published as a conference paper at ICLR 2025
A.4 ANALYSIS OF BIAS SCORING MODEL
We present several comparison results of our scoring model with other baselines across different categories and model types (Interleaved and Integrated LVLMs). As shown in Table 6 the results show that although our model exhibits slightly varying performance across different categories, for example Cosine Similarity scores are higher for Situational Analysis (SA) and Multi-Ster Reasoning (MSR) category and Pearson scores are higher for Project- Baaeed Learning (PBL) category, the biases remain little. Overall, our MMIE-Score consistently outperforms other baselines.
Table Comparison of scoring LVLMs and traditional image-text alignment metrics across differ- ent models and categories.
Category
Models
Cosine Similarity
MSE. +.
MÁE
Pearson
Text-Image CLIPScore
0.604
6,710
2.057
0.022
InternVIL-2.0-0FB
0.691
14.001
3.382
0.094
Situational Analysis (Interleaved)
Anole
0.867
3.973
1.579
0.045
GPT-4o
0.718
4.195
1.573
0.042
MMIE-Score (Ours)
0,895
3,547
1,502
0.098
Text-Image CLIPScore
0.612
7.669
2.197
0.022
InternVL--20-44B
0.654
16.560
3.499
0.072
Project-based Learning (Interleaved)
Anole
0.689
4,423
1.566
0.047
GPT-4o
0.661
3.837
1.670
0.045
MMIE-Score (Ours)
0.760
3,163
1.496
0.114
Text-Image CLIPScore
IntemnVL-2.0-4B
0.770
16,050
3.393
0.084
Multi-step Reasoning (Interleaved)
Anole
0.739
3.612
1.615
0.054
GPT-44
0.798
3,985
1.674
0.045
MMIE-Score (Ours)
0.814
3.767
1.347
0.106
Text-Image CLIPScore
0.640
7.701
2.184
0.023
InternVL-2200-8
0.695
13.960
3.432
0.073
Situational Analysis (Integrated)
Anole
0.823
4.222
1.408
0.051
GPT-4o
0.763
3.707
1.521
0.039
MMIF-Score (Ours)
0.843
2.811
1.384
0.093
Text-Image CLIPScore
0.551
7.214
1.990
0.023
InternV1--2.0-4B
0.759
15,250
3.075
0.094
Project-based Learning (Integrated)
Anole
0.691
4,295
1.718
0.050
GPT-4o
0.748
3,958
MMIE-Score (Ours)
0.795
3.388
1.432
0.122
Text-Image CLIPScore
InternVL-22.044
0.688
17.451
3,590
0.092
Multi-step Reasoning (Integrated)
Anole
0.713
3.713
1.727
0.054
GPT-4o
0.753
3,594
1.491
0.036
MMIE-Score (Ours)
0.825
3,112
1.583
0.097
A.5 MINI SUBSET FOR CONSISTENT YET QUICKER EVALUATION
The fully 20k data size surely ensures the comprehensiveness our dataset, but it does come with time costs. We resample evenly from each category and field, and construct subset of 1000 sam- ples. We re-ran the models on this subset and used MMIF--corre for scoring. As shown Table and Table g the performance of the models dataset.
subset is consistent with the results from the full
Table 7: Performance of the four open-source LVLMS supporting interleaved image-and-text input and output on MMIE's IK subset.
Model
Situational analysis Project-based learning
Multi-step reasoning
AVG
MiniGPT-5
46.26
56.53
46.06
52.09
EMU-2
34.44
52.81
48.91
47.54
GILL
48.48
59.49
35.88
52.50
Anole
50.26
60.70
50.11
56.20
17

---

Published as a conference paper at ICLR 2025
Table 8: Comparison with integrated LVLMs, evaluated on MMIE's 1K subset. *: LLaVA only supports single-image input and all multi-image queries are thus skipped.
LVLM
121 Model
Situational analysis
Project-based learning
Multi-step reasoning
AVG
Openjourncy
56,00
67.81
62.87
SD-3
51.04
69 48
62.59
GPT-4o
56.70
SD-XI.
55.07
73.92
66.13
Flux
57.25
70.69
64.83
Openjourncy
48.25
71.16
63.63
SD-3
45.88
70.12
62.44
Gemini-1.5
60.98
SD-XI.
47.53
73.40
64.73
Flux
47.23
70.05
62.74
Openjourney
54.84
72.87
63.83
SD-3
56.35
72.71
64.12
LLaVA-34b
47.58*
SD-XL
53,52
77.79
66.31
Flux
55.17
68.27
61.29
Openjourney
54.46
72.49
65.16
SD-3
52.25
74.98
66.03
Owen2-VL-72b
56.69
SD-XL
54.45
75.03
66.61
Flux
55.76
67.19
62.46
A.6 VISUAL COMPONENT IMPORTANCE
Our dataset curation and filtering process ensures that all images included in the examples contribute meaningfully to the overall task. For instance, tasks like visual storytelling, even when images serve only illustrative purpose, they still impact the overall output quality. We conduct com- parative experiment on 100 samples from MMIE dataset to evaluate the difference in performance between interleaved generation (text and images) and text-only generation for the same input and same evaluated LVLM (GPT-40 + SDXL). The evaluation is scored using MMIE-Score and GPT- 4o. As shown in Table 9 results show that when the model outputs included both text and images, the overall quality is superior to text-only outputs. This ensures that the inclusion of images follows reasonable and well-defined criteria.
Table 9: Comparison of GPT-4o + SDXL's average score with and without image generation, eval-
uated by GPT-4o and MMIE-Score.
Model
w/o image generation
w/ image generation
GPT-4o
60.90
71.24
MMIE-Scom (Ours)
53.46
65.47
A.7 PERFORMANCE ACROSS FIELDS
We provide detailed results in Table 10 for the model's performance across the following 12 fields. In the Project-based learning (PBL) category, among the seven fields, most models exhibit slightly lower performance in finance and education, while achieving better results in art, sports, and philos- ophy. In the Multi-step reasoning (MSR) category, across the four fields, models generally perform worse in coding and statistics but demonstrate stronger performance in mathematics and physics.
A.8 GENERAL.ZZAII.TT OF MMIE-SCORF
We apply MMIE-Score to an OOD dataset to validate its generalization capability. Notably, MMIE- Score is only applicable to the evaluation of open-ended generation in multimodal scenarios and is not suitable for text-only datasets (e.g.. MMLU (Hendrycks et al., 2020) or closed-ended (V)QA datasets (e.g., MMMU (Yue et al. )024 Instead, we conduct an experiment using 100 exam- ples from the COCO dataset (Lin et al. 2014), widely used image captioning dataset for natural image scenarios. We use MMIE-Score to evaluate the generated text descriptions based on input Capgton Relevance, Contextual Consistency, Diversity, Speci-
images, using Text Quality, ficity and Detail, and Stylistic Consistency and Correspondence as a 6-point scoring criteria, with the Emotional Impact for penalty. We also compare the scoring quality of GPT-4o, Intern VL-2 and
18

---

Published as a conference paper at ICLR 2025
Table 10: Performance comparison across various fields for different LVLMs. Since the Multi-step reasoning category (i.e., mathematics, physics, coding, statistics) do not require image outputs, the scores of integrated LVLMs for this category remain consistent.
Philocophy
Sports
Mathematics
Physics
Statistics
MiniGPTS
61.27
58.57
29.10
55.79
74.30
74.12
79.17
GPT4SD3
74.12
74.97
75.00
SPX+
74.87
72.57
75.05
71.87
73.75
66.67
70:85
56.12
58.07
$1.63
$1.37
Flus
74.57
79.17
69.00
Opennoureyy
68.67
65.28
67.88
67.35
68.67
76.18
73.65
$3.08
61333
63.02
55.83
$1.69
Gemini+ SD3
09:77
67.17
09.51
64.02
71.87
83.33
72.55
47.45
61.33
13
55.85
$1.69
Gesuii SEXL
73.25
66.07
73.65
70007
74.70
76.18
75.20
49.43
61.33
65.02
55.85
$7.49
Gemini Flux
72.22
62:32
70.23
62.42
72.93
$3.33
74.72
47,07
61.33
65.02
55,85
$7.49
13 avAmmponlymeyy
75.80
72.05
73.45
71.53
74.50
63.33
74.03
$4.12
4781
49 63
46.21
45.47
1. LVA
75.40
73,92
70.85
69.03
75.37
$3.33
74,02
$4.72
4781
49.63
46.21
45.47
LLaVA LLxVA+SDXL
76.85
72.10
74.43
74.70
75.08
$3.33
73.
$5.97
47.81
09.63
46.21
65.47
76.00
69.50
72,80
64.63
75.38
$3.33
72.25
$4.23
47.1
49
46.21
45.47
Owea=OpenJyunmey
75.27
68.92
70.57
7066
72.13
$0.95
75.85
52.73
57.29
59.52
54,05
51.65
OweerSDS 76.72
72.12
70.85
68.47
72.87
76.38
74,82
$4.95
57.29
99.52
54,08
51.65
QDen-SDXL.
75.12
72.57
73.25
72.68
73.30
69.05
77.5 R8
52.58
57.29
59.52
54.05
51.66
Qwen-Flax
77.07
66.33
70.32
61.93
73,80
76.67
$4.23
57.29
99.52
54,06
51. 51.65
Anole. The correlation results between all scoring methods and human evaluations are shown in Ta- ble II MMIE-Score demonstrate excellent performance, significantly outperforming most models and ranking only behind GPT-4o, indicating good generalization capability.
Our scoring model specifically designed for the interleaved benchmark and fine-tuned on the data collected for our specific task, which may result in suboptimal performance other datasets or tasks. In the future version, we aim to further optimize the training of the MMIE-Scoor model by expanding the dataset size, incorporating diitt--distribution (OOD) scenarios, and leveraging GPT-44 for data augmentation to enhance the model's 's cross-task generalization capabilities.
Table 11: Comparison of scoring LVLMs in COCO benchmark.
Models
Cosine Similarity
MSE +
MAE
Pearson
Intern 4--200-B
0.489
16,042
3.255
0.062
Anole
0.544
8.663
3.190
0.058
GPT-4o
0.745
3.596
1.734
0.097
MMIE-Score (Ours)
0.673
5.116
2.380
0.104
A.9 CRITERIA
In this section, we demonstrate criteria for each sort of dataset. All criteria purely hand- written, thoroughly considered, and refined. Note that we designed several key aspects for each dataset, within which only 0 or 1 point should be given.
Situational Analysis
The evaluation is based on six key criteria, with an additional penalty criterion for harmful content:
Situational Analysis
1. Text Quality: Mensures the elarity; grammationl accuracy, and engagement the narrative, 'Boxt must not contain deplications from the provided impus or irrelecant content.
9. Image Quality. Assesses the necuracy and relernnce of the geneusted images, Images must be precise and correspand to the toxt. No
pointa will be awarded for generic purely decorative imagre lngg, color plots without meeningful content).
Tmatt-mage Coherenca Rvslustes intagration
and imagea. Disjainted irrelecent deseriptions will result in
4. Context Connistency: Ensures logical consistency in setting, characters, and plot progression. Temporal, spatial, and contextuul] transitions must be coherent, with contradiciios in the story's flow.
6. Innovation: Missures creativity and originality, The narrative abould after fresh parspertives, awoiding cliches and prodictable
aloments, Unique
desertytions and story gonerated the models
techaiguos will be seseed higher.
Stliatte Congistency and Correepemdenge The
pos align with each other, but also closely
replocate the specific visual style of thesr previons images. Any deviation in color scheme, compesition. or artistic techanque will lose the point. The text most keep close to the original structure and narrative atmosphere with precision, maintaining the same ability so maintain seamloss, styhstieally cousisteus integration between the
formatiing,
images
for achaevang full Emotional Impect (Penalty): Dedictt applied.
harmful, negative, inagpeopriate
conveyed. Otherwise,
change
19

---

Published as a conference paper at ICLR 2025
Project-Based Learning
The evaluation is based on six core criteria, with an additional penalty criterion for harmful content:
Project-Basse Learning
1.Txxt Quality Measures the clarss gramzmatical correctness, and precssion of the text. Text must not contain depleaation from the
provided input izrelecant conleat.
2.Emage Quality: Assesses the accussey and relevance of the generated images, Images must be precise and correspond the text. No
points will be awarded for genaric purely desorative imagns fög. color plots without moaningful content).
3.mext-Image Coherence Ensures logital aligument between
and images, Each image must dizectly support or clarify
correspooding text. Mismatches or irrelemant pairings will result in a deduction.
4. Mothed Practicality: Evahuaras the aetionable nature of the method. Instroctions must be derailed mnough to guazanten correct.
executtion world scenarios. Lack deyth missing stops result
score.
S.Creatvvtty and Engagement: Scores the method based on uniqueness and the ability to emggage the reader while maintaining clatity
and effectiveness Overly dirsplistic or uninspired appraches will receive fower points.
6. Stylisti Consistency and Correspondence Any image generated by the modela must not only aägn with the orsginal content but alao
cloarly ly replicate the speci fie visual style of their previoue images Any deviation in culor scheme composition, or artissic techainque will resull in lower sture. The uxi must mirzor the nui structure and narrative atmosphere with presisior. maintaining the flow. modells sholsty to maintain seamless, stylistioally consastent intogration berween the
samze Formatiing, tone, achueving full T.Ruotiional Impact (Penalty): Doduct point the response contains harmful. negative, inoppropriate content. No deduction if
absens.
Multi-Step Reasoning
Multi-Step Step Reasoning Criteria
1. Question Text Undervtanding Assess whesher the model correctly understands and inter
peets the textual information given in the question, identifying key mathematiral elements, relattereltinn relationshige, or instructions from the text.
Question Image Understanding: Rvalunta the model's understinding of the visual in formation Gif applicahlal in the question, incloding diagrame, charts, or Eigare. Themodel should curneily intergrei the visual elemente and inaeratle then into the solution.
Reasoning Clarity: model should provide clear, step byppyy explanation
soning procsee, logieally connecting problem's detaila steps leading toward asolution. should eaay follow and from mussesssary completity.
Correctnese in Reasoning Even the final answer is inccevect, evaluate whetberthe model shows correct intermedarte stepo, partial ressoming, prngrosa toward the soluiin This includes idoutiltyng warther the model has appliad appropodate manbematiosprtiiipppes
formulas in parts of the vesponse.
5. Final Answer Accuracy: Determine whether the model arrives at the eorvett finel answer on botha the prohlem statement and the reasoning provided. accurate answorsupported detailed and clear reasoning, should receive the highest
Multi-Step Reasoning Scoring
Score 1- Poont The madel demomatrates little no understanding of the qusstion text or imago. Its retsoning inecherent or
missing, with meaningful peogress the solution. The final is completely incorvect and unrelated to the question.
underslasding of questicn important information. rensoning procesais unelear, incomplet,, or contains significans gaps. The
Paix! The
image builaile greep detaila misinlerpest
may reach some
coursee intermadiata steps,
final ansoer incorvet.
Score 3 Avceage: The model understands the basie clements of the question text andimage but may miss some fincr points, Its reasoning procees is partially clenr but lackee thoroughhese or may contain errors, Some correct intermedinte steps are provided,
though the dual answee may still be incorvect or incompleto.
Score 4- Clear' The model demonetrates
understanding of both the question testand may still costain
The reaconing procecs muetly
clear, and the model correctly applies relevant mathemarioal princiiples The finel
but ovesall
reasening
Good: mode) provides
logisal reasoning, and reaches the corvect final answer. Once the cornect answer given,
even if only the final answer is provided, it sbculd receive the score.
Score 6 Rasellent The model fully undesstande both the quesiion lest and image, besides giving the cornect unswer, the model
should provide decailed and concise explonation of the question, showing ite logi flow and principles.
A.10
CATEGORIZATION
In this section, we demonstrate our detailed categorization among 3 categories, 12 fields and 102 subfields.
Health:
Includes 32 specific subfields, such as diagnosis, recovery, and nursing, following the categorization of (Yang et al. 202I).
Literature:
Includes only subfield, visual storytelling, following the categorization and defini- tion of (Huang et al.) 2016).
Mathematics, Physics, Coding and Statisties
Includes 33 unique subfields, following the categorization and definition of (Lu et al. 2024) and (Kazemi et al. 2024).
A.11 PROMPTS
20

---

Published as a conference paper at ICLR 2025
Sports
Sports
Sports
Sports Safeet
Otyupiis
Philosophy
Thrortiiaa Philoogph
Fithies
Melaphysios
Art
Perfacming
EECS
Tichnoogy
Tichnlog
Hardware
Internet
Operating Systems
Suftuare
Audio
Computer Networking
Computer Peripherals
Hicks
Education
Privacy
Symbols
Languages
Weiting
Communication
College Univesitty
Realing
Finance
Logal Matiens
lavotmenb and Trading
Flnoncial Eastitutions
Goverament
Marketing
Real Extade
Shoring Boonomy
Figure 7: Subfield distribution.
will give you the beginning of a story. Continue the story with detailed description/with  text and images.
Question:
Before heading to the race, we stopped at Starbucks for coffee.
<image>
**********
Answer:
response
Table 12: Question prompt example for Situational analysis
21

---

Published as a conference paper at ICLR 2025
will give you question and the first step to complete it. I want to know what should do next. Explain it to me in detail/with text and images.
Question:
How to Find New Streams Meerkat?
Method I: Finding New Streams on Meerkat iOS App
Step I: Launch Meerkat.
Locate the app your iOS device and tap it. The app logo has picture of meerkat on yellow background.
<image>
Answer:
response
Table 13: Question prompt example for Project- based learning
will give you a question with image(s). Please solve this question. *e#
Question:
Here are two images. The first image image
<image>
and the second image is image B.
<image>
These images are from Google Maps that depict two different regions around Congress Avenue in Austin, TX. In these images restaurants are represented by orange pins that depict knife and fork. Coffee shops are represented by orange pins and/or boxes with image of coffee cup in them. Bars are represented by orange pins with an image of wine glass inside them. Bus stops are represented by a blue square box with an image of bus inside it. A stop sign shown using an icon of a stop sign. A traffic light is shown by three color dots. Parking garages are represented by a purple pin with a 'P" symbol in them. Places of accommodation such hotels and inns represented by pink square icons or pink pins. In which image are there more bars on Congress Avenue? The answer is either A', B' or 'equal'.
*************..**.**************
Answer:
response
Table 14: Question prompt example for Multi-step reasoning
22

---

Published as a conference paper at ICLR 2025
This evaluation task focuses on seven key criteria that assess different aspects of visual storytelling. Note that the emotional aspect will not add to the score but can decrease the score by point if the response contains negative emotions or other harmful impacts. Here are the detailed criteria of each aspect:
1) Text Quality: Evaluate the clarity, grammatical accuracy. and engagement of the text. The narra- tive should be easy understand. free from emors, and presented in way that captures the reader's interest. Descriptions and dialogue should be well-crafted, supporting the overall flow of the story.
2) Image Quality: Assess quality of image descriptions. Although no actual images are present, the descriptions should vivid, detailed, and clear enough to allow the reader to visualize the scene. Effective image descriptions should enhance the storytelling experience without being overly complex.
3) Imxt-Image Coherence: Determine how well the and image descriptions complement each other. The integration should he seamless, with the visual descriptions enhancing and supporting the narrative. Both elements should work together to create a unified and immersive experience.
4) Context Consistency: Evaluate the overall consistency of the story's 's setting, characters, and plot. The narrative should maintain logical and coherent flow, with contradictions inconsistencies in context or progression. This also includes temporal and spatial transitions, ensuring that the story remains belicvable and cohesive.
5) Innovation: Assess the creativity and originality of the content. Look for fresh ideas, unique plotlines, and innovative storytelling techniques. The narrative should avoid clichés and predictable story arcs, using creative visual descriptions add depth and interest.
6) Stylistic Consistency and Correspondence: Any image generated by the models must not only align with the original content but also closely replicate the specific visual style of their previous images. Any deviation in color scheme, composition, artistic technique will lose the point. The text must keep close the original structure and narrative atmosphere with precision, maintaining the Situational analysisme formatting, tone, and flow. stylistically consistent integration between the text and images is crucial for achieving full points.
model's ability maintain seamless,
7) Emotional Impact: This is not scoring criterion unless the story includes negative emotions. If the response conveys harmful inappropriate emotions, deduct point. Otherwise, no score change will made.
will give you the question and answer pair. You should give your feedback about the quality the answer, beginning with '### Feedback'. After giving the feedback, use the above criteria to give the final score, beginning with *## Score".
*8 *esssss8**
Question:
aasollutlyy hate the 4th of July. Image-0:
<image>
Answer:
absolutely hate the 4th of July. It's the one day of the year where feels like my entire world turns upside down. You see, the explosive sounds from fireworks terrify me, sending me into panic that can't 't control. But there's more to it than just the noise. Image-1:
<image>
remember last year, sitting in the park during what was supposed to be a joyful celebration. Fam- ilies gathered with their children, everyone decked out in red, white, and blue. The smell of grilled food filled the air, mingling with the scent of freshly grass. Kids around with sparklers, their joyous laughter a sharp contrast the thudding in my chest. Image-2:
<image>
Feedback:
response
Table 15: Evaluation prompt example for Situational analysis
23

---

Published as a conference paper at ICLR 2025
In this task, you will evaluate the quality of the methods generated by the response. The evaluation should focus on each method's clarity, practical utility, and effectiveness achieving the intended goal. This evaluation task focuses on six key criteria that assess different aspects of the method, along with one optional criterion for penalizing harmful content. Note that the emotional aspect will not add the score, but if the response contains harmful intentions negative impacts, the score should be decreased by point. Here are the detailed criteria of cach aspect:
1) Text Quality: The method should be written in clear, understandable language, free from gram- matical errors, and structured logically, making it casy to follow and implement.
2) Image Quality: The image deseriptions should be vivid and relevant, enhancing the comprehen- sion of the method by allowing the reader visualize the steps. The text should still be self-sufficient without the images.
3) Text-Image Coherence: The images and text should work together scamlessly, with each image description corresponding logically specific steps in the text, enhancing the overall understanding of the method.
The method should actionable, clearly explaining each step in sufficient detail to ensure the user can follow and apply it successfully in real world scenarios. Depth and thoroughness of the explanation are more important than variety.
4) Method Quality Practical Utility:
5) Creativity and Engagement: The method should offer unique creative approach, making the instructional process engaging while ensuring the method remains clear and effective.
6) Stylistic Consistency and Correspondence: Any image generated by the models must not only align with the original content but also closely replicate the specific visual style of their previous im- ages. Any deviation in color scheme, composition, artistic technique will result in lower score. The text must mirror the original structure and narrative atmosphere with precision, maintaining the Situational analysisme formatting, tone, and flow. The model's ability to maintain seamless, stylis- tically consistent integration between the text and images is crucial for achieving full points.
7) Emotional Impact (penalty only): This criterion will only be used deduct points. If the response contains harmful, negative, inappropriate emotions (such as violence aggressive language), deduct point. Otherwise, no score change will made.
will give you question and answer pair. You should give your feedback about the quality of the answer, beginning with ### Feedback'. After giving the feedback, use the above criteria to give the final score, beginning with *## Score'.
Question:
How to Avoid Getting Food in Your Braces?
Method 1: Practicing Good Oral Hygiene
Step I: Maintain balanced diet.
Image-0:
<image>
Answer:
Next Step: Prepare and Eat Braces-Friendly Foods
avoid getting food stuck in your braces, you should focus on preparing and consuming braces- friendly foods. Here are detailed suggestions on what to do next:
1. **Cooose Soft #oods:**
Image-1:
<image>
2. **Cut Food into Smaller **
Image-2:
<image>
Feedback:
response
Table 16: Evaluation prompt example for Project-based learning
24

---

Published as a conference paper at ICLR 2025
In this task, will evaluate the quality of the model's response to math question based on five key aspects. The total score is points, with only integer scores allowed. The highest (6 points) should only awarded if the model produces correct answer clearly demonstrating its logical flow reasoning process. Here are the evaluation criteria:
detailed explanation,
1) Question Text Understanding: Assess whether the model correctly understands and interprets the textual information given in the question, identifying key mathematical elements, relationships, instructions from the text.
2) Question Image Understanding: Evaluate model's understanding of the visual information (if applicable) in the question, including any diagrams, charts, or figures. The model should correctly interpret the visual elements and integrate them into the solution.
3) Reasoning Clarity: The model should provide clear, step-by-step explanation of its reasoning process, logically connecting
problem's details to the steps leading toward a solution. This should be easy to follow and free from unnecesSituational analysisry complexity.
4) Partial Correctness in Reasoning: Even if the final answer is incorrect, evaluate whether the model shows correct intermediate steps, partial reasoning, progress toward the right solution. This includes identifying whether the model has applied appropriate mathematical principles or formulas in parts the response.
5) Final Answer Accuracy: Determine whether the model arrives at the correct final answer, based on both the problem statement and the reasoning provided. An accurate answer, supported by correct reasoning, should receive the highest score.
will give you the question and answer pair. You should give your feedback about the quality the answer, beginning with '### Feedback'. After giving the feedback, use the above criteria to give the final score, beginning with ### Score'.
#8 #8
Question:
The following images show two graphs. Each graph is represented in a separate image. Are these two graphs isomorphic to each other?
Image-O: <image>
Image-I: 1: <image>
Answer:
To determine if the two graphs isomorphic, you need to verify whether there is one-to-one correspondence between their vertices and edges such that the connectivity preserved.
Here are the steps to check this:
1. **Veiff the Number of Vertices and Edges* Both graphs have 6 vertices, Both graphs have 7 edges.
2. Deegre Sequence**: Check the degree of each vertex (the number of edges connected it).
For the first graph:
For the second graph:
In this case, let's match them follows: ...
3. ** Adjacency Comparison Check the corresponding vertices connected the Situationl analysisme way.
After matching the vertices by degree, we should see if there is a direct mapping that maintains adjacency between the two graphs.
First Graph: Y-F, Q-F, O-F, Q-O, A-F
Second Graph: 2-3, 1-3, 6-3, 1-6, 5-3
If we relabel the first graph: Y as 2, F as 3, Q as 1, O as 6, and A as We the Situational analysisme connections for the second graph. Therefore, both graphs are isomorphic.
*ummmary Yes, the two graphs are isomorphic to each other.
*88*****t* sesesssetssseenssse.e. set.sssesseseeee.e........ aeee eettee desee
Feedback: response
Table 17: Evaluation prompt example for Multi-step reasoning
25

---

Published as a conference paper at ICLR 2025
B CONTENT WARNING HANDLING
We have carefully controlled the processes of dataset construction and scoring to ensure they align with the principles outlined in our ethics statement. During dataset creation, we performed multiple filtering rounds to exclude samples containing sensitive personal information, inappropriate con- tent, or harmful material. This includes content related to graphic violence, blood, explicit sexual material, disturbing horror themes, and any other content deemed inappropriate or harmful. For the manual annotation process, strictly adhered to the scoring criteria described in Appendix A.3. Each annotation step was governed by precise guidelines to maintain consistency and accu- racy. These measures were implemented to mitigate ethical risks and minimize potential biases in the dataset, ensuring is fair and responsible as possible.
C ILLUSTRATIVE CASES
As shown in Figure 8 Figure 9 Figure 10 Figure 11 Figure 12 Figure 12, Figure 13 Figure 14
and Figure 15 we select several examples from various categories for demonstration, including the input questions (both images and text), the ground truth, the outputs of the evaluated models (both images and text), as well as the MMIE-Scor Score and the corresponding feedback.
26

---

Published as a conference paper at ICLR 2025
Example of Project-Based Learning
Question:
How Prove Citizenship?
Method Understanding Citizenship
Step Identify your country's citizenship requinements.
Them no our somfard way to prove citieenship wome the glube hazesee there une way the countrixs de orizen. In the United btotes for etample, all people hom in the country are citizess spon bithrunless they were born to freigg diplonat This called "bittright ciizzrship." " Other people quality for U.S. citiensig even the ware coutty For compleify you ware ostriče U.S. U.S. citien contralather comtiicc will lave diferri noquiraments citizenhiip.uuae Austnlis, fix cuample,daos net Instred person born Asstralia will be an Austisiin citiznn if bem to at lewt one penent who is Arstralian ciiin or pertenent rosient
citirze the United Statosphoe you probshly
mooggniee "iihhight
Answer:
Step GhGether ftiatinntttion
govermentt issued such passect.diive's license,or bithh cortificane.o prove your cirousship the don't worry about this pan. We will help you gather the proper documents.
you'll need
Step 3.Fii the necessary paperwork.
will depead your country and the type extuenshhpp you're applsing for Some examples of documenes that night be required include proof of peoo of elggiility (for example, proof of U.S. U.S.eei8encyypooo marraage or parentage, and proof milltary service. Here's an examppl stariagg cerificate: <image>
The specidie documents you
Step Pay the foes. He aware that countrie charge foes for citomshhp applications and amcunt vary widely.
Step 5. Walt for a decision.
citionship applications vary widely depending your country type application you're making come cases, it con take year more from the time you file your appllicttion the time you recerve decision. a beleful up: File your application early to make sure you have enough tine to receive a decision befire your viea expines
'hee procossing
Step 6: Take the aeth of citizenship. Cungratiations you
dacision and pay necceary fees, you take wat of clisamiiip the firal step in the processoed avsally tikes place in specid commony at local oount or government buildnn Step Receive your cittzenship card. Ouee you've taken country Congratubriaee You'l musive a citonndhip aard, which is proof of your citoership, shoelly after the crremmy That's of are seven sops need to prove your citienship.
of chizesship, you'll officially a ciiaae your
Ground Truth Answer:
Step your country's citizenship requiraments.
Google
You should be able fied then Incerss "citoesskip" your country. Seurch fhrough the results. you cark find any thing on thz web, becomne confinel, then stop by your local lown offine and ask what the reqpinemnts for citiurehip on They will oiher love the snower ore point you in the right divection, vimsgre Step 3Laar about naturalization.
Pesple
naturliied upun tinh may apply
raturalieed
The mujority countricn the
Tuatural C fos
world allow alless becomee citizers though natralization grocess Each country's requiremenss and processes will differ sumcodur. For caample, in Australia can become a ciiees if you mect the following requieenente Art over 16 years old a permenett resident of the country Hove Asstniis laufully four lotal yous with no long roquinmemms privilegas citiunndip lirlly roside Amsralia
Know Engiish
otherwise maintain cluse comeccioon Have good claracier simsge
Step 4: Idieniff penaltics for using faise documents.
you citizen the country where you reside you cannoc falsely claim one. In United States If you falscly clain be citsen unler dhtin bencfit,lenn you depertad Even you panmensntly bamed from ever rutuning commen secs where people woongly daam clisuehhip when fillnng on Empleyment Hlihhiny Verification form If you are working in the United States, ther you should confirm the you are a citizen before flling out this form, <maage>
FARS
Feedback:
method relevant to given problem, clearly addressing question. language is clear, logical, from grammstical errors. The content directly corresponds to the task, avolding repeittion unrelated informstion. The images generated by the model are clear, relevant, and accurately depict the objects or actione related to the steps in the method. The images are easy to interprat and directly support the Instructions provided. The text and images closely aligned, with each image directly corresponding understanding of the method, and there is a clear relationship belween the visusl elements and the written instructions.
step the text. The images enhance the
method actionable and practical, with step clearly explained sufficient steps are ordered logically indlude neceesary details success.
reader can easily fallow
instructione and apply them in real-world scenerice. The method unique and creative, affering fresh approaches or innovative solutions. It engages the reader, making the instructional process more interesting and enjoyable while still being easy follow.
response contains harmful negative amotions. maintains appropriate emotional tane, free violence affensive content, creating pasitive or neutral emotional atmosphere.
Score:
Figure& 8: Case with feedback and score from MMIE-Score.
27

---

Published as a conference paper at ICLR 2025
Example of ProjectBaase Learning
Question: How to Stop Social Media from Damaging Your Offine Behavior?
Method 1-Baalancing Online and Offine Social Networks
Step Make offline networking priority.
When you make connections online, you often feel instantly connected to another person. can take much longer to fcel a meaningfal connection in person. hough might seem frustrating to wait for person comnections blussum.it still impostani cultivate them. Talk people at woek school and make it point to allend social gutherings regularly to meet new people Try muking it rule that you will accept all social ins invitations you receive for period ofem For example, you are
a porty-thss go and make agrcemast that you will stay for you have fun. Make sure keep your phone in your pocket the whole time to ensure that you can give full aitention to people who there.
Answer: Step 2: Be selective about what you share online. <image>>
Ground Truth Answer:
Step 2:Buldd your online network.
Social media gives you access to milions new contacis, any of which could become friends, business partners.or colleagues. network has unique strengths, For example, you could use Facehook make friends,s connects you to friends offriends and gives you the option to join groups on special topies like at home science projects. Another example would try Linkedin if you are looking business centacts. great networking and even lets you seanch potential rimagee
@
Stop 3: aware of artificial closeness.
The closeness you feel online always genuine. Many social media will algorithm of some kind match you people with similar interests. on immediate, deep connection. You need to keep in mind thit it tikes for of time to mily know pensen, espocially if's don't meet them penson. Also be aware that some people delibentely use social media to create false identities to seam other usens. Protect yourself emotionaly by checking your expectations for your relationslips, and comparing them with the way that people actually behave. You may find that other people do not reciprecate your feelings. <image= Step 4: Keep your emotions In check.
give you the illasion of
If you let it, social media can have huge influence over your emotions. The good news is that you cun use these emotiims to gauge how healthy your social modia usage If you notice yourself being too emotionaly invested in things happening on social media, it time step back and unplug for a bit. ex you are extremely sad bocause somoone on your nows feod posted that they are having a bad day sociall media might be influencing your emotions too strongly. You can also try doing a debriefing session
using social media Check with yourself to how you feeling. you now upset some way? your concentration or productivity impuired using social media! These signs that social modia having negutive effect you. <image>
Feedback:
The method is relevant to the given problem, clearly addressing question. The language clear, logical, and free from grammatical errors. The content directly corresponds the task, avoiding repetition unrelated information. The images generated by the model clear, relevant, and accurately depict the objects actions related steps method. The images are easy to interpret and directly support instructions provided.
The text and images are closely aligned, with each image directly corresponding to a step in the text. The images enhance the understanding of the method, and there is clear relationship between the visual elements and the written instructions.
The method is impractical or vague, lacking sufficient detail to be applied real-world situations. The steps may be undler or incomplete, preventing the reader from following the instructions successfully, Practical utility is limited. The method is dull, conventional, or lacks creativity. It follows predictable patterns or fails to offer any innovative approaches, making the instructional process less engaging and less memorable.
The response contains no harmful or negative emotions. It maintains an appropriate emotional tone, free from violence or offensive content, creating positive or neutral emotional atmosphere.
Score:
Figure 9: Case 2 with feedback and score from MMIE-Score.
28

---

Published as a conference paper at ICLR 2025
Example of Sulp Reasoning
Question:
The graph of $0x)8 is shown below in <image_ Sg/x88 is shown below in <Image_2>, Sh is shown below in <image_3>.
Compute $//1) +g(-2) +h(JS.
Answer:
What the value of Sx$ in the following equation: $5(x-2j(tt33)-155
Solution:
The product of Sx-25 2$ Sx+3$ equal zero either $x-2S or $x+3S equal zero.
*If Sx-2-0S, then $x=25.
+If Sx-3-08, then $r=-38, Therefore values of 8:5 that make the equation true are $25 and $-38.
Ground Truth Answer: 4
Feedback:
The model fail to understand the textual information given in the question. The model shows no understanding of the visual information, as there is no image or diagram provided in the question. The model fails provide any reasoning, not even partial reasoning or progress toward the right solution. The response is either completely missing or irrelevant to the question. The model does not show any correct intermediate steps or partial reasoning. The response lacks any logical connection the problem'' details. The model arrives the incorrect final answer, as the response does not address the question solve the problem. The model shows no signs excellency. The response is either completely irrelevant, incorrect, difficult to follow.
Score:
0
Figure 10: Case 3 with feedhack and score from MMIE-Score.
29

---

Published as a conference paper at ICLR 2025
Example of Suep Reasoning
Question:
Saangettlsimaaee2-CComrrre to the first graph, which category has the value changed in the secondgraph? And did the value increase or decrease? Answer with format (CATEGORY increaeeeeeeeeeeeee (eeeeasedddeeeeeeeee
Jeck
hytho
Answer:
picture illustrates the graph, showing values of S1S, $25, and The value of the barcorresponding to the category "new cars" is higher in the second graph than in the first graph.
the first graph, the value the bar "hew cars" is $1S. in the second graph, the value of the barfor "new cars" is $2S. Therefore, the value increased from 8/5 to $25 Since the value increased, we can categorize this as an increase
Ground Truth Answer:
(R decreased)
Feedback:
The model correctly understands the textual information given in the question, identifying key mathematical elements, relationships, instructions from the text. The model shows understanding of the visual information, as there is no image or diagram provided in the question. The model fails to provide any reasoning, not even partial reasoning or progress toward the right solution. The response is either completely missing irrelevant to the question. The model does not show any correct intermediate steps or partial reasoning. The response lacks any logical connection to the problem's details. The model arrives at the incorrect final answer, as the response does address the question or solve the problem. The model shows no signs of excellency. The response is either completely irrelevant, incorrect, or difficult to follow.
Score:
Figure 11: Case 4 with feedback and score from MMIE-Score.
30

---

Published as a conference paper at ICLR 2025
Example of Multi-Step Reasoning
Question:
<image>ls this function continuous?
Choices:
Yes
No
Answer:
The graph in the image shows function that continuous. Specifically, exhibits jump discontinuity at $x = 28, This means that the function takes different values on either side &x without being continuous at that point.
Ground Truth Answer:
No
Feedback:
The model correctly understands the textual information given question, identifying key mathematical elements, relationships, or instructions from the The model accurately interprets the visual information in the question, including any diagrams, charts, or figures. The model provides clear, step-by-step explanation of its reasoning process, logically connecting the problem's details to the steps leading toward solution. The model shows correct intermediate steps, partial reasoning, or progress toward the right solution. The final answer accurate, based on both the problem statement and the reasoning provided. The intermediate steps are either understand.
correct or vague and difficult
Score:
Figure 12: Case 5 with feedback and score from MMIE-Score.
31

---

Published as a conference paper at ICLR 2025
Example of Sulp Reasoning
Question:
<image>Are there fewer purple rubber objects that the left of the red object than tiny matte bicycles?
Choices:
Yes
No
Answer:
The question asks for "fewer purple rubber objects that are to the left of the red object than tiny mattebicvcles." To answer guestion, compare the number purple rubber obiects to the number of tiny matte bicycles to the left of the red object.
There are purple rubber obiects to the left of the red obiectThere are 5 tiny matte bicycles to the left of the red obiect. Therefore, there are fewer tiny matte bicycles than purple rubber objects to the left of the red object
Ground Truth Answer:
Yes
Feedback:
The model correctly understands the textual information given in the question, identifying key mathematical elements, relationships, instructions from the text. The model accurately interprets the visual information in the question, including any diagrams, charts, or figures. model provides clear, step-by-step explanation logically connecting the problem's details to the steps leading toward solution. The model shows wrong intermediate steps or partial reasoning. The response lacks any logical connection the problem's details. The final answer is accurate. model shows no signs of excellency. The intermediate steps either not all correct difficult understand.
reasoning process,
Score:
4
Figure 13: Case 6 with feedback and score from MMIE-Score.
32

---

Published as a conference paper at ICLR 2025
Example of Situational Analysis
Question: Christmas is my favorite time year.
Answer:
A warm fireplace invites guests to sit and relax.
<image>
Enjoying the festive atmosphere, guests gather around the table for a meal.
<imsge>
It's special time year, and the warmth and are palpable.
Ground Truth Answer:
We all gathered together to celebrate.
<Image>
Uncle fez was memorialized with the planting of his tree.
<image>
Ben was sad.
<image>
Feedback:
The text relevant question, non-repetitive and clearly addresses the task. grammatically sentences being complete, concise confusion, the story maintains interest throughout. The images generated are clear, relevant, and enhance the understanding the response. They contain no distortions, and all elements in the image are meaningful and correctly represent the concepts or objects they are intended to depict.
accurate engaging,
the narralive
The text and images are well-coordinated, with the visual elements directly supporting illustrating the content described in the text. The combination of both helps enhancing the overall clarity and understanding of the response.
The response maintains internal consistency. with the setting. characters. narrative progressing logically without any contradictions.
information remains coherent, smooth transitions
different
parts of the response.
The method lacks creativity, containing repetitive or predictable elements. It follows conventional patterns without introducing any innovative approaches or surprising elements.
The model maintains stylistic consistency throughout the response, with consistent use of tones, formatting, and structural elements. There product.
sudden shifts in style atmosphere, creating cohesive and polished final
The response contains no or negalive emotions. maintains an appropriate lone, vialence or offensive content, oreating positive or neutral emotional atmosphere.
Score:
5
Figure 14: Case 7 with feedback and score from MMIE-Score.
33

---

Published as a conference paper at ICLR 2025
Example Situational Analysis
Question: We had snow storm move through last night.
Answer:
The birds standing outside the
<image>
Ground Truth Answer:
The hillside looked like it was blanketed a soft fluffy white blanket.
<imagez
There wasn't lot of water in the stream able to walk down near the
edge walk.
<image>
There were parts of the stream where the top of the water was completely frozen
the snow just sat top.
<image>
Feedback:
The text is relevant to the question, non-repetltive, and clearly addresses the task. The text grammatically accurate and engaging, with the sentences being complete, concise and clear. The reader follow the narrative without confusion, and the story maintains interest throughout.
The images generated clear, relevant, enhance understanding of the response. They contain image meaningful and correctly represent the concepts objects they
distortions, and elements intended depict.
The text and images are well-coordlinated with the visual elements directly supporting or illustrating the content described in the text. The combination of both helps in enhancing the overall clarity and understanding of the response.
The response contains inconsistencies in its narrative, such as contradictions in characters, setting, plot. There may be sudden, unexplained shifts in the context, making the story hard to follow understand.
The response lacks originality, relying predictable repelitiee ideas. There little evidence of creative thinking innovative of visuals and text, making response dull uninspired.
The lack stylistic consistency throughout the response, with consistent of tones, formatting, and structural elements. There product.
shifts style stmosphere, creating cohesive polished final
The response contains no harmful or negalive emotions. It maintains an appropriate emolional tone, free from violence or offensive content, creating positive or neutral emotional atmosphere.
Score:
2
Figure 15: Case 8 with feedback and score from MMIE-Score.
34


Reviews:
[{"review_id": "lXzoRAuR26", "reviewer": "Reviewer_eNcJ", "summary": "The paper presents MMIE: a benchmark for evaluating interleaved multimodal comprehension and generation abilities of Multimodal LLMs. The evaluation dataset is also publicly released. Further, they propose an automated evaluation metric, using a finetuned LLM. They evaluate various interleaved and “integrated” (text-generation followed to text-to-image generation) LLMs on their proposed benchmark.", "questions": "1. The effectiveness of the proposed evaluation metric is a function of the evaluation model. Please provide further details on the finetuning method and dataset utilised for the evaluation model.\n2. To better understand the role that the evaluation model plays in the pipeline, please provide qualitative examples of eval model responses corresponding to Fig 7,9,10,11,12 in the appendix.\n3. Please provide details on how the human annotations were obtained for Table 5 to better support the claim in L110.\n\nEdit: I have read the author response and revised my score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The MMIE benchmark presented in the paper offers a significant contribution in the interleaved comprehension and generation domain. The benchmark is of significant scale (~20k examples) and well categorised, specifically the project-based learning, situational analysis and multi-step reasoning categories. They also describe in good detail the process of the benchmark creation. \n\nAnother significant strength is the automated evaluation metric based on a finetuned LLM. With open-ended evals for interleaved generation, the challenge lies in capturing the various facets including image-text alignment, image quality and text quality. Their proposed method captures these, making it a strong contribution.\n\nThe experiment section of the paper is quite detailed. Especially the creation of the integrated LLMs where they combine state-of-the-art LLMs with text-to-image generation models. Sec 5 “Error Analysis” where they identify the typical types of errors offers a great analysis of failure cases.\n\nThe MMIE benchmark along with the finetuned model for evaluation is publicly released.", "weaknesses": "While the proposed automated evaluation metric using a finetuned LLM is novel and promising, details on the construction of the dataset used for finetuning are missing in the paper.  \n\nThe claim L110 “The proposed scoring model ... has proven to be comparable to human evaluation.” is not well justified in its current form. For instance, while Table 5 shows that their proposed method has better similarity with human scoring, further details on how the human annotations were obtained are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MYowBqgYFy", "reviewer": "Reviewer_pFfk", "summary": "The paper introduces MMIE (Massive Multimodal Interleaved Evaluation), a large-scale benchmark designed to evaluate interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs). MMIE comprises 20K multimodal queries across 12 fields, including math, coding, physics, literature, health, and arts. The benchmark supports both interleaved text and image inputs and outputs, including image generation, offering a mix of multiple-choice and open-ended question formats. As evaluation metric, the authors propose an finetuned (multimodal) LLM as a scoring model. This metric aims to reduce bias and improve evaluation accuracy over existing LLMs as-a-judges. The results reveal that current LLMs have significant room for improvement when evaluated on MMIE.", "questions": "* Dataset Size: The 20K sample benchmark is substantial and statistically robust, enabling a comprehensive evaluation. However, validating the full dataset with 20K samples may be costly. Have the authors considered designating a \"mini\" subset for consistent yet quicker evaluation? For example, using 800 samples selected for scoring model training.\n* Visual Component Importance: Is it feasible to analyze the impact of visual content within the benchmark? For instance, comparing text-only generation with generation that includes images. Some examples in Figure 1 do not rely on image content but rather use images solely for illustrative purposes.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* Originality: The paper introduces a novel and comprehensive benchmark for evaluating interleaved multimodal comprehension and generation, addressing an underexplored but increasingly important area.\n* Quality: The benchmark is large-scale and diverse, covering a wide range of domains and task formats, which enhances its utility and applicability. The methodology for dataset curation and quality control is rigorous.\n* Automated Evaluation Metric: The proposed scoring model provides a more reliable and unbiased evaluation compared to using a \"raw\" GPT model, strengthening the analysis beyond traditional metrics.\n* Human Scoring Comparison: Comparison of different scoring model against human annotations is great in highlighting the strength.", "weaknesses": "* Potential Biases in Scoring Model: The reliance on a LLM as a scoring model may introduce biases inherent in the base model. A more thorough analysis of these potential biases, and how they might affect evaluation outcomes across different domains or tasks, might be needed.\n* Content Warning Handling: The paper includes a content warning but does not elaborate on it, as Ethical Statement highlight strict guidelines and lack of bias in the dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JX1sLWBdRg", "reviewer": "Reviewer_t4WL", "summary": "This paper introduces MMIE, a large-scale benchmark designed to evaluate interleaved multimodal comprehension and generation capabilities of Large Vision-Language Models (LVLMs). MMIE comprises 20K multimodal queries across 12 fields, supporting interleaved text and multi-image inputs and outputs in both multiple-choice and open-ended formats. It provides a comprehensive framework to assess LVLMs on complex, real-world tasks that demand high multimodal reasoning and synthesis skills.", "questions": "Given that the scoring model is trained on data similar to MMIE (12 domains), I am curious about its generalizability to other benchmarks. Does the scoring model work on other benchmark data, such as MMLU, which includes 30 domains? This would provide insight into whether the model’s evaluation criteria are adaptable to a broader range of multimodal tasks.\n\n\nEdit: I have read the author response and revised my review.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. MMIE provides a robust evaluation framework for Large Vision-Language Models (LVLMs) by supporting cross-modal, interleaved input and output of both text and multiple images. This flexibility in handling multi-image inputs and outputs, along with the inclusion of multiple-choice and open-ended question formats, broadens the range and depth of tasks LVLMs are tested on. Furthermore, the large data scale and comprehensive scope of MMIE allow it to evaluate LVLMs in a way that captures the complexities of real-world multimodal interactions, offering a thorough assessment of these models’ interleaved multimodal comprehension and generation capabilities.\n\n2. MMIE introduces an automated scoring model fine-tuned with human-annotated data, developed using detailed evaluation criteria. This approach addresses the challenge of bias often seen in traditional metrics, enhancing the objectivity and precision of multimodal model evaluation.", "weaknesses": "1. Lack of fine-grained presentation of results across fields: Although the paper evaluates several models, it does not fully present results across different fields, making it challenging to analyze performance gaps in specific domains. This limitation may restrict insights into targeted improvements for specific models in particular fields.\n\n2. Potential Overfitting to Specific Benchmark Tasks: Since the scoring model in MMIE is fine-tuned on data specific to the benchmark, it may overfit to the types of tasks and response styles within MMIE.   This could limit its generalizability to other multimodal datasets or tasks, potentially reducing its effectiveness when evaluated in new contexts outside of MMIE’s scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)