MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models
Authors: Wulin Xie
YiFan Zhang
Chaoyou Fu
Yang Shi
Jianshu Zeng
Bingyan Nie
Hongkai Chen
Zhang Zhang
Liang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Unify Model; Multi-Modal Language Model; Benchmark
Primary area: datasets and benchmarks

Abstract: Unified Multimodal Large Language Models (U-MLLMs) have garnered considerable interest for their ability to seamlessly integrate generation and comprehension tasks. However, existing research lacks a unified evaluation standard, often relying on isolated benchmarks to assess these capabilities. Moreover, current work highlights the potential of “mixed-modality generation capabilities”
through case studies—such as generating auxiliary lines in images to solve geometric problems, or reasoning through a problem before generating a corresponding image. Despite this, there is no standardized benchmark to assess models on such unified tasks. To address this gap, we introduce MME-Unify, also termed as MME-U, the first open and reproducible benchmark designed to evaluate multimodal comprehension, generation, and mixed-modality generation capabilities. For comprehension and generation tasks, we curate a diverse set of tasks from 12 datasets, aligning their formats and metrics to develop a standardized evaluation framework. For unified tasks, we design five subtasks to rigorously assess how models’ understanding and generation capabilities can mutually enhance each other. Evaluation of 17 U-MLLMs, including Janus-Pro, Bagel, and Gemini2-Flash, reveals significant room for improvement, particularly in areas such as instruction following and image generation quality.

Published as a conference paper at ICLR 2026
MME-UNIFY: A COMPREHENSIVE BENCHMARK FOR UNIFIED MULTIMODAL UNDERSTANDING AND GEN- ERATION MODELS
Wulin Xie Yi-Fan Zhang 1,, Chaoyou Fu?.5, Yang Shi Bingyan Nie Hongkai Chen', Zhang Zhang'. Liang Wang'
CCASIA, NJU, 9PKU, Viioo, MM-M-E *Equal Contribution 'Project leader
https: /tmm//mmmmfyyy unify. www.ggitub..i..
ABSTRACT
Unified Multimodal Large Language Models (U-MLLMs) have garnered consid- erable interest for their ability seamlessly integrate generation and comprehen- sion tasks. However, existing research lacks unified evaluation standard, of- ten relying on isolated benchmarks to assess these capabilities. Moreover, cur- rent work highlights the potential of "mixed-modality generation capabilities" through case studies-sude as generating auxiliary lines images to solve geo- metric problems, or reasoning through problem before generating correspond- ing image. Despite this, there no standardized benchmark to assess models on such unified tasks. To address this gap, introduce MME-Unify, also termed as MME-U, the first open and reproducible benchmark designed to evaluate mul- timodal comprehension, generation, and mixed-modality generation capabilities. For comprehension and generation tasks, we curate a diverse set of tasks from 12 datasets, aligning their formats and metrics develop standardized evaluation framework. For unified tasks, we design five subtasks to rigorously assess how models' understanding and generation capabilities can mutually enhance each other. Evaluation of 17 U-MLLMs. including Janus-Pro, Bagel, and Gemini2- Flash, reveals significant room for improvement, particularly in areas such as in- struction following and image generation quality.
Rank
MME-U
Score
Tour
Veeul
Gemini2. Gemini2.5-fllast
50.04
Grovestee
Vidmn
Iniee Rit
Gemini2.0-flon
45.57
Mayy
Cott
RecA
42.60
-4 -Imaee
41,06
Tuat
Bagel
40.35
Kowanik
bener
1id
Monts
MIO-Instruee
37.17
SEED-LLaMA
28.45
Jow Reveee
Anole
18.59
(a) MME-U tasks.
(b) Leaderboard.
Figure 1: A comprehensive visualization of the diverse tasks in MME-U and the leaderboard. The figure (a) illustrates the wide-ranging nature of the tasks covered in our benchmark, which spans from traditional understanding tasks to complex mixed-modality generation challenges. The leaderboard (b) highlights the performance rankings of various U-MLLMs our benchmark.

---

Published as a conference paper at ICLR 2026
Geriie
CDSP OME
OSTT
Gamini
LOOD DOG
BARNARY
WELCOME NEWERA
EVELE
A min soaled télphane pole suburtan street with dightyy damp "LOS? postor featuring information missing dog. Inolading the soon locabin, contact number, and a $500 neward
actronatt stands an alten pi garing brarthaking sky
Tronss and vibrant auroras, A mission Flag pi planted besde thom roads: 'W COME TO THE NEW ERA bold.
Gerin
Grmmni
WS
PMILZ
photonelistic kitchen with minimalist Galartic Tiet" poster festuring an appetizing tutt, acoompenied by desaled recipe and soft morning light Tliiinnting the scene.
Ghibii-style dog
Figure 2: Complex instruction-based image generation comparison of results from open-source U-MLLMS (DeepSeekk-Jauss Flow, EMU3), closed-source U-MLLMs (GPT-4o, Gemini-2), and proprietary models (DALLE-3). The closed-source U-MLLMs have demonstrated abilities surpass- ing proprietary generation models, with a significantly larger gap compared open-source models.
INTRODUCTION
Unlike traditional MLLMS (c.g.. GPT-4V) and purely generative models (e.g., DALL-E 3), U- MLLMS Xie al. (2025b); Wang et al. (2024b); Chen al. (2025c); Ma al. (2024) excel in processing mixed-moda inputs and outputs, providing enhanced flexibility and the ability to ad- dress broader spectrum of complex tasks. Recently, closed-souree U-MLLMs, such as Gemini 2.0 Flash, have demonstrated exceptional generative capabilities, impressing in both instruction com- prehension and image creation, as shown in Figure These models exhibit an extraordinary grasp of image details, even surpassing proprietary generative models. However, this versatility also intro- duces considerable challenges in comprchensively evaluating their capabilities under unified evalu- ation criteria, primarily due to two key issues:
Lack of Standardized Benchmarks for Traditional Tasks. Existing works typically evaluate traditional generation and understanding tasks separately, using various bench- marks. However, the benchmarks chosen across studies inconsistent, and often in- compatible, leading to unfair comparisons. Moreover, the evaluation methods differ sig- dolcantly-multimodae understanding tasks may involve varied formats such as multiple- choice questions, GPT-4 scoring, or binary classification, while multimodal generation tasks may rely on metrics like CLIP score FID. This diversity evaluation makes difficult to derive an intuitive and unified performance score.
The most distinctive feature of U-MLLMs is their mixed-modality generation capabilities. which demonstrate the syn- ergistic interaction between multiple modalities Shi et al. (2025); Yang et al. (2025b). For instance, image editing requires accurately understanding textual instructions and identi- fying objects to be modified, while solving geometry problems involves comprehending the problem, drawing auxiliary lines, and performing logical reasoning. Despite these advanced capabilities, most methods only showcase simple cases, lacking standardized benchmark to rigorously assess these complex mixed-modality tasks.
Absence of Benchmarks for Mixed-Modality Generation]]
To address these challenges, we propose a comprchensive evaluation framework for U-MLLMs, which is shown in Figure [ For traditional generation and understanding tasks, sample data
also termed as unify tasks
2

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of MME-U and other Benchmark. SIPU: Single Image Perception & Un- derstanding; MITIU: Multiple & Interleaved Image-Text Understanding: VPU: Video Perception &
Understanding: CIVG: Conditional Image-to- Video Generation; FIR: Fine-gras tine-grained Image Recon-
struction; TIE: Text-Guided Image Editing: TIG: Texl-to-Image Generation; TVG: Text-to- Text-to-Video
Generation; VP: Video Prediction; UT: Unified Task.
Benchmark
Question
Year
SIPU
Understanding
MITIU
VPU
FIR
TIE
TIG
Generation
CTVG
TVG
Unify
UT
MSR-VTI al
2016
10,000
CVPR 2016
X
X
X
MMBench
3.217
arXiv 2023
GenEval Glmah cl
1,200
arXiv 2023
X
MagicBrish Phing
A:
2125
10,338
NewrPP 2023
VBench Huang al.
X
SEED Bencaz Liclal
2026
19,242
arXiv 2024
/
V
V
V
V
X
X
Einu-Edit Shaynn
X
TIP-1VV Wang & Yarg
11744
5000000
arXiv 2024
X
V
MMBenzh: Pang
2,000
NeurIPS 2024
MMEFuu
2,374
arXiv 2023
X
X
Vidko-MMME
2024
2,700
CVPR 2025
MME-Rea/Wonn///1
20250
29,429
ICLR 2025
X
Wise NNu er A123
1,000
arXiv 2025
MME-Umiy (ours)
4,100
2025
from 12 existing datasets, resulting in 10 tasks with 30 subtasks. On the understanding side, these tasks encompass single-image, multi-image, and video-based perception and reasoning tasks, cov- ering wide range of difficulties- from simple visual guestion-answering (VQA) to high-resolution VQA in real-world scenarios and long-video understanding. On the generation side, we include tasks such image/video generation and editing, as well as more complex conditional image gener- ation and maae-to-videe generation, aiming to cover the full spectrum of existing generative tasks. To simplify evaluation and provide a unified score, we manually reformat all understanding tasks into multiple-choice questions, reporting accuracy as the primary metric. For generation tasks, we standardize the evaluation scores and normalize them to provide a consistent metric. This approach reduces the difficulty of benchmark collection and mitigates the issue of inconsistent evaluation metrics across studies.
For the Unified Tasks, constructed five tasks: (1) Image Editing and Explaining, where the model first understands complex editing instructions and edits an image; (2) Common Sense Ques- tion Answering, where the model answers question and generates the corresponding image; (3) Auxiliary Lines, where the model draws auxiliary lines for geometry problems and then solves them; (4) SpotDiff, where the model identifies and draws the differences between two images; and (5) Vi- sual COT, where the model generates step-by-step strategies for navigating a maze and visualizes the next state. These tasks evaluate model's ability to perform sequential reasoning and generate corresponding multimodal outputs at each step. All tasks are carefully formatted as multiple-choice questions facilitate consistent, fair, and objective evaluation.
We evaluate 17 existing U-MLLMs, including Jamus-Pro, Bagel, VILA-U, and MiniGPT-5. To pro- vide context for their performance, we also compare them with specialized understanding models (e.g., 335 Sonnet, Qwen2.5-VD VL) and generative models (e.g.. DALL-E-2, DALL-E-3). This comprehensive evaluation not only underscores the strengths and weaknesses U-MLLMs but also establishes a standardized benchmark for future research in this rapidly evolving field. For exam- plc, we uncover several key experimental findings, as illustrated in Figure Currently, U-MLLMs exhibit significant variance in rankings across three dimensions, and no single model has emerged as the best performer across multiple capabilitics. Moreover, the performance gap between models substantial. Finally, the current open-sourced U-MLLMs still exhibit significant gap in perfor- mance compared to specialized models in both understanding and generation tasks. Additionally, while many works claim to handle mixed-modality generation, our unify task tests demonstrate that most of existing U-MLLMs struggle to consistently and effectively process these types of tasks.
2 MME-UNIFY
This section outlines the data collection, question annotation, and evaluation strategy for MME- Unify. Figures and visual representations of subtasks and samples across three domains, while Table compares MME-U with existing benchmarks. MME-U categorizes U-MLLM capa-
3

---

Published as a conference paper at ICLR 2026
Understanding
Generation
Comprehension
Image
Multiple Images Campechendon
Stegle Image Comprehenoin
Text to Video
Generation
Coaditional
Image
Test-lmage Generation
Video
an deethe te rervege
moregge eil Lk
thaliad boiked
Image Editing
The tot Ehereee ee
Pranpt: Adds
Video Perception and Understanding
Arowdl for perfimmen
Video Prediction
Fine Grained
Reconstraction
Proage ca snne Sarg
ass totermm detile tith ap
faality pusering buod
Freeuc
Evaluation Pipeline of Unify Tasks
Choices Matching
Text Choices
Input Imsage
Output
Bry C.r, D
Question
Image Choices
Text Choices
OpenAI CLIP
th B
Output Image
0.84, 0.92, 0.56, 0.45
CLIP-Score Matching
X
Unify Tasks
Quentia Greaao HILA
Spot
Image Explainiin and Editing
Commotsemu Question Amwering
di Sind Becencc
aljuct
yoer
and Jace
8:13
E:09
Ausiliory
Nomis faruu you rbjecte guiding th chancee po ph wock Loyatoooo Jored the
Visual CuT
hupi Taaage Chitaon
galee lna Cova delle line
Tox Chalecs Image Cholctr:
Actinn Cheienn
ALaBE lnd Sight Loans
A/13)8
Figure 3: Diagram of our MME-Unify. Our benchmark consists of 3 main domains, encompassing 15 subtasks comprehensively evaluate U-MLLMs understanding, generation, and unified capa- bilities. Specifically, each unify task includes at least one question, an input image, multiple text choices, and image choices. The image choices consist correct answer image and of man- ually crafted negative samples. During the evaluation process, we input the image, question, and text options, and the U-MLLMs are required select the correct text answer and generate an image. The text answer is evaluated by matching it with the correct answer, while the generated image is compared with the constructed image choices. If the CLIP score between the generated image and the correct answer image is the highest, it is considered correct; otherwise, it is deemed incorrect.
4

---

Published as a conference paper at ICLR 2026
bilities into three areas: (1) Multimodal Understanding, (2) Multimodal Generation, and (3) Unify Capability, highlighting the diverse aspects of model performance.
2.1 MULATI-MODAL. UNDERSTANDING
Data Collection. Multimodal understanding tasks are divided into three subcategories based on visual input type:
Single- Image Perception and Understanding (SIPU). This task is designed to evaluate image-text pair comprehension.
Multi-Image & Interleaved Text-Image Understanding (MITIU). Assesses the model's abil- ity handle and process multi-image and interleaved text-image inputs.
Video Perception and Understanding (VPU). Measures video comprehension capability.
To ensure comprehensive coverage of various image and video understanding scenarios, we collect 1, 1,900 samples from benchmarks such as MME and MMBench, encompassing over 24 tasks. This includes 1,600 perception tasks, such as OCR, diagram and table understanding, and spatial per- ception, along with 300 reasoning tasks, including attribute reasoning and action reasoning, with at least 50 QA pairs per sub-task. Additional details can be found in Appendix Figure 7 and Appendix Table 13 More visualization examples can be found in Appendix Figure 5
QA Pairs Reformulation. To standardize the evaluation of the understanding task, we convert all the collcted data into multiple-choice QA pairs, with one correct option and the remaining options carefully designed to be closely related to it. For models that can accept only single-image input, we use the first image from the multi-image input or the first frame from the video input. For models that cannot process video files (e.g., MP4 files), we uniformly sample six key frames from the video to serve as the visual input.
Evaluation Strategy. To fairly evaluate MLLM outputs, we apply rule-based filtering to match model responses with answer options, similar to MME-Realworld Zhang et al. (2025d); |uu et al. (2024)). Furthermore, to eliminate positional bias inherent in multiple-choice questions, the correct answer is randomly shuffled among the four available options. We then calculate the average ac- curacy across all sub-tasks and derive the overall understanding score, providing fair, robust, and unbiased evaluation of the model's performance.
2.2 MULTI-MODAL GENERATION
Multimodal generation involves various tasks for image and video modalities, which can be further subdivided based on application, as shown in Figure 3 1. Fine-grained Image Reconstruction (FIR). Given an original image. the model is required to restore detailed features and local textures. 2. Text-guided image Editing (TIE). Edit or modify an image based on textual instructions. 3. Text- guided Image Generation (TIG). Given a text description, the model needs to generate an image that matches it. Conditional Image-to-Videe Video Generation (CTVG). Generate dynamic video sequence based given image and text prompt. 5. Text-guided Video Generation (TVG). Generate video sequence based on a textual description. 6. Video Prediction (VP), Predict subsequent frames or the complete video sequence based on the information from the first frame.
Data Collection. Data is collected from benchmark datasets, such as COCO Lin et al. (2014a), Emu-Edit Sheynin et al. (024)) MSR-VTT Xu et al. (2016), ensuring at least 200 samples for each task. For video prediction, videos are sourced from the Pexel Video website") and the first frame is used for prediction. Detailed data sources and sample sizes in Appendix Table B3 More visualization examples can be found in Figure 6
QA Pairs Reformulation. Due to the diversity of generation tasks and their varied data sources, the collected samples contain redundant attributes and inconsistent number of images, videos, and other multimodal data. We aim to provide streamlined, unified evaluation framework. To achieve this, we contribute the following:
httpp::/ ftttps://www.rr hwwp://wwwwpxxw/w/l/m com/ / decs/
5

---

Published as a conference paper at ICLR 2026
Attribute Unification Pipeline. First, we summarize all attributes appearing in the data, which exceed 30 types, creating significant complexity. We then manually eliminate task- irrelevant attributes and merge similar attributes across different tasks. For example, text attributes are represented as Text Prompt, image attributes as Sre Image and Ref Image based on their input/output roles, and video attributes as Video. For any task where an attribute is not required, its corresponding value remains empty.
Task-Specific Prompt Engineering. To ensure that the model can effectively generate outputs that meet the task requirements, we establish specific system prompts for each subtask. Each sample's ' text prompt or stc image serves the input, while the reference image or video acts as the ground truth answer. Through standardizing attribute values and constructing tailored prompts, convert diverse samples from different tasks into unified format for evaluating multimodal generation tasks.
Evaluation Strategy. Evaluating multimodal generation tasks with a unified metric challenging due to the diversity of subdomains and their distinct metries (e.g., CLIP-I, CLIP-T, FVD, FID). To address this, we: (1) Perform domain-speciffi preliminary evaluations using standard metries; (2) Standardize all metries to consistent (0, 100) scale, converting non-positive indicators into positive ones; and (3) Compute the average of standardized scores to derive the final generation score. This approach ensures cross-task comparability while maintaining domain-specific evaluation rigor. Detailed metrics and standardization methods are provided in Appendix H
2.3 UNIFY CAPABILITY
MME-U contains five unified subtasks: (1) Common Sense Question Answering (CSQ), (2) Image Editing and Explaining (IEE), (3) SpotDiff (SD), (4) Auxiliary Lines (AL., and (5) Visual CoT (VCoT). Each subtask includes at least 50 manually constructed samples and is structured with task- specific instructions and question templates that require mixed-modality input-to-outpu generation.
Common Sense Question Answering. This task evaluates U-MLLMs' ability to associate com- monsense descriptions with visual features, such as linking "the tomb of an ancient Egyptian pharaoh" to a pyramid or "China's 's national treasure" to a panda. Our approach involves: 1. Question Construction. Using GPT-4o, we generate riddle-like questions based on commonsense concepts, with similar but incorrect words as negative options. For example, when the answer is "panda," select "brown bear" or "polar bear" as negative options to increase difficulty. 2. Image Collection. We manually gather images from the internet corresponding to the correct and their negative op- tions. 3. Task Execution. U-MLLMs are prompted to select the correct textual option and generate the corresponding image. Detailed procedures and the prompt are in Figure 9 a) and 20
Image Editing and Explanation. This task evaluates U-MLLMs' ability to understand complex editing instructions and generate accurate modifications. Our methodology includes: 1. Data Col- lection. We source data (source images, editing instructions, and reference images) from the Emu- Edit dataset. 2. Textual QA Construction. Using GPT-40, we generate accurate interpretations of editing targets and three incorrect interpretations for textual multiple-choice questions. 3. Visual QA Construction. The correct instruction corresponds to the target image in Emu-Edit. For incor- rect instructions, we input them into InstructPix2Pix [Brooks et al. (2023) to generate negatively edited images, forming image-based multiple-choice questions. 4. Task Execution. Given the cor- responding prompt, source image, and editing instructions, the model must first produce correct understanding the editing target and instructions, and then generate an edited image based on that understanding. Detailed procedures and the system prompt are in Figure (b) and 21
SpotDiff. When identifying differences between two similar images, humans typically need to recall the exact locations of these differences accurately count them. This task evaluates U-MLLMs' ability to identify and recall differences between similar images, simulating human visual reasoning. Our approach involves: 1.Data Collection: We sample image pairs with annotated differences from the SpftDiff w 2. Textual QA Construction. Using the annotated difference count, we create textual multiple-chote questions with three incorrect counts (+10 from the true value). Visual QA Construction. We place the annotated difference regions from the image pair onto white background as the correct answer, and randomly crop other areas to place them on the background
https: https://www.ar a www.ululeeseee com spot.difofffff ff

---

Published as a conference paper at ICLR 2026
as incorrect answers. 4.
Task Execution. U-MLLMs must identify the difference regions between the two images and draw them onto the white background, while also selecting the correct difference count. Detailed procedures in Figure cc.. and the system prompt is provided in Figure
This task evaluates U-MLLMS' ability to integrate understanding and generation by solving geometric problems requiring auxiliary lines. Our methodology includes: 1. Data Se- lection. We filter the Ge dataset for problems requiring auxiliary lines, extracting logical forms (e.g.. "Triangle(A, B. C)"), choices, and answers. 2. Textual QA Construction. Using GPT- 40, we generate natural language QA pairs (Question, Choices, Answer) for textual multiple-choiee questions. Visual QA Construction. We manually solve each sampled geometric problem by drawing the correct auxiliary lines on its diagram, and we construct three additional diagrams with erroneous auxiliary lines. 4. Task Execution. U-MILMs must first generate a geometric diagram with auxiliary lines, and then, based on that diagram, solve the problem by selecting the correct answer. Detailed procedures appear in Figure d), and the prompt is provided in Figure 23
Auxiliary Lines
Visual CoT. This task evaluates U-MLLMs' step-by-step decision-making in maze navigation, sim- ulating real-world problem-solving. Our approach involves: 1. Maze Generation. Using the Ope- nAI API, we create maze configurations of varying sizes (3x3, 4x4, 5x5) and layouts. 2. Action Specification. For each step, we manually define actions (Up. Right, Down, Left, Finish) and co- ordinates, updating the maze layout via the API. 3. QA Construction. Action Questions. Options are uniformly set as Up. Right, Down, and Left, with the correct answer manually determined. Coordinate/lmage Questions. The correct answers for each step's coordinates and state image manually defined, and negative samples are also manually specified. 4. Task Execution. U-MLLMs receive the initial maze state and task definition, then are prompted to generate actions, coordinates, and maze images iteratively. After the first step, we add the action, coordinate, and image from the previous decision into the system prompt as history information. The model iterates, outputting each step's 's decision until the target is reached EI Detailed procedures appear in Figure ((e), and the prompts are in Figure and 25
Evaluation Strategy. The unified tasks evaluation combines text-based and image-based multiple- choice questions across all subtasks. Our evaluation framework includes:
1. Textual QA Evaluation. For image explanation and editing, we compute CLIP-T similarity between the generated explanation and each option, selecting the one with the highest sim- ilarity as the correct answer. For other tasks, U-MLLMs directly select the correct option from the provided multiple-choice set.
2. Image-Based QA Evaluation. We compute CLIP-I similarity between the generated im- age and each candidate image option, selecting the image option with the highest CLIP-I similarity as the model's prediction.
3. Task-Specific Rules. For each task we calculate two accuracy metrics-ace and acc+ where acc is defined as the average of the text option accuracy and the image accuracy, and acc+ represents the accuracy for samples where both the textual and image-based answers are correct. Specifically, for the Visual CoT task, each step is treated as a multiple-choice question, and the accuracy of action, accordinate and image the average of these three accuracies is calculated as acc, while the accuracy of successfully completing the maze is used to calculate acc+. The detailed calculation process can be found in the Appendix H
calculated separately. and
We then calculate the average acc of all subtasks as the unified score, and the overall MME-U score is the average of the understanding, generation, and unified scores.
3 EXPERIMENT
We evaluate a total of 31 MLLMS and U-MLLMs, including DeepSeck- Janus--ro Chen et al.
2025ch. DeepSeek-Janus-F Ma et al.] 20044), SliME Zhang et al. 20224). VITA-1.5 Fu et al.
2025). Gemini2.00 Semini2.0-fflash DeepMind (002)). Gemini2.5-pry DeepMind (202)). Gemini2.0-flash
exp Mind 2024). Gemini2.5 www.5--flhh--mme -ashe DeepMind (2024).
Anthropic
'task requires average of 3.5 steps per sample, with minimum of two and maximum of seven steps (as shown in Figure B)
7

---

Published as a conference paper at ICLR 2026
2024), Emm3 Wang al. (2024b), GPT-4o OpenAl (2024c). GPT-4o- -o-maee O OpenAl 2024c
OmniGen Xiao et al. (002). DALL-E-2 OpenAl (2024a). D-LL-E-3 OpenAl 2024bb Qwen-
Image Wu et al. (2025). Owen-Image Edit Wu et al. (2025), Cog Video Yang al. (025c) Her-
mesFlow Yang et al. 2025a). Qwen2.5- Innsstrructtt Wang et al. (2024a) Intern- VI V-- //hu et al.
(2025). Show-o Xie al. (2025b), Show-o2 Xie et al. (025c), VILA-U W. (2024) GILLI Koh
ct al. (023). Anole Chern et al. (2024), MIO-Instruct Wang et al. (2024c), SEED-LLaMA Ge al. (024), MiniGPT-5 Zheng al. (2023), Bagel Deng et al. (2025), RecA Xie et al. (2025a).
Among the baselines, SliME, VITA-1.5, Qwen2.5-Vi Intern-VL VL-3, 6emini2.0-ftashh Gemini2.5.
Pro, Claude-3.5-sonned OmniGen, D-L DALL-E-3, Qwen-Image, twen-Image Edit are specialized understanding models or generative models. Notably, GILL, Anole, MIO-Instruct, SEED-LLaMA, MiniGPT-5, Gemini2.0-flan fexss-exp, Bagel, RecA, GPT-4o-Image, Gemini2.5-flashh image can generate interleaved images and texts. Some MILMS also can generate arbitrarily in- terlaced modalities, but they are not available as open-source code or model weights yet, such as PUMA Fang et al. (2024), VITRON Fei et al. (2024) and TextHarmony Zhao et al. 20244)
Mithool
IIM
Undentanding
Georration
Umily
MMR-U Score
Sptit
SIPC
Avg
CIVG
FIR
TIG
VP
Any CSQ
SD VCaT Ang
Avg
1200
1964
600
200
250
104
Understanding Models
SIME-7B
Vicana
58.50
43.55
30,02
46002
15.54
VITA-L5
70.67
Qwenz Seonz.-VLLLaeee VL-tnsric
Qwen-
75.08
53.50
57.04
61.91
20.64
Intn--V133
5-38
75.58
Cluode-3 s-senatt
75.83
62.53
GPT do
7401
54.50
29.34
62.62
20.87
Gemini B-Baah
9093
10
Gomini2 S-Pao
$7.00
74.00
Generntive Models
DALL-E-2
50.02
8:44
2.81
51
72.43
13:00
4.07
OmniGenn
48,52 43.02 $1.05
23.05
7.08
Qwes
CogVideoX
09.6 57.61
12.51
Unified Models
Pti-1.5
32:47
30,96
43.54
12.74
LLama 48
45.75
30.50
21.32
33.19
49.05
K.15
13.79
Flow
1.5-1.5
34.22
GILL
OPT6-78
28.18
50.67
35.71
46
24.25
21.29
6.67
1,50
15.10
Jans Flow
DeepSeek 1LM 1.56
63.17
52.00
35.16
43.44
32.85
5.45
16.31
MisiCPT.5'
10:25
38.96
35:45
34,13
5.00
2.08 15.67
Jamss
DeepSeek LLM
59.50
48.43
V-L
1. Lamo-77
51.04
10:25
56.54
99.05
45. 49.64
15.79
18.58
Ansle!
17.17
36.54
41.52
99.65
15.00
3.69
22.30
18.99
Show o2
Qwen2.5
08.33
55.11
50.15
06.34
SUND-LLLMA'
LLaMAA-Che 131
49.17
33,00
16.76 19.48
$7.00
42.26
41
23.54
22
00 51
49
12.50
22.00
3.61
33.99
78.45
MIO-Inirus'
MI0-7B
52.00
51.34
50.20
48.23
$1.38
38.50
11.50
0
16.56
37.17
Bogel
Qwan2.5-318 5-38
76.67
53.00
$1.10 60.76
44 48.46
99-91
24.00
53.34
7.23
91.73
35.50
11:20
15.00
-00.5
GPT Enugge-
65.90
49.50
45.05 53.35
50.07 46.56 63
28.72
43.00
96.64
42.31
22.50
11,02
41.10
41.06
RecA"
Qwen2.5:7B
76.00
57.00
63.01
46.30
72.58
79.21
26.50
12.22
37.45
42.00
Geminz 1- D-Hadh-eepp exp"
72.58
68.25
54.00
65.24
27A1 43.5 57
29.79
18-42
74.75
47.12
76.00
12:41
.0.74
Camiss2 S-Blds-inngs""
50.25
70.75
58.39
69.03
56 $2.90 85 34.00
48.75
86.02
50.62
25.50
15.23
47.02
30.04
Table 2: Comparison of MILMs on understanding, generation, unifying tasks, and overall MME-U Score. SIPU: Single Image Perception & Understanding: MITTU: Multiple & Interleaved Image-Text Understanding: VPU: Video Perception & Understanding; CTVG: Conditional Image-
to- VidVodeo Generation; FIR: Fine- Fine-grained Image Reconstruction; TIE: Text-Guided Image Editing;
TIG: Text-to-Image Generation: TVG: Text-to-Videe Generation; VP: Video Prediction; IEE: Im-
age Editing and Explaining, CSQ: Common Sense Question Answering: AL; Auxiliary Lines; SD:
SpotDiff; VCoT: Visual COT. denotes U-MILMs with the ability to generate interleaved images and texts, while indicates that the model is unable to finish the corresponding task and underlined content signifies the best performance within a single model across all methods on this task.
3.1 RESULTS
The evaluation results of various MLLMS in MME-U, as shown in Table indicate that Gemini2.5- flash-image achieves the highest MME-U score at 50.04. Although compared to MIO-Instruce it does not encompass all subtasks. it demonstrates very balanced performance across understanding, gen- eration, and unify tasks, unlike other models that may exhibit deficiencies in certain test dimensions.
8

---

Published as a conference paper at ICLR 2026
Arole Camii120 Ep
SEED MIKEPTI
Ancle
Garhh Exp
MINGTTIS
0 Ardi
Futh
SEED Niv
50
90
Acuuray
15 1
43
10
0
Pxp
Sep?
5ep5
Skp
Sap5
Sap
5kp5
Stepa
Steps
Steps
(a) Action Accuracy.
(b) Coordinate Accuracy.
(c) Image Accuracy.
Figure 4: Accuracy distribution across different dimensions on visual cot task. (a) action, (b)
location, and (c) image.
is evident that, compared to traditional MLLMs or generative models, U-MLLMs are capable of handling wider range of tasks, including more complex image-text interleaved reasoning. How- ever, overall, the development of U-MILMs is still in its early stages, and even the best-performing models only achieve scores of around 50 on MME-U. Next, will provide separate analysis of understanding, generation, and unify tasks.
Understanding. It is evident that Gemmini2.5-llahh-immee DeepMind (2024) demonstrates the best understanding capability among U-MLLMs, while also being a closed-source model. For open- source models, the two U-MLLMs with the best understanding capabilities are RecA Xie et al.) (2025)) and Bagel Deng et al. (202)). These models utilize two separate vision encoders to han- dle generation and understanding tasks independently, thus overcoming the limitations of tokenizers like VQGAN Yu et al. (2022), which are not well-suited for extracting semantic features. In con- trast, models like Emu3 Wang et al. (2024b) and Show-o Xie et al. (2025b), which use a single tokenizer for all image tasks, perform poorly on understanding tasks and still show a significant performance gap compared to currently available open-source MLLMS of similar size. However, experiments also show that models like Janus-Pro perform poorly generation tasks. They even fail to support multimodal generation, scoring zero on unified tasks. Therefore, how to strike balance between understanding and generation capabilities, indeed complement each other, remains an open question. We also see potential in bridging this gap in understanding capabilities by leveraging existing U-MLLMs alongside strong MLLM baselines. For instance, MIO-instruct [Wang et al. (2024c) achieves impressive understanding results through extensive training data, including video, audio, image-text pairs, and a complex three-stage training pipeline. This suggests that U-MLLMs may require a broader variety or larger volume of data for training.
whether the two capabilities can
Generation. We compare the performance gap between various U-MLLMs and current state-of-the- art generative models such as DALLE-3. is evident that, compared to understanding capabilities, the gap in generation tasks is not significant. For the simplest TIG task, 2 outperforms the best generative model DALLE-3 by six points, while U-MLLMs such as EMU3, HermesFlow, and GILL all achieve average score above 48. However, it is clear that most U- MLLMs still do not perform well on video generation tasks. Notably, although the original paper for Emu3 mentions its capability for video generation, the corresponding checkpoints have not been released. It's clear that the open-source community still has a long way to go before U-MLLMs that support video generation become widely available. Detailed results on the generation tasks can be found in Table g In Figure TO, we showcase the generation results from various models using the following text prompt: "A man is standing in a park with 'Run for Rights' banner in the background. He is wearing a white t-shirt with the number 28 on it, grey shorts, and grey socks with black shoes. The park is filled with people, some sitting on benches, and there bicycle leaning against tree." It is evident that most generated images, such as those from VILA-U, Show- O, and Janus-Pro, fail to capture key details from the caption, such as the number on the jersey or specific text. In contrast, the results from EMU3 more closely resemble the textual description, while
even
outputs are more aligned with realistic scenes (we hypothesize this is because Instructt was trained a large amount of real- world data, enhancing its ability to generate lifelike images). However, when it comes to image detail, current open-source U-MLLMs still lag significantly behind dedicated generative models.

---

Published as a conference paper at ICLR 2026
Unify Capability. Our systematic unify task testing shows that, while U-MLLMs have indeed expanded the potential for such tasks compared to traditional understandingggeeertttin performance remains insufficient. For each unify task in Table [ we require the models to generate the correct image and perform correct reasoning. Under these conditions, even for simple tasks such as answering common questions and generating images, the best open-sourced model (Anole) only achieves an accuracy of 59.65% and accuracy-plus of 38% (TableB3). In other tasks, no open-sourced model is able to surpass the 30% accuracy. It is worth noting that models perform even worse on tasks like Visual CoT, which require multi-step image generation and reasoning. No model is able to successfully complete tasks involving multiple steps. This finding underscores the importance of our MME-U, as relying solely on case studies to demonstrate model's mix-modality generation capabilities is clearly insufficient. We will further analyze these models' performance, weaknesses, and provide examples in the analysis section.
models, their
Due to space limitations, we have included additional in-depth analyses in Appendix F and G which contain expanded analyses of current U-MLLMs' weakness, detailed visualizations of the U-MLLMs' generation results, as well as specific examples from the unify tasks.
Medied
C50
AE
5D
YCUT
Tuily Soore
Natk
Tost Arc Smage ABC
Acce
Tex Asc Imsage AcT
JACC
Ame
TEST
Ace
Inap: Acc
Are
Acce
Teal Arc Emagr
Apc
Apee Actie
Certiaat Ace Imoas Acc
Arc
Acce
Act Auce
CILL
27.50
34.25
8.30
27.82
21.38
7.00
9.62
1-92
1.00
Shor
53.50
$6.14
11:34
24.00
32.75
5.30
33.50
54.13
5.60
23.08
34:37
3.64
5.00
2000
2.22
2.52
213
15407
MIo
21:00
21.00
31.00
7:30
18.03
13:31
3.66
35.00
11:50
06.56 1:30
13.00
MEDIALLINA
12.00
20.0
22:30
4.00
36.4
43.35
51.40
37.02
13.40
11.54
12.30
3.64
2500
23.50
2240 410
4.33
2.5
333
381
12.2 3.77
BezA
23.20
4750
35.57
12:30
3010
15.32
7921
9141
44.25
2388
3586
13.00
3600
23.03
20.50
520
11:33
1840
11.11
12322
57A2
15:45
35.90
60.58
13:30
17.00
0700
15.35
36.61
16.50
43.31
03.31
03:31
15:31
2600
28.00
2500
920
11.
8:17
1306
11:0
01.0
23.33
Vrage
13.01
01.00
Table 3: Comparison of U-MLLMs on various unify tasks and overall unify Score.
4 CONCLUSION AND LIMITATION
The MME-U benchmark framework presented here serves as a foundational step towards evaluating U-MLLMs on a diverse array of tasks encompassing multimodal understanding, generation, and their integration. This benchmark reveals the current landscape of U-MLLMs, highlighting their ca- pabilities and areas for improvement. While these models demonstrate proficiency in handling vari- ous multimodal tasks, they struggle with balancing understanding and generation, handling complex instructions, and performing well on unify tasks. Moreover, current U-MLLMs exhibit significant inconsistencies in aligning textual instructions with their visual outputs, highlighting the need for further research to improve multimodal reasoning and generation integration. However, this study questions,
simplifies the evaluation of unify tasks by framing image generation as multiple- which may allow model "hacking". For instance, SEED -enerrtee images may not meet style stan- dards but achieve high similarity scores, inflating accuracy metrics. Future work will incorporate MILM or CLIP scores for stricter evaluation.
REFERENCES
Anthropic. Introducing claude 3.5 sonnet, 2024. URL httpss:/www.atthrrpp..c. news/c cl aude 3- s-nnettttt]
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
Chubin Chen, Sujie Hu, Jiashu Zhu, Meiqi Wu, Jintao Chen, Yanxun Li, Nisha Huang, Chengyu Fang. Jiahong Wu, Xiangxiang Chu, et al. Taming preference mode collapse via directional de- coupling alignment diffusion reinforcement learning, arXiv preprint arXiv:2512.24146, 2025a.
Chubin Chen, Jiashu Zhu, Xiaokun Feng, et al. S -g-guidance: Stochastic self guidance for training- free enhancement of diffusion models. arXiv preprint arXiv:2508. / /2880, 2025b.
10

---

Published as a conference paper at ICLR 2026
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, arXiv,, 2025c.
Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large multimodal models for interleaved image-texi generation. arXiv, 2024.
Google DeepMind.
Gemini flash, 2024.
URL https:// gww.://..ppmiiooooo...o.oo..oe
tecchno wtt.ooieeegeeiingiig/gggee
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Wei- hao Yu, Xiaonan Nie, Ziang Song. Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining, 2025. httttt:::///aatt
wrg.absggga......... 2200. 14683
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. doi: 10. 10.1100/CVPR.2009.550 55066448
Xingjian Diao, Chunhui Zhang, Keyi Kong, Weiyi Wu, Chiyu Ma, Zhongyu Ouyang, Peijun Qing, Soroush Vosoughi, and Jiang Gui. Soundmind: Rl-incentivized logic reasoning for audio- language models. arXiv preprint arXiv:2506. 12935, 2025a.
Xingjian Diao, Chunhui Zhang, Weiyi Wu, Zhongyu Ouyang, Peijun Qing, Ming Cheng, Soroush Vosoughi, and Jiang Gui. Temporal working memory: Query-guidee segment refinement for enhanced multimodal understanding. arXiv preprint 17Xiv:2502.06020 2025b.
Xingjian Diao, Zheyuan Liu, Chunhui Zhang, Weiyi Wu, Keyi Kong, Lin Shi, Kaize Ding, Soroush Vosoughi, and Jiang Gui. Addressing overthinking in large vision-language models via gated perception-reassoinng optimization. arXiv preprini arXiv:2601.04442 2026.
Rongyao Fang, Chengqi Duan, Kun Wang, Hao Li, Hao Tian, Xingyu Zeng, Rui Zhao Jifeng Dai, Hongsheng Li, and Xihui Liu. Puma: Empowering unified mllm with multi-granulaa visual generation. arXiv, 2024.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. NeurIPS, 2025.
Hao Fei, Shengqiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision Ilm for understanding, generating, segmenting, editing. In NeurlPS, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv, 2023.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal Iims video analysis. arXiv, 2024a.
Chaoyou Fu, Yi-Fan Zhang, Shukang Yin. Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, et al. Mme-survey: A comprehensive survey on evaluation of multimodal llms. arXiv, 2024b.
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long. Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01955 2025.
Yuying Ge Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making Ilama see and draw with seed tokenizer. In ICLR, 2024.
Dhruba Ghosh, Hannaneh Hajishirzi, and I Ludwig Schmidt. Geneval: An foouce- framework for evaluating text-to-image alignment. NewrIPS, 2023.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si. Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Champaisit, et al. Vbench: Comprchensive benchmark suite for video generative models. In CVPR, 2024.
11

---

Published as a conference paper at ICLR 2026
Jing Yu Koh, Daniel Fried. and Ruslan Salakhutdinov. Generating images with multimodal language models. NeurIPS, 2023.
Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. Imagen- hub: Standardizing the evaluation of conditional image generation models. In ICLR, 2024. URL
https: pettps//ooee htttp:://openevieet net forum 1d=ood922kkkk0r
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan.
Seed-bench-2: Benchmarking multimodal large language models. arXiv, 2023.
Chengzu Li, Wenshan Wu, Huanyu Zhang. Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, and Furu Wei. Imagine while reasoning in space: Multimodal ifsualization-o-tthoulll arXiv, 2025.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014a.
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft coco: Common objects in context. ECCV, 2014b.
www.seemeiiieee or org/ :ttp://orpps::::::: 1411337611
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? ECCV, 2024.
Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. NeurIPS, 2024.
Yuwei Niu, Munan Ning, Mengren Zheng, Bin Lin, Peng Jin, Jiaqi Liao, Kunpeng Ning, Bin Zhu, and Li Yuan. Wise: A world knowledge-infommed semantic evaluation for text-to-image genera- tion. arXiv preprint arXiv:2503..07265 2025.
OpenAl. Dall-e 2, 2024a. [tt_itppps///eeeo...cooo///////s
OpenAl. Dall-e 2024b. http://p//opppaa...oo///////////
OpenAl. Hello gpt-4o, 2024c. http://ppp///ppaa...ccco///eeeee
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. In CVPR, 2024.
Yang Shi, Yuhao Dong. Yue Ding, Yuran Wang. Xuanyu Zhu, Sheng Zhou, Wenting Liu, Haochen Tian, Rundong Wang, Huanqian Wang. Zuyan Liu, Bohan Zeng, Ruizhe Chen, Qixun Wang, Zhuoran Zhang, Xinlong Chen, Chengzhuo Tong, Bozhou Li, Chaoyou Fu, Qiang Liu, Haotian Wang, Wenjing Yang, Yuanxing Zhang, Pengfei Wan, Yi-Fan Zhang, and Ziwei Liu. Realunify: Do unified models truly benefit from unification? a comprehensive benchmark. 2025. URL
https://arxxit wrg. 2509. 0225092.489 248977
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv, 2024.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv, 2024a.
Wenhao Wang and Yi Yang. Tip-i2v: A million-scale real text and image prompt dataset for image- to-video generation. arXiv, 2024.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv, 2024b.
Zekun Wang, King Zhu, Chunpu Xu, Wangchunshu Zhou, Jiaheng Liu, Yibo Zhang, Jiashuo Wang, Ning Shi, Siyu Li, Yizhi Li, et al. Mio: A foundation model multimodal tokens. arXiv, 2024c.
12

---

Published as a conference paper at ICLR 2026
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deging Li, Hang Zhang. Hao Meng. Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng. Lin Qu, Minggang Wu, Peng Wang, Shuting Yu, Tingkun Wen, Wensen Feng, Xiaoxiao Xu, Yi Wang, Yichang Zhang. Yonggiang Zhu, Yujia Wu, Yuxuan Cai, and Zenan Liu. Qwen-image technical report, 2025. URL BRLtttts:://att http://xxvvv../// org/bbs/
2508. 02324
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv, 2024.
Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang. Wenhao Zheng, Zhaorun Chen, Chenhang Cui, Mingyu Ding, Linjie Li, Lijuan Wang, and Huaxiu Yao. Mmie: Massive mul- timodal interleaved comprehension benchmark for large vision-language models, 2025. URL
attps:///rrxii org/ 2410..111..00 10139
Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. arXiv, 2024.
Ji Xie, Trevor Darrell, Luke Zettlemoyer, and XuDong Wang. Reconstruction alignment improves unified multimodal models, 2025a. URL /ttt::: http://aaxxvv...o.. wwtp/////////.....///// 07295
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin,
Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. ICLR, 2025b.
Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2 o2: Improved native unified multimodal models, 2025c. StRp:s/ts:://ee /wwpiv..vvvv//////////..../.aav /rg//b////////a 2506. 15564
Jun Xu, Tao Mci, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.
Yibo Yan, Shen Wang, Jiahao Huo, Hang Li, Boyan Li, Jiamin Su, Xiong Gao. Yi-Fan Zhang. Tianlong Xu, Zhendong Chu, et al. Errorradar: Benchmarking complex mathematical reasoning of multimodal large language models via error detection. arXiv, 2024.
Ling Yang, Xinchen Zhang, Ye Tian, Chenming Shang, Minghao Xu, Wentao Zhang, and Bin Cui. Hermesflow: Seamlessly closing the gap in multimodal understanding and generation. arXiv preprint arXiv:2502. 12148 2025a.
Yan Yang, Haochen Tian, Yang Shi, Wulin Xie, Yi-Fan Zhang, Yuhao Dong, Yibo Hu, Liang Wang, Ran He, Caifeng Shan, Chaoyou Fu, and Tieniu Tan. A survey of unified multimodal understanding and generation: Advances and challenges. November 2025b. doi: 10.36227/ techrxiv. 176289211111.... 16802577v1. URL http:://xxdooi..oooo htt.:/.oo..rr//..3222///////// 176289261.168027777
Zhuoyi Yang, Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an expert transformer. ICLR, 2025c. URL http:s///app..seentttccccoo////
orq//ooppssD: 2D:
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. ICLR, 2022.
Tao Yu, Chaoyou Fu, Junkang Wu, Jinda Lu, Kun Wang. Xingyu Lu, Yunhang Shen, Guibin Zhang, Dingjie Song, Yibo Yan, et al. Aligning multimodal Im with human preference: A survey. arXiv preprint arXiv:2503.144 14504, 2025.
Huanyu Zhang, Chengzu Li, Wenshan Wu, Shaoguang Mao, Yifan Zhang, Haochen Tian, Ivan Vulic, Zhang Zhang, Liang Wang. Tieniu Tan, et al. Scaling and beyond: Advancing spatial reasoning in milms requires new recipes. arXiv preprint arXiv:2504. 15037, 2025a.
13

---

Published as a conference paper at ICLR 2026
Huanyu Zhang, Wenshan Wu, Chengzu Li, Ning Shang, Yan Xia, Yangyu Huang, Yifan Zhang, Li Dong, Zhang Zhang, Liang Wang, et al. Latent sketchpad: Sketching visual thoughts to elicit multimodal reasoning in milms. arXiv preprint arXiv:2510.245:5 2025b.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. NeurIPS, 2023.
Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond Ilava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406. 08487, 2024.
Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al. Mm-rlhf: The next step forward in multimodal Ilm alignment. arXiv, 2025c.
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al. Mme-realworid: Could your multimodal Ilm challenge high-resolution real-world scenarios that are difficult for humans? ICLR, 2025d.
Zhen Zhao, Jingqun Tang, Binghong Wu, Chunhui Lin, Shubo Wei, Hao Liu, Xin Tan, Zhizhong Zhang, Can Huang, and Yuan Xic. Harmonizing visual text comprehension and generation. ArXiv, 2024. URLpttpss:
Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and-language genera- tion via generative vokens. arXiv, 2023.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. ICLR, 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang. Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. In- ternv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. URL https: // a arxiv org abs / 2504 10479
14

---

Published as a conference paper at ICLR 2026
Appendix
Ethics statement. This work involves no human subjects or sensitive data, uses only publicly available, properly licensed datasets without identifiable information, and presents no foreseeable safety, privacy, bias/iscrimination, misuse, conflict-of-itterest, or legal concerns.
Reproducibility statement. To ensure the full reproducibility of our benchmark, have provided comprehensive implementation details throughout the paper. The construction and statistics of our MME-Unify are detailed Sec and Figure
Figure 9
The Use of Large Language Models (LLMs). The authors employ the LLM exclusively for gram- mar and style refinement. The LLM was not used to generate or substantively revise scientific content, design experiments, analyze data, write code, or select references.
A RELATED WORKS
Unified Multimodal Large Language Models. Building on the success of MLLMs Diao et al. (2025b) Wang et al. (2024a); Fu et al. (2025); Yu et al. 2025); Zhang et al. 2004 2025a b): Chen et (2025b a Diao et al. (2026) ). recent studies U-MLLMss, which can understand and gen- erate multiple modalities in an end-to-end manner. Some approaches have adopted unified training objective, projecting both text and images into a discrete token space and employing a next-token prediction loss function for training Wang al. (2024c); |Wu al. (2024): Team (2024). This train- ing method and framework are notably straightforward However, using discrete image tokens (e.g,, extracted from VQVAE image features) may not be optimal for image understanding tasks. There- fore, works like Janus-Flow Chen et al. (2025c), Janus-Pro Chen et al. (2025c), among others, have employed different vision encoders such as VQVAE for image generation and SigL.IP for image comprchension, significantly enhancing the understanding capabilities of U-MLLMs. Additionally, other methods have found that diffusion training is more suitable for image generation. Thus, adopt- ing diffusion- baaddd training for image generation and next-token prediction for text generation aims to strengthen the image generation capabilities further Xie et al. (2025b); Zhou et al. (2025). Recent research has also explored fine-tuning U-MLLMs further enhance their performance on unified tasks] Li et al. (2025). However, despite the rapid advancements of U-MLLMs, there remains lack of comprehensive benchmarks for systematically and fairly evaluating their capabilities in under- standing, generation, and multimodal synthesis tasks.
Benchmarks for Understanding. With the rapid development of MLLMs, several concurrent works Fu et al.] (2024b) have proposed various benchmarks to evaluate the models' capabilities multimodal comprehension tasks, such as single-image perception and understanding Fuu et al.) (203);; (Zhang et al. (2025d) (e.g., MME series), interleaved image & text understanding, and video understanding Fang et al. (2025) (e.g., MMBench- Video, Video-MME). Additionally, some bench- marks focus multimodal safety Zhang et al. (2025c) or mathematical reasoning Yan et al. (0244).
These benchmarks differ in coverage and metrics.
Benchmarks for Generation. Various benchmarks have been proposed to assess multi-modal gen- cration capabilities WWan & Yang (2024): |; Sheynin et al. (2024); Deng et al. (2009); (uu et al. (2016);
Ku et 2024);; [i et al. (2023), ) including tasks like image reconstruction Deng et al. (2009). image editing Sheynin et al. (2024); Ku et al. (2024), and conditional image & video generation Wang & Yang (202)); Lin et al. (2014b). However, these benchmarks mainly focus individual tasks within single modalities, failing to capture the full scope of multi-modal comprehension and gen- eration. While some benchmarks, such as 2EED-Bench-2 Li et al. (2023) and MMIE Xia et al. (2025). provide hierarchical evaluation for both understanding and generation, they do not assess unified tasks, and the range of tasks is limited.
B STABILITY ANALYSIS OF MME-UNIFY
For the multimodal understanding task. MME-Unify collects 1,900 multiple-choice questions from five public benchmarks spanning 24 subtasks (SIPU, MITIU, VPU), with at least 50 QA pairs per subtask. For the multimodal generation task, we evaluate six aubtasks-FIR. TIE, TIG, CIVG, TVG,
15

---

Published as a conference paper at ICLR 2026
and VP- at least 200 samples per subtask, yielding 1,200 evaluated instances. These scored using standard domain-spern sspuiiii metrics and subsequently normalized to a [0, 100] scale. For the Unify tasks, four subtasks (IEE, CSQ. AL, SD) each contain at least 50 samples, and each sample induces two multiple-choice questions (text and image), resulting in over 100 questions per subtask. In particular, the VCoT subtask includes 90 mazes with average of 3.5 steps; each step requires three decisions (action, coordinate, image), yielding approximately 945 step-level questions (90 x 3.5 3). In total, MME-Unify comprises over 3,200 multiple-choice questions alongside more than 1,200 generation instances.
We further assess the stability of MME-U scores and rankings. Specifically, halve the benchmark samples and re-evaluate the four U-MLLMs. As shown in Table 4 While few subtask rankings (e.g., for Anole and SEED-LLaMA) exhibit minor fluctuations due the reduced sample size, the overall MME-U score ranking remains consistent with the full-dataset evaluation. These results in- dicate that the benchmark's sample sizes and scoring strategy can assess stable scores and consistent model rankings, supporting the reliability of MME-U for comparative assessment.
Split
Method
Understanding
Generation
Unify
MME-U
MiniGPTS
14.08
18.60
14.65
15.78
Anole
12.75
19.50
20.24
17.53
Split
SEED-LLaMA
24.50
23.92
21.57
23.33
MIO-Instruct
40.88
52.68
16.34
36.63
MiniGPT5
16.67
19.40
15.01
17.03
Anole
14.33
19.73
23.40
19.15
Split
SEED-LLaMA
27.67
23.73
23.15
24.85
MIO-Instruct
42.15
53.98
16.78
37.64
MiniGPT5
15.37
18.25
15.67
16.43
Anole
13.56
19.91
22.30
18.59
Overall
SEED-LLaMA
26.19
23.54
22.32
28.45
MIO-Instruct
41.50
53.45
16.56
37.17
Table Split-half evaluation on four U-MLLMs.
We aslo supplement the split-half experimental results for each sub-task in the unified tasks. As The ranking of the four representative models remains stable basically across different half-sample divisions for the unify tasks.
shown in Table 5
Method
IEE
CSQ
AL
SD
VCoT
Avg
Split A
MiniGPTS
19
24
21.5
23.8
35.6
29.6
7.7 23.1
15.4
4
8
6
3.2
1.3
3.8
2.8
14.7
Anole
15
17
16
70.3
51.4
60.8
19.2
11.5
15.4
20
12
16
4.4
0
6.4
3.6
20.2
SEED-LLAMA
17
22
19.5
54.1
43.2
48.7
11.5
11.5
11.5
26
24
25
3.2
2.5
3.8
3.2
21.6
MIO-Instruch
22
21
21.5
73.3
36.6
23.1
0
11.5
24
0
12
0
0
0
16.3
Split
MiniGPTS
23.5
41.6
38.6
23.1
1.3
1.3
1.9
Anole
21.5
81
47.6
64.3
11.5
15.4
13.5
14
14
2.5
1.3
8.9
23.4
SEED-LLAMA
21
28
24.5
58.7
50.8
54.8
15.4
11.5
13.5
20
18
19
5.1
3.2
3.8
23.2
InS-Iustr
26
27
26.5
79.2 0 39.6
11.5 0
5.8
22
0
11
0
0
0
0
16.8
Overall
MiniGPTS
21.5
24
22.8
29.7
38.6
34.1
5.7
23.1
14.4
4
6
5
2.2
1.3
2.9
2.1
15.7
Anole
17
20
18.5
70.3
49
59.7
15.4
13.5
14.4
17
13
15
3.5
0.6
7.6
3.9
22.3
SEED-LLAMA
19
25
22
56.4
46.5
51.5
13.5
11.5
12.5
23
21
22
4.1
2.9
388
3.6
22.3
MIO-Instruct
24
77.2
38.5
17.3
11.5
Table 5: Performance comparison on unified tasks across different data splits. T: Text Accu-
racy; I:
Image Accuracy; A: Average Accuracy; Act: Action Accuracy; Crd: Coordinate Accuracy. IEE: Image Editing and Explaining: CSQ: Common Sense Question Answering: AL: Auxiliary
Lines; SD: SpotDiff; VCoT: Visual Chain-of-Thought.
16

---

Published as a conference paper at ICLR 2026
C DEFINITION OF TRADITIONAL MLLM AND UNIFY MLLM
We use the term "Traditional MLLMs" denote understanding-centrr multimodal models that ac- cept visual inputs (images/videos) but do natively generate visual outputs inference time, such as Qwen2.5-VL, and their outputs are textual only. In contrast, "Generative MLLMs" specialize in visual generation, without general multimodal understanding capability, and "U-MLLMs" support both understanding and generation, including interleaved image- generation.
D COMPARISON OF EVALUATION STRATEGIES FOR THE UNIFY TASKS
We compare two evaluation strategies for assessing Unify Tasks: (i) generation- strategy, which we term CLIP-Choice and (ii) an option-selection strategy, which we term Select-Choice. In strategy, the model first generates an image; we then compute its CLIP similarity four candidate references and take the argmax as the model's implicit choice. contrast, Select- Choice feeds all image options the model and asks it to select the correct one directly. We condect experiments based on Geminí 2.0-flash-exp, which shown in Table 6 Compared with the CLIP-Chois strategy, the Select-Choice strategy achieves higher accuracy text and image multiple-choice questions evaluated on all unified tasks except the IEE task. This may be because choosing the correct option easier than generating the correct image. Moreover, after introducing the reference image into the model input, the model is more likely to rely on the given reference image answer rather than the image generated by the model itself. Therefore, compared with the ELIP-Similanee evaluation strategy, Select- deviates from our original intention of evaluating the unify capability of U-MLLM on the unify task.
the
Strabogy
IEE
CSQ
AL
SD
VCVI
Unily
Cchicc
94.6
91.1
72.8
CLIPSChiie
338
483
98.
10.0
832
66.3
748
63.
996
346
47.1
368
3600
2400
26.0
5.0
178
16.1
9.4
124
0
46.7
21.1
Table 6: Comparison of two evaluation strategies on the Unify task with detailed metrics per subtask.
E DISSCUSSION FOR THE EFFECTIVENESS OF OUR EVALUATION STRATEGY
In MME-U, goal is not independently evaluate the capability of understanding and generation, but rather to quantify the synergistic capability of U-MLLMs to understand an input, reason and then generate the correct visual output. To this end, we adopt a CLIP-I based multiple-choice image evaluation strategy that is aligned with the text multiple-choice format used in other parts of our benchmark. Below we explain and provide experimental evidence supporting its effectiveness.
E.1
WHY CLIP-I MULTIPLE CHOICE FOR UNIFIED EVALUATION
Unified representation and metric design. Our benchmark aims to measure unified understanding segrsstandinngggeerattooon synergy with single, discrete metric. Existing continuous metrics like LLM-Judge provide fine-grained and interpretable assessments of visual quality, but they com- putationally expensive and difficult to map directly to a discrete "success" signal that comparable to text multiple-choice accuracy. To achieve the balance, treat image evaluation similar text multiple-choice, each unified task instance associated with several candidate images, and the model-generated image is scored via CLIP-I similarity against each candidate. The candidate with the highest CLIP-I similarity is treated as the model's implicit choice, and we compute accuracy over all instances. This design has two advantages: it maps multimodal synergistic behavior to a discrete accuracy score that is directly comparable across tasks, and (ii) it enables unified leaderboard over both text and image questions.
Task design mitigates CLIP-I score hacking. A natural concern is that CLIP-based evaluation might be exploited by score hacking. For example, by generating generic but CLIP-friendly images that superficially match the prompt. To this end, unified tasks designed explicitly minimize this risk, since each sub-task requires prior understanding and reasoning before correct generation is possible. For instance, AL requires the model to correctly parse geometric configuration and
17

---

Published as a conference paper at ICLR 2026
Table 7: Kendall's 75 correlation between CLIP-I scores and human ratings on 200 randomly sam- pled unified-task instances. We observe consistently strong correlation (7, ~ 0.69-0.73) across sub-tasks, with an overall correlation of 0.71, indicating stable alignment between CLIP-I and hu- man perceptual judgments under our task design.
IEE
CSQ
AL
SD
VCoT
Overall
Kendall's Tb
0.712
0.732
0.717
0.707
0.692
0.709
infer the appropriate auxiliary line before drawing; VCoT requires the model to infer the next action and generate the corresponding future state. In such settings, the correct visual output depends on specific reasoning chain rather than generic visual patterns. Since the candidate options encode these reasoning outcomes (e.g., correct vs. incorrect auxiliary lines, correct vs. incorrect navigation states), an image generated solely from the surface form of the instruction, without solving the underlying reasoning problem, extremely unlikely to match the correct option under CLIP-I. In other words, the design of unified tasks anchors CLIP-I scores to the correctness of understanding and reasoning of given input, rather than to superficial prompt alignment.
E.2 CORRELATION WITH HUMAN JUDGMENTS
To empirically verify that CLIP-I under our task design aligns with human perception, we conduct a correlation study on subset of the benchmark. We randomly sample 200 instances from the unified tasks and evaluate the generation results of four advanced U-MLLMs (Bagel Deng et al.
(2025), GPT-4o-Image OpenAl (2024c), RecA Xie et al.] (2025a). Gemini2.5-Flaase Flasss-mm- Deep- Mind 2024 using both CLIP-I and human ratings.
Three expert annotators score each model's generated image on 1-5 scale along three dimensions: (i) Text- Sott-Folooingg (ii) Image Quality, and (iii) Reference Similarity. We average scores across annotators and dimensions to obtain a single human rating per sample. We then compute Kendall's To correlation coefficient between CLIP-I scores and human ratings for each sub-task and overall.
As shown in Table 7 the Kendall's тh values for all sub-tasks lie in the range of approximately 0.69-0.73, with an overall correlation of 0.71, which falls into the regime of strong correlation.
E.3 RANKING CONSISTENCY WITH LLM-JUDGE
We further validate the robustness of our evaluation strategy by comparing CLIP-I based rankings with LLM-Judge (GPT-4o) based ranking that closely mirrors human evaluation.
On the complete unified dataset, we evaluate five U-MLLMs (Show-o2, Bagel, RecA, GPT-4o- Image, Gemini2.5-Flash-Imagee under two scoring protocols: (i) CLIP-I multiple-choice accuracy, and (ii) an LLM-Judge that follows the same scoring dimensions as the human evaluation (text- following, image quality, and reference similarity), aggregated into normalized score. For each model, we compute per-tate per-task and overall scores under both evaluation strategies for direct compari- son. As shown in Table g the relative ordering of models is highly consistent between CLIP-I and LLMM-udge across most sub-tasks and in the overall average. In particular, G and GPT-44oImmaee consistently achieve the top two positions under both metrics, while RecA and Bagel form stable middle tier above Show-o2. Although Bagel and RecA exhibit similar per- formance on the VCoT task, leading to local inversion in their per-task ranking for that specific sub-task, the global ordering of the five models remains stable between the two scoring methods.
This consistency in model rankings indicates that CLIP-I based multiple-choice accuracy captures essentially the same relative performance ranking much more expensive LLM-Judge strategy. Together with the Kendall's т. analysis, this provides robust evidence that CLIP-I is a reliable and efficient evaluation strategy for human and L-LM-based judgments under our Benchmark.
18

---

Published as a conference paper at ICLR 2026
Model
IEE
CSQ
AL
SD
VCoT
Avg
CLIP-I
Show-o2
53.47
10.69
Bagel
43.89
64.36
21.15
24.00
10.69
32.82
RecA
47.95
68.32
23.08
27.00
11.11
35.49
GPT-4o-Image
60.50
76.24
42.31
21.00
13.06
42.62
Gemini 2.5-Flash-Image
65.50
73.12
51.92
24.00
13.47
45.60
LLM-Judge
Show-o2
56.33
11.27
Bagel
49.47
66.93
37.07
47.67
52.33
50.69
RecA
49.60
68.07
43.73
48.67
52.00
52.41
GPT-4o-Imag
68.87
78.80
58.60
45.47
52.60
60.87
Gemini 2.5- Image
80.33
72.33
65.40
53.07
58.67
65.96
Table 8: Comparison of CLIP-I and LLM-Judge based evaluation on unified image QA tasks.
Malhed
FBI
TIK
TIG
BVG
Censratin
Ang
Genenative
69.33
31:11
50.00
5.44
DALZ
5.57
Over Iaage
18.81
6533
45:22
(5.)
13.72
01.02
71.0
3812
51.0
23:05
YL:1
30.38
13:35
32.04
3.68
Torpbork lraw Pre
55.66
15:11
5.50
Raw
03.51
7:56
2910
3.15
VILAU
27.86
43.30
1322
2014
GILL
3060
30.07
34.35
17:27
35.71
8023
23:44
22.00
22.56
SHHINLLMA
59.05
51.00
07:12
17:30
42.36
0033
3130
4156
21.50
56.
15.90
45.77
66
11.87
2115
85.30
8661
3132
6000
25.22
Greiee 20 Back
71.6
716
19:00
015
1355
01.56
Table 9: Comparison of multimodal models on various generation tasks. CIVG: Conditional Image-to- Video Generation; FIR: Fine-grained Image Reconstruction; TIE: Text-Guide Image Editing; TIG: Text-to-lmage Generation; TVG: Text-to- Text-to-Video Generation; VP: Video Prediction. denotes MLLMs with the ability to generate interleaved images and texts. while "-' indicates that the model does not have the ability to achieve the corresponding task and underlined content signifies the best performance within single model across all methods on this task.
F ANALYSIS AND FINDINGS
Trade-off' Between Basic and Unified Capabilities. The experimental results reveal that current U-MLLMs face significant challenge in balancing their fundamental abilities-such under- standing and generating performance -iith the demands of unified tasks that require integrating multiple modalities. For instance, models like GILL, Anole, and MiniGPT-5 are designed to handle unified tasks but tend to exhibit relatively poor performance on basic tasks, which results in lower overall scores when compared to some non-unified MI.LMs. On the other hand, while MIO-Instruct demonstrates high performance in basic understanding and generation. its capability image and text generation effectively is notably deficient. This imbalance suggests that the cur- rent training paradigms may not be adequately aligning the learning objectives for basic and unified capabilities within a single framework.
interleave
Detailed Analysis of Model Performance Unify Tasks. In Table 3 provide detailed analysis of different models' performance on unify tasks, focusing on text reasoning accuracy and image generation accuracy. It is clear that InO-Instruct exhibits stronger understanding capabilities
19

---

Published as a conference paper at ICLR 2026
than generation abilities (as confirmed by the results in Table 2) As result, many of its tasks show high text reasoning performance, particularly in commonsense QA, where its text reasoning accuracy reaches 76.24%. However, it fails to generate a correct image, completely missing the potential for mutual reinforcement between generation and understanding. In contrast, other models show comparable performance in both text reasoning and image reasoning evaluation criteria, but their overall results are not impressive. Notably, for visual CoT tasks, despite our efforts to simplify the questions into multiple-choice format, none of the models have been able to correctly complete multi-step reasoning and generation tasks.
Poor Instruction Following Ability for Image Generation. There are two main issues with the current models in image generation: 1. Uncontrolled Style Generation. In Figure [ we present the intermediate state images generated by different models in the VCoT task. Only the Anole and Gemini2.0-faal --w.ss--aaahxppppp models are able to generate images with style similar to the initial image. In contrast, other models produce images with a clear style bias, which do not align well with our state diagrams. 2. Difficulty Understanding Complex Instructions. Many models, such as struggle with following complex instructions, such as generating auxiliary lines based on the original question. These models fail to generate images with auxiliary lines, often requiring multiple attempts to generate a relevant image, and the resulting images often bear little resemblance to the original reference. However, for simpler instructions, like generating an image of dog, these models able to execute the task correctly.
Inadequate Visual CoT Capability in Unified Models. In Figure we further illustrate the chal- lenges of the Visual CoT task. The accuracy of U-MLLMs declines as the number of steps in the VCoT task increases. Errors made in earlier steps compound over time, making increasingly dif- ficult for models to generate correct actions, coordinates, and images. This cascading error effect highlights a fundamental limitation in maintaining consistent reasoning across multi- step tasks. At the same time, this example further emphasizes the high requirements of our unify tasks for both generation and understanding capabilities. For instance, although Anole demonstrates relatively strong image accuracy in Figure 4 its weaker understanding abilities result in less effective action selection. This ultimately leads to worse final results compared to the other two basclines.
To further enhance the evaluation value of VCoT task, we additionally provide two complementary analyses: (i) difficulty stratification by maze size, and (ii) step-wise metrics across the full reasoning trajectory for further analysis of the unify capability of current U-MLLMs.
Difficulty stratification by maze size. We first stratify VCoT performance by maze size and report average accuracy for each model under different maze configurations (Table (10). This allows us to examine how U-MILMs behave as the underlying navigation problem becomes more complex.
3x3
4x4
5
Model
action
coord
image
action
coord
image
action
coord image
Bagel
21.14
23.58
21.14
19.70
23.48
19.51
12.00
18.00
14.67
RecA
28.46
28.46
20.33
19.70
28.03
20.45
14.67
19.33
16.67
GPT-4o-Ima Image
21.95
18.70
25.20
20.45
17.42
20.45
19.33
10.00
16.00
Gemini L--galas--mmage
43.09
29.27
31.71
38.64
20.45
23.48
31.33
12.00
18.00
Table 10: VCoT accuracy stratified by maze size. We report average action, coordinate, and image accuracy (%) on 3x3, 4x4, and 5x.5 mazes. For example, Gemini 2.5-Flash-Immae achieves 43.09% / 29.27% 31.71% action/coord/image accuracy on 3 mazes, which drops 31.33% / 12.00% / 18.00% on 5x5 mazes. Other models exhibit the same downward trend, providing an overall perspective on difficulty stratification in VCoT.
From Table I0 we observe that Gemini2.5- Faaasii----aas---maeess 's action/coord/imagg accuracy the 3x3 maze is 43.09% 29. 29.27%, and 31.71%, respectively, while these values drop to 31.33%, 12.00%, and 18.00% on the 5x5 maze. Other models exhibit similar monotonie decline maze size increases. This confirms that VCoT difficulty scales systematically with problem size and that all current U-MLLMs degrade under more challenging configurations.
Step-wise metrics and failure modes. Concurrently, we reconstruct VCoT into a set of step- wise metrics and compute average accuracy for four representative U-MLLMs (Bagel, RecA, GPT-
20

---

Published as a conference paper at ICLR 2026
Step-wise accuracy (%)
Model
Step
Step 2
Step 3
Step 4
Step
Step
Step
Action
Bagel
26.67
20.00
17.78
11.11
6.67
2.22
2.22
RecA
25.56
25,56
21.11
11.11
4,44
3.33
1.11
GPT-4o-Image
28.89
23.33
23.33
10.00
4.44
2.22
0.00
Gemini 2.5-Flash- Image
52.22
48.89
37.78
16.67
10.00
1.11
1.11
Coordinate
Bagel
67.78
20.00
17.78
11.11
1.11
0.00
0.00
RecA
75.56
25.56
12.22
2.22
1.11
1.11
0.00
GPT-44-Imaae
38.89
10.00
5.56
4.44
4.44
2.22
2.22
Gemini 2.5-Flashe 2.-FFlash-Image
23.33
20.00
17.78
17.78
10.00
1.11
0.00
Image
Bagel
43.33
25.56
7.78
7.78
1.11
1.11
0.00
RecA
46.67
22.22
5.56
5.56
2.22
1.11
0.00
GPT-4o-Imaae
51.11
28.89
10.00
12.22
0.00
0.00
0.00
Gemini E--Faah-mmaae
53.33
28.89
13.33
8.89
1.11
1.11
0.00
Table 11: Step-wise VCoT accuracy for different prediction dimensions. We report step-wise accuracy (%) for four representative U-MLLMs across steps 1-7. The three gray-shaded blocks correspond to action, coordinate, and image accuracy, respectively. All models start well above the random baseline at Step but by Steps 5-7 most entries drop to around 1-4%, especially in the coordinate and image dimensions.
4o-Image, Gemini2.5- 5-5inni...-Fa-h---- Flash-Image) across steps 1-7 three dimensions: action, coordinate, and image. The results are summarized in Table
From Table IT] we observe that Step the accuracies action, location, and image for all models are significantly higher than the random baseline, and remain reasonably good in Step 2. However, starting from Step 2, the coordinate and image dimensions exhibit a near "cliff-like" drop, reaching approximately 1-4% by Steps 5-7, while action accuracy decays more gradually. This pattern indi- cates that although the models can still correctly identify the current position and generate a roughly correct next frame in the first 1-2 steps, the first capability lost during the multi-step reasoning pro- cess is the consistent visual modeling of the maze state. Once this internal visual state becomes unreliable, it quickly propagates to degrade both coordinate localization and action prediction.
Therefore, even though the sample-level success rate over complete 7-step trajectories is close to 0%, the stratified analyses by maze size and step index allow to pinpoint two core bottlenecks of current U-MLLMs on VCoT: (i) multi-step visual state maintenance, and (ii) long-range con- sistency. These diagnosties provide essential guidance for targeted improvements planning and visual memory mechanisms future unified multimodal models.
F.1 COMPARE WITH RANDOM AND HUMAN BASELINES
To make the MME-U Score more interpretable, we compare our unified evaluation with two comple- mentary baselines, a random guessing baseline and a human baseline. Both baselines are evaluated under exactly the same multiple-choice protocol U-MLLMs across all Unify sub-tasks.
Random-Select baseline. For the random baseline (RANDOM SELECT), we uniformly sample one option for every text or image multiple-choce question, and compute all task metrics (Text Acc, Image Acc, Acc, and A for VCoT we additionally report Action/Coord/IInge Acc). As shown in Table 12 the Random-Select metrics the two high-precision / multi-step reasoning tasks, SiftDiff (SD) and VCoT, already exceed the performance of most current U-MLLMs. For exam- ple, VCoT, the random baseline attains 14.24% Acc and 21.73% Image Acc, which are higher than those of Bagel, RecA, and GPT--4-Image, and only slightly below Gemini2.5- 5-.a5--F-hhh-- Flass---maaee. Image. This indirectly suggests that existing U-MLLMs remain extremely weak in scenarios requiring fine-
21

---

Published as a conference paper at ICLR 2026
Method
CSQ
VOIT
Arg
T
A7
A7
A
AT
A
AT
Act.
Cid.
Ing.
A
A+
Shnnel
19
$6.44
Random Select
29.00
7.09
23:73
25.43
Buzel
22:78
43.89
33.34
11.67
31.73
10:33
12.08
10.09
11.20
0.00
33.30
RocA
25.99 42.85
10.1
79.21
11.3%
14.08
11:11
12.22
0.00
57.45
GP Image
25.50
06.50
43.00
17.00
97.03
76.24
86.64
14.26
42.51
42.31
42.31
15.38
29.00
21.00
25.00
9.00
11333
8.47
13.06
11.02
0.00
41.30
Genini-2 Imans
22.00
65.30
48.75
19.00
98.
73.12
85.02
22
67.51
51.92
99
52
27.00
24.00
23.30
9.00
20.97
11:25
13.47
15.23
0.00
47.02
Himar
99.00
10000
98.00
96.00 97.3
57.52 59.5
50.52
94.25
94.23
14,25
94.25
19000
1900
19,00
1900
10000
10000
100,00
100099
100000 81.75
Table 12: Calibration of Unify tasks with random and human baselines. We report unified metrics for each Unify sub-task (IEE, CSQ, AI, SD, VCoT) and overall Avg. Columns: =Txx Ace, = Image Acc, Acc, A+ joint accuracy (text image both correct), Act. action accuracy, Crd. coordinate accuracy, Img. image accuracy. RANDOM SELECT randomly chooses one option per question (non-reasoning lower bound), while HUMAN is the average of two experts (upper bound). Current U-MLLMs lie between these two baselines, often much closer to random SD and VCoT, highlighting substantial headroom for unified multimodal reasoning.
grained difference localization or long-range state tracking, confirming that SD and VCoT are highly challenging items with strong difficulty discrimination.
In contrast, for IEE, CSQ, and AL, although some U-MLLMs' single-modality Text/Image accuracy only modestly above the random baseline, their Acc scores scores-which require both the text and image answers to be correct simultaneously-ant substantially higher than the theoretical random upper bound of Rv 6.25% (i.e., randomly guessing one out of four text options and one out of four image options). For instance, RecA, GPT-40-Image, and Gemini2.5- Saiiii..--Flashmmmaa Image reach 63.37%, 59.41%, 74.26%, and 73.12% respectively. This indicates that U-MLLMs have learned non-trivial degree of cross-modal consis- tency and understanding generation synergy, and that our Acc/ / Acc* metrics effectively separate random strategies from genuine unified capability.
CSQ. Random-Select achieves only 5,94% Acct, while Bagel,
Human Baseline. For the human baseline (HUMAN), we asked two human experts with multi- modal experience to complete the Unify tasks, selecting the best mest-matching option for both text and image questions. We report the average score over the two annotators Table [2] Human perfor- mance on IEE, CSQ, AL, and VCoT is substantially higher than that of all current U-MLLMs, e.g.. 98.00% / 97.52% / 94. 23% Acc on IEE/ / CSQ AL., and 100% Ace/ Ace on VCoT, demonstrating that the human upper bound is well above existing models.
Overall, current U-MLLMs often sit closer to the random baseline on the most challenging tasks (SD, VCoT), yet substantially surpass it Acct for IEE/CSQ/AL, jointly shows that (i) there is still substantial room for improving unified multimodal capability, and (ii) our Unify task design and Acc / A metrics offer discriminative and effective measurement of understanding and generation synergy capability.
G EXTENDED EXPERIMENTAL RESULTS
G.1 MOST U-MLLMS EXIIBIT INFERIOR GENERATION CAPABILITIES
While the methods in Table show relatively small differences compared to the current state-of-the- art (SOTA) generation techniques, we found that using CLIP scores for evaluation introduces certain risks of manipulation.
In Figure 12 we present the results on the fine-grained image reconstruction task. For each model, we used unified prompt: "Reconstruct high-fidelity images from degraded inputs, preserving fine- grained details, textures, and structural integrity with perceptual realism." It is evident that GILL, SEED-LLaMA, and MIO-Instruct effectively capture the structural details of the input images and produce noticeably clearer outputs. In particular, SEED-LLaMA and MIO-Instruct demonstrate strong performance in restoring color fidelity, while Gemini2.00 O-flash-exp tends to preserve the in- tegrity of the input images. In contrast, MiniGPT-5 and Anole fail to effectively extract the necessary visual information: while MiniGPT-5 does generate an image, its output deviates significantly from the source, and Anole is unable to generate coherent image all.
22

---

Published as a conference paper at ICLR 2026
Figure 13 displays the results for the text-guided image editing task, where the editing instruction was "Change this image into a watercolor art." " Similar to the reconstruction task, SEED-LLaMA and MIO-Instruct generate images that more closely resemble the source image; however, they fall short in accurately executing the specified editing instruction. Meanwhile, GILL,, MiniGPT-5, and Anole show limited capability in capturing and manipulating the requisite visual details for the transformation. Notably, Gemini2.0 --llss--xxxpppp not only preserves the content of the source image effectively but also accurately implements modifications according the editing instructions.
Figure 14 illustrates the performance gap between pure video generation models and U-MLLMs on the conditional image-to-video generation task. Using the text prompt "The man is so tired. -camera zoom in," we observe that although InO-Instr produces video outputs with richer visual details compared to Cog VideoX, it struggles to effectively generate a coherent video sequence that adheres to the given instruction based on the initial image.
In Figure [1] the generation results of Cog VideoX and MIO-Instruct in the Text-to- --dee Genera- tion task are compared. The results clearly indicate that, in terms of both instruction adherence and video consistency, MIO-Instruct significantly underperforms compared to dedicated video genera- tion models.
Overall, while some U-MILLMs exhibit promising capabilities in capturing visual details and pro- ducing high-fidelity reconstructions, challenges remain in faithfully executing complex editing in- structions and generating consistent video sequences. These findings highlight critical areas for further improvement in enhancing the generation capabilities of U-MLLM systems.
G.2 CHALLENGES IN SIMULTANEOUSLY GENERATING HIGH--UALITY TEXT AND IMAGES IN U-MLLMS
Figures [6 17 18 and 19 present the results of U-MLLMs on the Unify tasks. Notably, MIO- Instruct fails to perform any text-image generation across all Unify tasks, GILL is unable to generate multimodal outputs in the SpotDiff task, and SEED-LLaMA does not support text-image generation in the Auxiliary Lines task. Overall, these results indicate that most U-MLLMs struggle to generate images that faithfully adhere to provided instructions or reference images, and their comprehension of the instructions is often flawed.
In the Image Editing and Explanation task, for instance, MiniGPT-5 produced images that bore no relation to the source images. Additionally, the textual outputs from GILL, MiniGPT-5, and SEED- LLaMA were insufficient for accurately describing the editing objects or the instructions. Similarly, in both the Commonsense Question Answering and SpotDiff tasks, although MiniGPT-5 and SEED- LLaMA correctly answered the textual multiple-choice questions, the images they generated were clearly unrelated the corresponding options. This further emphasizes the difficulty U-MLLMs face in maintaining consistency between textual and visual outputs.
For the Auxiliary Lines task, while Anole managed to generate images that retained some of the visual details of the source images, it failed to correctly draw the required auxiliary lines as per the instructions. GILL and MiniGPT-5, on the other hand, generated content that was completely disconnected from the original images.
These findings suggest several critical limitations in current U-MLLM systems. First, there is no- table gap in their ability to integrate and utilize multimodal cues effectively, as evidenced by the mis- alignment between textual instructions and visual outputs. Second, while some models can capture certain visual details, they often lack the robust reasoning required to follow complex instructions, especially in tasks demanding precise visual modifications. Finally, the decoupling between text and image generation in these systems underscores the need for further research aimed at improving cross-modee modsl coherence and instruction fidelity.
Overall, the experimental results highlight that, despite progress in individual modalities, existing U-MLLMs have considerable challenges in simultancously generating high-quality, coherent text and images that align with complex, multimodal instructions.
23

---

Published as conference paper at ICLR 2026
with
Autonomsous
peieii budding alowee?
200
Tou ea sorii len E. of deceene muriing
Maat can ar butsur plcc
sarting
Thi Casteet Chstath
bake de calilo giaad
whut seatt
A. hen
Muutiple-Imaaes Comprehension
Q What this comnic strip about?
valcome dasr
On anpee Guarteg dn
woud nor Cahoad
The cat teacher taugh other cats how to be cats, and the cat students answerve that cats with no vilf-estem would sloup in the cat bod.
The cat teacher teaches the cat students how sleep correctly.
Several cats are discussing the problem sf esteem.
The cat students contradict the cat teacher
Interleaved Image Comprehension
bighest
image.
clump
image.
picture lilllyy afternoom.
shape
image are pointed.
The shape of the window on the stone building is squara.
The shape of the plate near the top left corner the image unique brcause it's empty and rund.
Temporal Reasoning
QhWhich the following has the appearing
gym, interviens.. gym, intersiews. study librury, work goas.
Fat lunch, study
Study library:
interviews,
Action Recognition
Whhhich the following the video the
press.
Leg peess.
D.Bunning.
Figure 5: Data samples from understanding task, which includes single-image perception and rea- soning, multi-image and image-text interlaced perception and reasoning, video perception and rea- soning, etc.
H EVALUATION METRICS
H.1
UNDERSTANDING SCORE
Let the three subtasks in the Understanding Task be formally defined follows:
T = (SIPU, MITIU, VPU).
For each subtask T. let Q. represent the set of multiple-choice questions, where each question q E Q has exactly one correct answer. To evaluate correctness, we define the indicator function for each question as follows:
1, the selected answer for is correct, (.)))
0. otherwise.
The accuracy for subtask è is given by:
1
acc,
Te(g).
[Qr
gEQ.
24

---

Published as a conference paper at ICLR 2026
Generation
Fine Grained Reconstraction
is progr ess with oftter besse plate
Text Prorge A busebell game
Prampe Recusstract the preserving tatures, edees and si seustural liclts wiile mataraining tigh steul
Image
Edit lsstrantion
the The stands are filted with spectotors and thescene large stadium.
Add musse on
Text-to Video Generation
Prompt: 2 players missed to put goal in football
Video Prediction
Find Frome
Frame
Test Prompt: Generate seamless video from the given Initial Trame, preserving style, continuity.
Image-to-
Sec Iragee
Finst From
Second Frane
Text Prompt: Men walk fiery
The Demons eyes glow red smiles, Ash and sparks foreground.
Figure 6: Data samples from generation task. It includes subtasks such as Text-to- to-e Generation, Text-to-Image Editing, Fine-Grained Image Reconstruction, Text-to Eee-tt--iiiie Video Generation, conditional Image-to- Video Generation, and Video Prediction.
Remote Relations Physical Relations
OCR & Text Understanding
Temporal Understanding
Spatial Understanding
Counting Numerical
2.38%
11.62%
Object Recognition
6.62%%
2.75%
Monitoring
9.75%
Object Reasoning
3.5%
Information Synopsis
3.75%
MME-Unify
7%
Scene Understanding
3.75%
Landmarks
4.25%
6.25%
Attribute Reasoning
Action Recognition
4.5%
Iecognition Reasoning
6.12%
Action Reasaning
Artwork
5.38%
5.5%
Driving
Image Properties
Celebrity Recognition
Specializad Reasoning
Diagrams Tables
Figure 7: An overview of real-life scenarios included in the Understanding Task. The scores in the bars represent the proportion of the number of samples of the corresponding scenario to the total number of samples of the task.
Since equal weights are assigned each subtask, the Understanding Score (US) computed the arithmetic mean of the accuracies across all subtasks:
US
Σ
score,;
T (SIPU, MITIU, VPU).
IET
H.2 GENERATION SCORE
The generative task comprises six subtasks:
T = (CIVG, TVG, FIR, TIE, TIG).
All metric normalized the range [0, 100).
25

---

Published as a conference paper at ICLR 2026
3x3 Maze
4x4 Maze
5x5 Maze
15
Samples
0
2
3
Steps
Figure 8: Distribution of steps required for samples of different mazes in the Visual CoT task.
Tick
Imagen
TIP-
Imuge
Geomelry
Open
Samples
Understanding Tark
SIPU
400
400
400
0
01
0
B
1,200
MITIU
400
0
0
0
400
YPU
364
Generative
CIVG
200
FIR
0
()
200
0
0
200
TIG
0
0
()
200
0
0
200
TIE
01
400
200
01
0
600
IVG
VP
Unify
IE
0
2010
U
200
CSQ
0
0
0)
0
0
100
101
AL.
0
0
0
52
0
52
SD
104
Datasel
9.75%
9.75%
9.75%
9.75%
8.87%
9.75%
9.75%
4.87%
4.87%
4.87% 4.87%
4.73%
1,27%
2.54%
4.63%
100%
Table 13: Task-Dataset Sampling Statistics. This table presents the distribution of samples across different multimodal AI tasks and their source datasets. Tasks categorized into three main groups: Understanding Tasks (SIPU: Single Image Perception and Understanding, MITIU: Multi- Image Interleaved Text-Image Understanding, VPU: Video Perception and Understanding). Gen- tnade-to-Video erative Tasks (CIVG: Conditional Image-to- Video Generation, FIR: Fine- grained Image Reconstruc-
tion. TIG: Text-to-Image Generation, TIE: -eixe- Image Editing. TVG: Text-to to Video Gener- ation, VP: Video Prediction), and Unify Tasks (IEE: Image Editing and Explanation, CSQ: Common Sense Question Answring, AI.: Auxiliary Lines., SD: SpotDiff, VCoT: Visual CoT). The rightmost column shows the total number of samples used for each task across all datasets. A value of O indi- cates that no samples were drawn from that dataset for the corresponding task.
Normalization of FVD and FID Scores. Let denote the raw FVD or FID value for a sample, where E [1, 1000] and lower values indicate better performance. The normalized score S is computed as:
S = 100 (1
1000 -1 1-111 100 (1-
8-1
8-1
999
This ensures:
S 100 when (best performance),
S when 1000 (worst performance).
all raw scores across models are identical, each normalized score is set to 100 to maintain consis- tency in evaluation and prevent division by zero in the normalization process.
26

---

Published as a conference paper at ICLR 2026
Score Calculation for CIVG and TVG. The subtask score for t (CIVG, TVG) is given by:
FVD (t)
+ FID nerm
+ CLIPSIM
ScOrC;
norm
3
Score Calculation for VP. The VP subtask score is determined using the following formula:
Scorevp FVD (P)
FID (VP)
2
Score Calculation for FIR, TIE, and TIG. For FIR (Fine-Grained Image Reconstruction), the metric is LPIPS. To ensure higher values indicate better performance, the score is defined as:
ScOreFIR
LPIPS.
For both TIE (Text Image Editing) and TIG (Text-to-lmage Generation), two metrics used: CLIP-I and CLIP-T. The score for each subtask is computed as the average of these two metries:
CLIP-ITE + CLIP-TTIE
ScorетIE
CLIP-InG + CLIP-TTIG
ScoreTIG
Overall Generation Score. The overall Generation Score (GS) is the arithmetic mean of all six subtask scores:
GS
score,
= VP, FIR, TIE, TIG].
KE
H.3 UNIRY SCORE
Let the Unify Task consist of the subtasks
T (IEE, CSQ,AL, SD, VCoT).
For each subtask E T, denote by S, the set of samples.
H.3.1 SUBTASKS IEE, CSQ, AL, SD
For a given subtask / E (IEE, CSQ, AL,, SD} and for each sample E S,, there are two questions:
1. A text-based multiple-choice question.
2. An image-based based multiple-choice question.
Define the indicator functions for the text and image responses follow:
[ 1, the text answer for correct,
(s)
0, otherwise,
1, if the image answer for is correct,
[
0, otherwise.
Then, the text accuracy and image accuracy for subtask are, respectively,
text acc
(s),,,,
ing acc,
(s))
(s).
S.
S.
aES.
BES.
The overall accuracy for subtask / is then defined as the average of the two:
ace; acc + acc!.ng
2
Additionally, we define acc;+ to represent the accuracy for samples where both the textual and image- based answers are correct:
1
acc;
(s)
1iml (
S.S
SES.
27

---

Published as a conference paper at ICLR 2026
H.3.2 SUBTASK VCoT
For the VCoT subtask, each sample SvCoT represents a maze navigation task composed of K, sequential steps. For each step k E {1, 2,.,.... Ka). there are multiple-choice questions evaluating the model's prediction of:
1. An action.
2. A coordinate.
3. An image.
Calculation of acCvcot: Let Stteps = E.eSSver Kg be the total number of steps across all samples in the VCoT subtask. Define the indicator functions for the correctness of action, coordinate, and image predictions for step of sample as follow:
1, the action prediction for step k:
naction VCoT (s, k))
of sample is correct,
0, otherwise.
1, if the coordinate prediction for step k
VCoT (s,
of sample s correct,
0, otherwise.
1, if the image prediction for step k
VCST (s, k)
of sample is correct,
0,
otherwise.
Calculate the average accuracy for each prediction type across all steps:
action acoVCoT
paction (s, k),
Nsteps
Sveet
coord VCoT
Teaord (CoT(ss k)),
Nsteps
of Sycer
img acc VCC
Σ
Tlin cCoo (s, k))
Nsteps
Svour
The overall accycoT metric is the arithmetic mean of these three component accuracies:
accaction
acceoord
img accyCoT
acCyCoT
Calculation of vCoT: Define indicator function for the full correctness of single step k in sample s:
Cstepall (s, k) Inction (s, k) x
Teoord VC.T (s, k) x veur (s, k)).
ring
This function if all three predictions for step are correct, and otherwise.
Now, define the indicator function for the perfect completion of sample
if Mapp all correcr (s, for all E (1, o, otherwise.
sample perfect (8)
The acc-tycor metric is the proportion of perfectly completed samples:
acc+vcoT
sample.perfect, LVCT
(s).
|SvCOT]
SE
28

---

Published as a conference paper at ICLR 2026
H.4 UNIFY SCORES
The Unify Score (Unify-S) is defined as the arithmetic mean of the acc; metrics across all subtasks:
Unify-S
Σ
acct,
[
tE
H.S
MME-U SCORE
The MME-U Score is computed the arithmetic mean of the Understanding Score (US), Generation Score (GS), and Unify Score (Unify-S):
MME-U = (US + GS + Unify-S).
3
where:
US is the Understanding Score,
GS is the Generation Score,
Unify-S is the Unify Score.
Each component score is calculated as described in their respective sections.
29

---

Published as a conference paper at ICLR 2026
Prompt for Generation
Text Multi-choice
Question: device used capture phatagrupls
Aiuv help jesurutic auste coromon vonue qhestan, idrntiging plante and onimols.
Queation: dinice copture photographs,
essential in both art and journaiion Choice:
essentiol both journaiion.
Choice:
4. Smorrphone
Camera
Go
Telemmpe
Image Multi-choice
(a) Common Sense Question Answering
Give the
Expintsrng: The torget etyer joradting aceeen stordingg wattig.
Multi-cbuice
womun in bieckdresse red rose in her hand
Pleuse grne
turget object specific editing
explatshygz:
requirement
8. Plerget oljxtt... Tieaglii aaling-
bike
hiinng retticcttig..
<arc imagen edft instruction>
Nog explatsigg The turget oiject wlitnng
The siget The ype writhng..
setting explatshgg The oijen eiiting cVdi unaritg strppd ahtre playveg gandie wrttins
cinsdigg
Thesenols
diffrrett axplonationr.
Image
Prompts Text Generation
Please generate three
Prompis
Ang-lnstraction)
Aifferent volii instr secions
Image
Avg-lostrartion22
Generation
Aigg Ang-instraction??
Edit Model
(b) Image Editing and Explaining
Imago Multi Multi-choice
Prompts Image
Image_A
oxty Alfirent puito eccorsiing alffreat pooiiios, and gesancte incerrect
Aind sue
diforsso
Imagen
done as the sumber of dgfeast count, gevente tsmplate aptioes
Different locations
Qucation: dioe difereenur
Chatee
Prompts
d:17
8-16
0:15
(c) SpotDiff
Prompts for Generation
Geometry3K
Please revort this istve according to the options and tost Neenats.
Questton: In trinngir ANC, point totr the foet ofe perpendicular from
and ZACH determine the leugth
Fraen
Choice:
Ansuer:
Image Multi-choice
2.A4
Ausiliary required
cheds
sec image
A(answer)
(d) Auxiliary Lines
New the player ot (3.3))))
ployer (2.2).
nextt strp oction Up
step oction
API
coorsinntt (2.2)
cvorelinat (1.2
Fimiseed
Open ALAPI
Initial Image
Step1
AP
Stop2
Stop3
(e) Visual CoT
Figure 9: The overall construction process for five unified tasks, which consists of (a) Common Sense Question Answering, (b) Image Editing and Explaining, (p)oDf,,iff, (d) Auxiliary Lines, and (e) Visual COT.
30

---

Published as a conference paper at ICLR 2026
Run for Rights
RU FOR RIGHTO
RUN FOR RIGHTS
Run for Rights
RUN POPNT
28
28
(a) Ground Truth.
(b) DALLE-2.
(c) DALLE-3.
(d) Gemini2. O-flash-exx.
F SFT ROLE
00
(e) EMU3.
(I) GILL.
HermesFlow.
(h) Janus-Pro.
88
(i) MiniGPT-5.
(j) MIO-Instruct.
(k) Show-O.
() Vila-u
Figure 10: The generated results from various models in the text-to-image generation task, based on the following text prompt: A man is standing in park with a 'Rnn for Rights' banner in the background. He is wearing a white t-shirt with the number 28 it, grey shorts, and grey socks with black shoes. The park is filled with people, some sitting on benches, and there is a bicycle leaning against a tree.
(e) Demini2.0-flass
(a) Anole.
(b) GILL,
(c) SEED.
(d) MiniGPT-S.
exp.
Figure 11: Intermediate process images generated by different models in VCoT. The figure illus- trates the intermediate outputs of various models the VCoT (Visual Composition Task), showing distinct approaches in processing and generating visual content. The models shown include (a)
Anole, (b) GILL,, (c) SEED, (d) MiniGPT-5, and (e) Geminii2.0--ftssh-eexp. each producing unique visual patterns and compositions.
31

---

Published as a conference paper at ICLR 2026
(a) Source Image.
(b) OmniGen.
(c) GILL.
(d) MiniGPT-S.
(c) Anole.
(f) SEED-LLaMA.
(g) MIO-Instruct.
(h) Gemini2. 0-flash-exp.
Figure 12: The generated results from various models in the fine-grained image reconstruction task, based on the following text prompt: Reconstruct high-fidelity images from degraded inpuls, preserving fine- grained details, textures, and structural integrity with perceptual realism.
ING
LGLRA
EACY
AA
A&
E TIAA
(a) Source Image.
(b) Ground Truth.
(c) GILL.
(d) MiniGPT-5.
P Paay
Anole.
SEED-LLaMA.
Gemini2.0 Seemmn2.0--aash-epp (g)
MIO-Instruc..
Figure 13: The generated results from various models in the text-guided image editing task, based on the following text prompt: Change this image into a watercolor art.
32

---

Published as a conference paper at ICLR 2026
(a) Source Image.
(b) Cog VideoX.
(c) MIO- Inst-utt
Figure 14: The generated results from various models in the conditional image-to-video gener- ation task, based on the following text prompt: The man is so tired. -camera zoom in.
33

---

Published as a conference paper at ICLR 2026
(a) Cog VideoX.
(b) MIO-Instruct.
The generated results from various models in the text-to-video generation task, based on the following text prompt: Men wearing sunglasses and women with hats take photographs of themselves.
Figure 15:
34

---

Published as a conference paper at ICLR 2026
Image Editing and Explaining
Src Image
Ground Truth
Edit Instruction: Add the phrase ("FAST FOOD!" " in small letters.
Anole Response
Output Explanation: This image features burger and French fries from a fast-food restaurant. The phrase "FAST FOOD" is written in small letters, applying artistic touch the image while also accurately reflecting the content.
Output Image
GILL Response
Output Explanation: DOG DARE YOU
Output Image [NNGG/NGGCJJNGG///GG/RMGA/MAAAAA MG7J.
MiniGPT5 Response
Output Explanation: I'm sorry, but your question is bit vague and doesn 't provide enough context for give an accurate response. Could you please rephrase or clarify the task? Are we talking about adding text in general oor specifically when it comes small letters ? [IMGO]
11660
Output Image
Response
Output Explanation: Successfully added
Output Image
phrase (FFAST FOOD)" in small letters to the image.
Figure 16: The generated results from various models in the image editing and explaining task.
35

---

Published as a conference paper at ICLR 2026
Common Sense Question Answering
Question: Ancient monumental tombs located in Egypt near the city of Cairo.
Ground Truth
Colosseum Stonhenge
C. Machu Picchu D. Pyramids of Giza
Answer: D
Anole Response
Output Image
Output Choice:
GILL Response
Output Image
Output Choice:
MiniGPT5 Response
Output Image
Output Choice: D
SEED LEED-LLaMA Response
Output Image
Output Choice: D
Figure 17: The generated results from various models in the common sense question answering task.
36

---

Published as a conference paper at ICLR 2026
Auxiliary Lines
Question: In circle O, the radius 10 units, and chord AB is 16 units long. The distance from the center of the circle O to the midpoint P of AB is 6 units. Using this information, calculate the area of triangle OAP
Src Image
Choices: A. 11 B. 36 28 24 Answer: D
Anole Response
Output Image
Output Choice: B
GILL Response
Output Image
Output Choice:
MiniGPT5 Response
Output Image
Output Choice: A
Gemini2.00Fllah-EEe Exp Response
Output Image
Output Choice: A
Figure 18: The generated results from various models in the auxiliary lines task.
37

---

Published as a conference paper at ICLR 2026
SpotDiff Src Image 1
Src Image 1
Ground Truth
Anole Response
Output Image
Output Choice: B
MiniGPT5 Response
Output Image
Output Choice: C
SEED-LLaMA Response
vns Chokmere
Output Image
Output Choice: A
Gemini2.0-Flash- Response
Output Image
Output Choice: C
Figure 19: The generated results from various models in the spotdiff task.
38

---

Published as a conference paper at ICLR 2026
Common Sense Question Answering
System Prompt:
You are an Al system that answers common-sens, knowledge questions by selecting the correct answer from multiple choices and then generating image that visually represents the answer.
Input Data
Question: factual question requiring knowledge-basse reasoning.
Choice: A set of multiple-choice answers labeled A, B, C, and
Output Requirements
Answer Selection:
Analyze the question and determine the correct based general knowledge.
Output the selected answer the format: Answer: (where X
C, or D).
Image Generation:
Generate an image that visually represents the content of the chosen answer:
Processing Steps
Understand the Question: Extract key information from the question.
Evaluate the Choices: Compare each option and determine the most accurate
choice in required format. Generate Image: Create an image that correctly depicts the content the selected answer.
Select Correct Answer: Output the
Verify Coherence: Ensure the generated image aligns with the chosen answer.
Example
Input:
question: "Which planet known as the Red Planet?",
choice:
"A: Earth",
Mars",
"C: Venus",
"D: Jupiter"
Model Output:
Answer: B
<image> (Generating image of the Mars)
Figure 20: System prompt for Common Sense Question Answering task.
39

---

Published as a conference paper at ICLR 2026
Image Editing and Explaining
System Prompt:
You are an Al-powered image editing assistant. Your task is to modify provided initial image based question instruction and generate clear visual description of the edited object.
Input Data
Question: A natural language instruction specifying how the image should be modified.
Initial Image: The original image that needs editing.
Output Requirements
Explanation:
Identify the target object or region in the image that needs to be modified.
Provide concise visual description of the object before and after modification.
Clearly describe how the edit integrates into the scene.
Edited Image:
Generate an image that precisely follows the question instruction while ensuring realism and coherence.
Maintain the original image's quality, lighting, and perspective in the edited version. Processing Steps Analyze the Question: Extract key editing instructions (e.g. add, remove, modify, change color, reposition).
Identify the Target Object: Locate the relevant object or scene element that needs modification. Generate a Visual Description: Clearly describe the object before and after editing, ensuring it aligns with the given instruction. Apply the Modification: Edit the image accordingly, ensuring seamless integration with existing elements.
Verify Output: Ensure the modification meets the instruction while preserving natural aesthetics.
Example
Input:
Question: Add a fork to the plate.
<image>
Model Output:
Explanation: The target object for editing is the plate containing steak, potatoes, and mixed vegetables, with a slice of orange for garnish. The specific editing requirement is to add a fork to the plate, ensuring it complements the arrangement of the existing food items.
<edited image>
Figure 21: Systemp prompt for Image Editing and Explaining task
40

---

Published as a conference paper at ICLR 2026
SpotDiff
System Prompt:
You are an Al system designed to analyze two similar images (Img.a and img. and identify the number of differences between them. Your task is to:
Compare img img_b find differences.
Select the correct
from the provided multiple-choice options.
Extract the different regions from img. and place them white background of the same size. Input Format
Img_a: The first image.
img. The second image (similar but identical to img.a).
choice:
Multiple-choice answers indicating different counts differences, labeled as B,
Example Input:
img.e a: "<image_a>",
img b: "simage_bz" b>""
choice: "A: 14",
"C: 19",
"D: 10"
Output Format
Answer Selection:
Identify correct number differences and output the answer in the format:
Answer: (where Is
D)
Extracted Difference Image:
Identify regions img.a that differ from img b.
Extract these differing regions and place them a white background of the same size as img.a. a.
The final image should highlight only the different areas while preserving their original details.
Example Output:
Answer: B
<image> (Extracted difference regions placed white background)
Processing Steps
Compare img and img.h b to identify all differences (object position, shape, color, missing parts,
etc.). Count
multiple-choice answer. Extract differing regions from img_a and overlay them white background of the size. Output the selected answer and the processed image.
differences
match
Key Requirements Strictly select one answer from A, B, C,
Ensure extracted differences are accurately placed on clean white background. Maintain the original structure of differing regions (no modifications, extraction).
Figure 22: System prompt for SpotDiff task.
41

---

Published as a conference paper at ICLR 2026
Auxiliary Lines
System Prompt:
You are an Al system designed to solve junior high school geometry problems. Your task to:
Analyze the given geometry question, image, multiple-choice answers.
Draw auxiliary lines the geometric diagram assist problem-solving.
Determine the correct
based the problem's conditions.
Input Data
Question: A geometry-related word problem describing angles, lengths, relationships.
Image: geometric diagram corresponding to the problem statement.
Choice: A set of multiple-choice answers labeled A, B, C, and D.
Output Requirements
Answer Selection:
Use geometric reasoning to determine correct answer. Output the selected answer the format:
(where Dr.
Image with Auxiliary Lines:
Draw necessary auxiliary lines (such perpendiculars, bisectors, diagonals) the geometric diagram facilitate solving.
Ensure the lines are clear and logically placed according the problem's constraints. Maintain the original structure of the diagram while highlighting the new construction. Processing Steps
Understand the Problem: Analyze given conditions (parallel lines, angles, lengths, etc.). Identify Key Geometric Properties: Determine the relationships between elements the diagram.
Auxiliary Lines: Add necessary constructions simplify calculations. Solve for the Answer: Apply geometric theorems and algebraic calculations. Output Answer and Edited Image: Provide the correct answer and the diagram with auxiliary lines.
Example
Input:
question:
"Given the quadrilateral ABCD, where line segment AB is parallel to line segment DC, measure LABC 60", the measure LADCis 45". Additionally length units, the length
units. Determine the perimeter quadrilateral ABCD.",
choice:
26 (|sgr((3] ) \/qrrt (6)",
"B: 26 4 \\sqrt [3) (6)"
5C:52 \/sqrt (33) \/grr(66)
"D: [3) V\sqrt ["
(geomery diagram)
Model Output:
<image> (image with auxiliary lines)
Figure 23: System prompt for Auxiliary Lines task.
42

---

Published as a conference paper at ICLR 2026
Visual CoT
System Prompt for first step:
You are given grid-based puzzle game map where each grid square can either be safe square (land) or hole. Your goal is to reach the target while avoiding the holes and using as few moves possible. You can move in four directions: Left, Right, Up, Down. The grid is 3.3.The top-left cell is (0,0), the top-right cell Is (2,0), the bottom-left cell is (0,2), and so downward, and columns increase to the right.
increase.
** *Game Settings:**
The grid map fully observable.
The player starts designated grid square.
The goal is located elsewhere the map.
Each grid square either safe (land) or contains hole (non-safe).
The player must avoid holes, moving into hole results in failure.
The objective is guide the player to the goal without falling into holes.
"Movement Rules:**
The player can move left, right, up, or down to an adjacent square, provided is a safe square.
The player cannot move more than one square at time.
Moving outside the edge of the map has no effect. The player stays in the same position.
Do not fall into holes.
The player wins by reaching the goal.
*Your task:**
Based on the current state of the game, decide the next move for the player.
Provide the next action: "Left", "Right", "Up", "Down".
After selecting the action, specify the coordinates the player's location [x.
Also, output representation of the grid map after the selected action.
**Output *
Action: (Your move choice]
Location: [x. y]
Image: [Generated Image]
Here the Initial grid map: (Shown Initial Figure)
Please choose the next move and give output:
Figure 24: Systemp prompt for Visual CoT task in the first step.
43

---

Published as a conference paper at ICLR 2026
Visual CoT
System Prompt after first step:
You are given grid grid-based puzzle game map each grid square can (land) hole. Your goal reach the goal while avoiding holes and using a safe square possible. You move four directions: left, right, up, down. The grid is 3x3. The top-left cell is (0,0), the top-right cell is (2,0), the bottom-left cell (0,2), and downwards and columns increase rightwards.
Rows increase
**aame Setup:**
The grid map is fully observable.
The player starts at designated grid square.
The goal is somewhere else on the map.
Each grid square either (land) contains (non-safe).
The player must avoid holes, and entering hole will result in failure.
The goal is guide the player the goal without falling into hole.
"Movement Rules:
The player can move left, right, or down to an adjacent square, provided is safe square.
The player cannot move more than one block at time.
Moving beyond the edge the map has no effect. The player remains the same position. not fall into hole.
The player wins by reaching goal.
"Your Task:**
Determine the next
for the player based the initial grid map, the history information,
and the current state of the game.
Provide the next action: "Left", "Right", "Up", or "Down", and output "Finish" if you think the
goal position has been reached
After selecting an action, specify the coordinates of the player's 's new position as [x, y).
Also, output a representation the grid map after the selected action.
Please provide action, coordinates the maze image of player's new position
step
This is the initial grid map: (Showing Initial Map)
Here is the state of the game after last step: "Histoory Information:
action (e.g. Right", "Go Down", etc.). Current position.
image the grid after last Initial grid map:
**Output format:**
Action: fyour move selection]
Location: [x, y]
Image: [generated image]
select next step and give output:
Figure 25: Systemp prompt for Visual CoT task after first step.
44


Reviews:
[{"review_id": "8PdwlYlrU8", "reviewer": "Reviewer_xuiz", "summary": "This paper introduces MME-Unify (MME-U), a new and comprehensive benchmark designed to evaluate Unified Multimodal Large Language Models. The authors identify a critical gap in existing evaluations: the lack of a standardized benchmark that assesses comprehension, generation, and mixed-modality generation capabilities simultaneously. MME-U addresses this by integrating tasks into three domains: (1) Multimodal Understanding, (2) Multimodal Generation, and (3) a novel set of unify tasks. The Unify tasks are specifically designed to test how a model's understanding and generation capabilities can mutually enhance each other, featuring five new subtasks like Visual CoT and Auxiliary Lines . The authors evaluate 12 unified MLLMs and find that current models have significant room for improvement.", "questions": "- The evaluation is missing several key sota models (e.g., Gemini 2.5-Pro, Bagel, Qwen-image). \n- See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper's primary contribution is the introduction of five unify subtasks. This is the first standardized benchmark designed to rigorously assess the synergistic mixed-modality generation capabilities of U-MLLMs (e.g., reasoning before drawing), which has been a major gap in the field.\n- The benchmark curates and unifies a wide array of tasks from 12 existing datasets for understanding and generation. By reformatting all understanding and unify tasks into a standardized multiple-choice format and normalizing generation metrics, MME-U provides a consistent and reproducible framework for model comparison.\n- The evaluation of 22 models provides a clear snapshot of the current U-MLLM landscape. The results effectively highlight the performance gap between models and demonstrate that even top-performing models struggle with complex instruction following and multi-step unified tasks .", "weaknesses": "- The main weakness is the evaluation strategy for image generation in the \"Unify\" tasks. Using CLIP-I similarity to match a generated image against multiple-choice image options can be hacked. \n- While the Unify tasks are novel, the benchmark's Understanding and Generation sections are primarily a \"benchmark of benchmarks,\" curating tasks from many existing sources. This makes the overall contribution feel more incremental than groundbreaking.\n- The paper reports findings but fails to sufficiently discuss how its conclusions diverge from or challenge those of existing benchmarks. The analysis would be significantly stronger if it highlighted unique insights or model ranking differences revealed only by MME-U. This discussion is crucial for demonstrating the distinct value of this new comprehensive benchmark.\n- The benchmark appears to rely heavily on synthetic data generation, using models like GPT-4o to create QA pairs, explanations, and negative samples. The methodology for quality control and ensuring these synthetic data points are accurate, unbiased, and sufficiently challenging is not well-detailed, raising concerns about data quality and potential artifacts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JnmvRbCWSZ", "reviewer": "Reviewer_uqqm", "summary": "This paper introduces MME-Unify (MME-U), the first open benchmark for Unified Multimodal Large Language Models (U-MLLMs). It evaluates U-MLLMs across three core capabilities: understanding, generation, and hybrid (Unify) multimodal generation. MME-U integrates 12 datasets, standardizes conventional tasks, and designs five unique hybrid generation sub-tasks (e.g., Visual Chain-of-Thought and geometry problem-solving with auxiliary lines). Evaluations on 12 U-MLLMs reveal that current models exhibit significant deficiencies in balancing different capabilities, following complex instructions, and performing multi-step unified tasks.", "questions": "Given the potential risk of exploitation in CLIP-I-based evaluation, do the authors plan to combine text generation metrics or use an LLM judge to perform a holistic assessment of generated images and text, in order to achieve a more rigorous evaluation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) MME-U is the first benchmark for standardized unified multimodal generation tasks (unify capability). These tasks require models to collaboratively integrate reasoning with multimodal outputs, which is quite novel and also fills certain gaps in existing evaluations.\n(2) MME-U provides a unified scoring framework for comprehension, generation, and unified tasks. It standardizes complex metrics to a [0, 100] scale, offering an intuitive and comparable overall MME-U score.", "weaknesses": "(1) In the unified tasks, image evaluation relies on multiple-choice questions based on CLIP similarity, which may allow models to exploit the evaluation. This simplification reduces the rigor in assessing generation quality and precise adherence to instructions. Additionally, if the options are very similar, such an evaluation may also be inaccurate.\n(2) On multi-step reasoning tasks like Visual CoT, none of the models succeeded, with Acc results at 0%. This indicates that the task is overly difficult and may not provide discriminative evaluation value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KStX51BX7R", "reviewer": "Reviewer_VRFA", "summary": "The paper introduces MME-Unify (MME-U), an open and reproducible benchmark targeting Unified Multimodal LLMs (U-MLLMs). It evaluates three capability axes: multimodal understanding, multimodal generation, and “unify” (mixed-modality generation that couples understanding with generation). To enable cross-task comparability, the authors convert understanding tasks into multiple-choice QA and standardize diverse generation metrics onto a common (0,100) scale. A key contribution is five unify subtasks—Image Editing & Explaining, Common-Sense QA + image generation, Auxiliary Lines for geometry, Spot-the-Difference, and Visual Chain-of-Thought (maze navigation)—intended to measure how understanding and generation reinforce each other. The benchmark aggregates 12 datasets into 30 subtasks and evaluates 22 models (open/closed, understanding/generation specialists, and U-MLLMs). Results show that even the strongest systems struggle on unify tasks and that instruction following and visual detail alignment remain open problems.", "questions": "**On the Trustworthiness of Scores: How do you prevent \"benchmark hacking\"? ** Your evaluation relies heavily on proxy metrics like CLIP scores and handcrafted negative sample matching. \n\nWhat evidence can you provide that a high score on MME-U strongly correlates with the generation of high-quality, high-fidelity outputs that strictly follow complex instructions?\n\nHave you considered incorporating evaluation dimensions that are harder to \"hack,\" such as human preference scores or more robust automated metrics (e.g., object-level editing consistency checks), to anchor your benchmark in real-world performance?\n\n**On the Diagnosability of Failures: How do your results help us locate and fix problems?**  A benchmark should not just assign a score; it should provide a diagnosis. Currently, a 0% success rate on a task like VCoT tells us that \"all models fail,\" but it offers little insight into where they fail in the process, limiting its value for model developers seeking actionable feedback. Could you introduce milestone-based scoring for multi-step tasks like VCoT, reporting metrics such as \"path recognition accuracy for step 1\" and \"visualization quality for step 1\"?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Clear problem motivation**: The benchmark directly targets the unique mixed-modality generation capability that prior benchmarks do not quantify. The unify tasks are not a simple concatenation of existing tasks but require genuine coupling of understanding and generation.\n\n**Breadth and coverage**: The suite spans single/multi-image and video inputs, plus text-to-image/video, editing, image-to-video, and prediction on the generation side. Table 1 convincingly positions MME-U as broader than existing benchmarks.\n\n**Transparent pipeline and reproducibility**: The attribute unification pipeline, domain-wise metrics with standardized scoring, rule-based output matching, option randomization, and model capability adapters (e.g., key-frame sampling for video) make the evaluation reasonably clear and replicable.", "weaknesses": "**Evaluation robustness (core concern)**: The scoring relies heavily on CLIP-based similarities and hand-crafted negative samples for both generation and unify tasks. This risks score hacking via feature-space proximity or style artifacts, and may not reflect human judgments on aesthetics, editing faithfulness, or geometric correctness. While the paper acknowledges this limitation, the current version lacks human evaluation or adversarial analyses to quantify the bias or alignment with human preferences.\n\n**Unify task share vs. weighting**: Unify subtasks comprise 546 QA items out of 4,104 total (≈13%) but constitute one of the three equally weighted top-level scores (≈33% of the final MME-U score). This mismatch can increase variance and reduce reliability, especially given the small per-subtask sizes (e.g., AL=52, VCoT=90). The paper does not report confidence intervals or stability analyses to mitigate this concern.\n\n**Diagnostic resolution on challenging unify tasks**: The authors provide stepwise accuracies for VCoT (action/coordinate/image), which is helpful, yet overall success is near zero. Without difficulty stratification (maze sizes/horizons) or milestone scoring, it remains hard to pinpoint where reasoning breaks down and to guide targeted improvement.\n\n**Missing baselines for calibration**: No random-guess baselines (e.g., 25% for 4-way MCQ), simple heuristic baselines, or human upper bounds are reported. This makes it harder to interpret absolute scores.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ItJmXedMTC", "reviewer": "Reviewer_QMnN", "summary": "This paper proposes MME-Unify, a new benchmark to evaluate Unified Multimodal Large Language Models. It solves the problem that existing studies lack a unified way to test MLLMs’ understanding, generation, and mixed-modality abilities. MME-Unify uses data from 12 datasets, unifies task formats (like multiple-choice questions) and metrics, and designs 5 mixed-modality “unify tasks”. The authors tested 12 MLLMs, finding Gemini2.0-flash-exp performs best, but all models still struggle with balance between understanding and generation, and mixed-modality tasks.", "questions": "A minor issue: The layout of Figure 3 needs to be adjusted. The images in this figure are too small, making it difficult to clearly see the details.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is generally well-written and easy to follow.\n2. This benchmarks evaluates the comprehensive abilities of unifed multimodal models, including multimodal understanding, generation, and mixed-modality integration.\n3. The 5 designed “unify tasks” (e.g., drawing auxiliary lines for geometry problems) effectively test how unifed multimodal models combine understanding and generation, targeting their most unique feature.", "weaknesses": "1. My biggest concern about this paper is the way the paper evaluates image generation—using CLIP score and multiple-choice questions. CLIP score only checks overall semantic similarity, not whether the generated image truly follows the prompt. SEED-Bench (a 2023 work) adopted this metric for evaluation of unified mutlimodal models, and by 2025, there should be more suitable metrics.\n2. When using multiple-choice questions for image generation evaluation, the paper does not check if the generated image’s details match the prompt—only if it is similar to the option images. This means a model might \"pass\" the test without actually following the task’s requirements.\n3. The paper does not evaluate newer models like Gemini-2.5-Pro or Bagel, so we cannot know how these new models perform on the MME-Unify benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)