MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?
Authors: YiFan Zhang
Huanyu Zhang
Haochen Tian
Chaoyou Fu
Shuangqing Zhang
Junfei Wu
Feng Li
Kun Wang
Qingsong Wen
Zhang Zhang
Liang Wang
Rong Jin
Venue: ICLR 2025 Poster (2025)
Keywords: multimodal Large Language Models
benchmark
high-resolution images
real-world scenarios
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce MME-RealWorld, the largest manually annotated benchmark for evaluating Multimodal Large Language Models, featuring over 29,000 question-answer pairs and high-resolution images to address significant challenges in real-world scenarios.

Abstract: Comprehensive evaluation of Multimodal Large Language Models (MLLMs) has recently garnered widespread attention in the research community. However, we observe that existing benchmarks present several common barriers that make it difficult to measure the significant challenges that models face in the real world, including: 1) small data scale leads to a large performance variance; 2) reliance on model-based annotations results in restricted data quality; 3) insufficient task difficulty, especially caused by the limited image resolution. To tackle these issues, we introduce MME-RealWorld. Specifically, we collect more than $300$ K images from public datasets and the Internet, filtering $13,366$ high-quality images for annotation. This involves the efforts of professional $25$ annotators and $7$ experts in MLLMs, contributing to $29,429$ question-answer pairs that cover $43$ subtasks across $5$ real-world scenarios, extremely challenging even for humans. As far as we know, **MME-RealWorld is the largest manually annotated benchmark to date, featuring the highest resolution and a targeted focus on real-world applications**. We further conduct a thorough evaluation involving $29$ prominent MLLMs, such as GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet. Our results show that even the most advanced models struggle with our benchmarks, where none of them reach 60\% accuracy. The challenges of perceiving high-resolution images and understanding complex real-world scenarios remain urgent issues to be addressed. The data and evaluation code are released in our Project Page.

Published as a conference paper at ICLR 2025
MME-REALWORLD COULD YOUR MULTIMODAL LLM CHALLENGE HIGH-RESOLUTION REAL-WOOL WORLD SCENARIOS THAT ARE DIFFICULT FOR HUMANS?
Yi-Fan Zhang Huanyu Zhang
Haochen Tian
Chaoyou Fu
Shuangqing Zhang² Junfei Wu
Feng Li
Kun Wang4.5 Qingsong Wen 8
Zhang Zhang Liang Wang
Rong Jin?
Tieniu Tan
CCSIIA, "NJU, SHKUST, 'NTU, SUAS, "Squirel AI Learning, "Meta Al
f Corresponding Author
ABSTRACT
Comprehensive evaluation of Multimodal Large Language Models (MLLMs) has recently garnered widespread attention the research community. However, we observe that existing benchmarks present several common barriers that make it difficult to measure the significant challenges that models face in the real world, including: 1) small data scale leads to large performance variance; 2) reliance on model-based annotations results in restricted data quality; 3) insufficient task dif- ficulty, especially caused by the limited image resolution. To tackle these issues, we introduce AME-RRa/Worldd Specifically, we collect more than 300 K images from public datasets and the Internet, filtering 13, 366 high-quality images for an- notation. This involves the efforts of professional 25 annotators and 7 experts in MLLMs, contributing to 29, 429 question-anss answer pairs that cover 43 subtasks across 5 real-world scenarios, extremely challenging even for humans. As far as we know, MME-Reallorldd is the largest manually annotated benchmark to date, featuring the highest resolution and targeted focus on real-world applications. We further conduct a thorough evaluation involving 29 prominent MLLMs, such as GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet. Our results show that even the most advanced models struggle with our benchmarks, where none of them reach 60% accuracy. The challenges of perceiving high-resolution images and understanding complex real-world scenarios remain urgent issues to be addressed.
INTRODUCTION
In recent years, we have witnessed significant flourish of Multimodal Large Language Models (MLLMs) (Dai et al., 2024; Liu et al., 2023b; Zhang et al., 2024; Fu et al., 2025). A primary ob- jective behind designing MLLMs has been develop general intelligent agents capable of compre- hensively perceiving human queries and environmental stituations through the integration of various multimodal sensory data. Consequently, plethora of comprehensive evaluation benchmarks have emerged to rigorously assess model capabilities. However, some common concerns also arise:
Data Scale. Many existing benchmarks contain fewer than 10K Question-Answer (QA) pairs (Fu et al., 2024b), such as MME (Fu et al., 2023a), MMbench (Liiu al., 2023c), MMStar (Chen et al., 2024), MM-Vet (Yu et al., 2024), TorchStone (Bai et al., 2023b), and BLINK (Fu et al., 2024c). The limited number of QA can lead evaluation fluctuations.
Annotation Quality. While some benchmarks, such as MMT-Bench (Ying et al., 2024) and SEED-Bench (Li et al., 2024b), are relatively larger in scale, their annotations are generated by LLMs or MILMs. This annotation process is inherently limited by the per- formance of the used models. In our benchmark, for example, the best-performing model, Intern merely achieves 50% accuracy. Consequently, relying on models would in- evitably introduce significant noise, compromising the quality of the annotations.

---

Published as a conference paper at ICLR 2025
OCR the Wild
Remote Sensing
Whu pf 8380 597722
Whs cole connart
(A Velle
1X Baak
50 Th luag fatus
Mari the adran spiiiing BRIN'S MARE FOR WOLLA PIM
pursacatt
(N1 nOn VIU MAOLUHH ATLLOW
HHN NAARE DUMIRIAN NROANN!
Video Monitoring
Driving
Wiw Bi opn Whare
cenirs imap??
on But maldi??
(N Tarn lelt
50 Avep anli
Brinct Siiiier
Vhiee srrgle"
Enco lof
Socius
Diogram and Toblle
Energy
Maatr pescetage CAFEX vico cincd S0 15 acsccs
Whit And Mende Mick Parm Phien?
(A) 12 su uC tur d81 1..1 bage del ta DATA
Figure 1: Diagram of MMr-RealWorid. Our benchmark contains 5 real-world domains, covering 43 perception and reasoning subtasks. Each QA pair offers options. highlight and magnify the image parts relevant to the question in a red box for better visibility.
Task Difficulty. To date, the top performance of some benchmarks has reached the accu- racy of 80%-90% (Mathew al., 2021; Masry al., 2022; Singh al., 2019; Liu al., 2023e; Li et al., 2023c), and the performance margin between advanced MLLMs is narrow. This makes it challenging verify the benefits improvements of advanced models and to distinguish which one is significantly better.
In light of these concerns, we propose new benchmark named MME-RealWorld, We first pay attention to a series of well-motivated families of datasets, considering images from sources such as autonomous driving, remote sensing, video surveillance, newspapers, street views, and financial charts. These scenarios are difficult even for humans, where we hope that MLLMs can really help. Considering these topics, we collect total of 13, 366 high-resolution images from more than 300K public and internet sources. These images have an average resolution of 2, 000 1, 500, containing rich image details. 25 professional annotators and experts MILMS are participated annotate and check the data quality, and meanwhile ensuring that all questions are challenging for MLLMs. Note that most questions are even hard for humans, requiring multiple annotators to answer and double-check the results. As shown in Fig. 2(a), MealReellWorld finally contains 29, 429 annota- tions for 43 sub-class tasks, where each one has at least 100 questions. 29 advanced MLLMs are evaluated benchmark, along with detailed analysis. We conclude the main advantages of compared to existing counterparts as follows:
Data Scale. With the efforts of total of 32 volunteers, we have manually annotated 29, 429 QA pairs focused real-world scenarios, making this the largest fully human-annotated benchmark known to date.
Data Quality. 1) Resolution: Many image details, such scoreboard in sports event, carry critical information. These details can only be properly interpreted with high- resolution images, which essential for providing meaningful assistance to humans. To
2

---

Published as a conference paper at ICLR 2025
Prosher' Mcherbhanee
Mutt-Cluu Contiie
baottin Ereixdinn
Object
Honalyy Informatioo
Model
Ace
Model
Ase
Ortesbadia Poceppiie
Bast, Man Map Pattr
Gwerr1-22
Quwentt-22
Color Broopetie
Sigener Oiher Taut
Interwi-22
53.5
Internii-a
54.3
Eedentaas Eatene
Breogatton
Cloude 3.5
Fineretia
Charis Fndersanciing
Petestrion Conrtiyg
Deeernl 4l-25
Gaade 3,5 Sernet
Vikice Lucetin
Offjcct Connting
TntereVL- -Chsh--------55
49.4
SIME-0R
45.8
Mmn-Gamn-340-10
45,0
YE-VL-44B
42.0
Vollete Faistonce
Maafturing
Video
DiR Wille
Spaiisl Weipfirndipp
MiniPP V 2.5
45.6
CoglL/N2
39,8
6PT-46
45.2
SIME-138
38.9
Volacie Caunting
Remote Seosing
Krespaition
Combrion-344 34B
44.1
Nini mri--emiiii-oo Nri-6emii--44-HH0 34B-HD
38.5
Inteotiass
Diagram Percrprion
Contruon800
42.7
Nenwey
$9.2
Pelesrite lutcation
Dingrom n0
Dole Poresption
88
Noxh
Eig Inieationn
Calrubdive (Tabl))
Gevini: 5 pro
38.7
Genbrion 348
35.7
Nignal Atceaion
Cabolution (Dlagrons)
GPT 4o
36.4
Scnini
34.9
Nututboobbp (E2P)
Comparisdos (Dlggran)
Morkey
IrterrEM6-2.5
Rebebconbig (OKO)
Cxepariiion (Tabd)
m'LLG--eeCOal
32.7
Corbnonn50
33.6
Betstrokiig LERY
Parmaniag (Rlggros)
DeepSeok.
32,4
LLaVA-Nert-785
30.6
Javnns Peroplo
Forraster (Tabk)
SHME 138
NPLUS bocOwl
Omjent Caronss
Moll: Vobilee Matin
VZ ML 3488
31.0
Semini 1. pro
28.1
Tralfe Soov
Talaeaa Mellen
Volice Malle
30.
LaVA-Naut-Bl
DeepSinek-V1.
27.6
28.7
TextMorkey
26.4
UsaVa1
28.0
6PT-4p-mini
ShoredPTTV 138
Shard6FT4y. 138
(a) Real-World Tasks
(b) Leaderboard
Figure 2: Task Categories (left). Our benchmark spans 5 key domains and 43 subtasks highly related to real-world scenarios, including 13, 366 high-resolution images and 29, 429 annotations. Model Performance (right). Average accuracies of advanced MLLMs are shown across both the English and Chinese splits of the dataset.
the best of our knowledge, MME-Re RealWorld features the highest average image resolution among existing competitors. 2) Annotation: All annotations are manually completed, with professional team cross-checking the results to ensure data quality.
Task Difficulty and Real-World Utility. The performance of different MLLMs is shown in Fig. 2(b), in which we can see that even the most advanced models have not surpassed 60% accuracy. Additionally, illustrated in Fig. 1, many real-world tasks are significantly more difficult than those in traditional benchmarks. For example, in video monitoring, a model needs count the presence of 133 vehicles, in remote sensing, it must identify and count small objects on a map with an average resolution exceeding 5000 x 5000.
MME-RealWorld-CNN Existing Chinese benchmark (Liu et al., 2023c) usually trans- lated from its English version. This has two limitations: 1) Question-image mismatch. The image may relate to an English scenario, which is not intuitively connected to a Chi- nese question. 2) Translation mismatch (Tang et al., 2024). The machine translation is not always precise and perfect enough. We collect additional images that focus on Chinese scenarios, asking Chinese volunteers for annotation. This results in 5, 917 QA pairs.
MME-REALWORLD
In this section, we outline the data collection process, question annotation procedure, and provide statistical overview of each domain and subtask in MME-RealWorld and its Chinese version. We visualize different tasks from the image domains in Fig. Detailed information on data sources, evaluation tasks, and visualized results can be found in Sec. B.
2.1 INSTRUCTION AND CRITERION
For each question, we manually construct four options, with one being the correct answer and the other three being the texts appearing in the image options similar to the correct one. This greatly
3

---

Published as a conference paper at ICLR 2025
Table 1: Prompt setting of MME-RealWordd
[Image] [Queation] The choices listed below:
(A) [Cheice A]
(B) [Choics B]
[Choice
(D) [Cheice D]
(E) [Choice B]
Select the best
to the above multiple-choice question based the image. Respond with
only the letter (A, B, C, D, or E) of the correct option.
'The best answer is:
enhances the difficulty, forcing the model to deeply understand the details of the image. We also provide additional choice E, which allows the model to reject for answering because there is no right answer. We try to use the model's default prompt for multiple-choice questions, but if the model does not have the default prompt, we use common prompt as shown in Tab. 1.
Evaluation Metric. We first apply a rule-based filter to the answers generated by MLLM, aligning them with the given answer options and checking for correctness against the ground truth. Let the dataset be denoted as D {D& = (TT) )2 where each domain Da consists of Ta subtasks. For each subtask, we calculate the accuracy across all annotations. For each domain, we compute two metries: 1) Average Accuracy (Avg). the weighted average accuracy across all subtasks, given by EE Avg(T.) x [Til/|Dal, where | is the instance number contained one set, and 2) Class- based Average Accuracy (Avg-C). the unweighted average accuracy across subtasks, given by IT Ii Avg(TT)/(a: Similarly, for the entire dataset, we report the overall Average Accuracy across all samples, and the class-based average accuracy across domains.
2.2 DATA COLLECTION AND ANNOTATION
Optical Character Recognition in the Wild (OCR). It is specifically designed to evaluate the model's ability to perceive and understand textual information in the real-world. We manually lecte 3, 293 images with complex scenes and recognizable text information from 150, 259 images in existing high-resolution datasets our image sources. These images span various categories such as street scenes, shops. posters, books, and competitions. The volunteers are worked for annotation, each with least foundational understanding of multimodal models, to independently generate questions and answers. These annotations are subsequently reviewed and further refined by another volunteers. Based on the image annotations, we categorize these 3, 297 images into 5 perception tasks, totaling 5, 740 QA pairs: contact information and addresses, identity information, products and advertisements, signage and other text, as well as natural text recognition in clevation maps and books. Additionally, there are two reasoning tasks with 500 QA pairs: 1) scene understanding of the entire image, which requires the model to locate and comprehend important text such as compe- tition results, and 2) character understanding, focusing on comics posters where the model needs to analyze relationships and personalities based on dialogue or presentation.
Remote Sensing (RS). The images have wide range of applications in real-world scenarios. Some images possess extremely high quality, with individual image sizes reaching up to 139MB and con- taining very rich details, which makes it difficult even for humans to perceive specific objects. We manually select 1, 298 high-resolution images from over 70, 000 public remote sensing images, en- suring that each image is of high quality, with sufficient resolution and rich detail. One professional researcher is involved in annotating the data, and another researcher checks and improves the annota- tions, resulting 3, 738 QA pairs. There are perception tasks: object counting, color recognition, and spatial relationship understanding.
Diagram and Table (DT). Although there are already some datasets related to table and chart un- derstanding, they mostly feature simple scenes. We focus on highly complex chart data, such as financial reports, which contain extensive numerical information and mathematical content, pre- senting new challenges for MLLMs. We filter 2, 570 images from the internet, with annotations performed by two volunteer and reviewed by another one. We categorize these annotations into tasks based on the question format: 1) Diagram and Table Perception (5, 433 QA pairs): involve locating specific values of elements within the diagrams and tables; 2) Diagram Reasoning (250 QA pairs): include tasks such identifying the maximum and minimum values in a chart, performing
4

---

Published as a conference paper at ICLR 2025
simple calculations, and predicting trends; and 3) table Reasoning (250 QA pairs): focus simple calculations related to specific elements, understanding mathematical concepts like maximum and minimum values, and locating corresponding elements.
Autonomous Driving (AD). It demands extensive general knowledge and embodied understanding capability. We emphasize challenging driving scenarios that involve distant perceptions and intricate interactions among dynamic traffic agents. Specifically, we manually select subset of 715 images from over 40, 000 front-view images captured by onboard cameras in open-source datasets. These images cover a diverse range of weather conditions, geographic locations, and traffic scenarios. Besides, a volunteer carefully annotates each image, and the other one conducts a thorough review, resulting in 3, 660 QA pairs for perception tasks and 1, 334 QA pairs for reasoning tasks. The perception tasks include objects identification, object attribute identification, and object counting for traffic elements such as vehicle, pedestrian, and signals. The latter is categorized into 3 main tasks: 1) Intention Prediction: focus on predicting driving intention of designated traffic agent in the short-term future. 2) Interaction Relation Understanding: involve reasoning about ego vehicle's reaction to other traffic elements, and the interactions between these elements. 3) Driver Attention require reasoning about the traffic signal that the driver should pay attention to.
Understanding:
Monitoring (MO). The images are from various application scenarios for public safety, e.g., streets, shopping malls, and expressway intersections. We focus complex high-resolution monitoring images that include many real-world challenges, like scale variations and out-of-view, as possible which could test whether the model handles them robustly in practice. Specifically, 1, 601 high- resolution images are manually selected from over 10, 000 public dataset images, which are captured from broad range of cameras, viewpoints, scene complexities, and environmental factors across day and night. In terms of annotations, two volunteers manually annotate each image carefully, and multi-stage careful inspections and modifications are performed by another one. When these refined image annotations are completed, 1, 601 images are categorized into 3 main perception tasks, total- ing 2, 196 QA pairs, including object counting and location, and attribute recognition. Furthermore, 3 reasoning tasks are well-designed with 498 QA pairs: 1) calculate the sum of different objects, which requires the model to perceive various objects and calculate their total number accurately; 2) intention reasoning, focusing reasoning the next route and turn of the specific object; 3) attribute reasoning, focusing on reasoning the specific materials and functions of the given objects.
Fig. 15 shows the distribution of tasks across various domains.
2.3 MME-REAL.WRRLD-OR
The traditional general VQA approach (Liu et al., 2023c) uses a translation engine to extend QA pairs from English to Chinese. However, it may face visual-textual misalignment problems (Tang et al., 2024), failing to address complexities related to nuanced meaning, contextual distortion, lan- guage bias, and question-type diversity. Additionally, asking questions in Chinese about images containing only English texts is not intuitive for benchmarking Chinese VQA capabilities. By con- trast, we follow the steps below to construct high-quality Chinese benchmark:
Selection. For video monitoring, autonomous driving, and remote sensing, many images do not contain English information. Therefore, we select subset of the aforementioned question pairs, double-checking to ensure they do not contain any English information.
Translation. Translate the questions and answers by four professional researchers, all of whom are familiar with both English and Chinese.
Collection. For diagrams and tables, since the original images often contain English in- formation (e.g., legends/captions). we collect additional 300 tables and 301 diagrams from the Internet, where the contents are in Chinese. This data is further annotated by one vol- unteer, resulting in 301 x4 QA pairs, where the task type is the same as diagram and table in MME- Real Similarly, for OCR in the wild, we also collect additional 939 images for all the subtasks.
In total, Reae-Relloodddd has 1, 889 additional images and total 917 QA pairs, which is a smaller version of Weal-eel but it retains similar task types, image quality, and task difficulty. The examples can be seen in Fig. 12.
5

---

Published as a conference paper at ICLR 2025
2.4 QUALITY CONTROL AND ANALYSIS
During the annotation process, we impose the following requirements on annotators: 1. We ensure that all questions can be answered based on the image (except for specially constructed questions where the correct option is "E"), meaning that humans can always find the an- swers within the image. This approach prevents foreing annotators to provide answers based on low-quality images or images containing vague information. 2. The area of the object be- ing questioned in each image must not exceed 1/10 of the total image area. This ensures that the object not overly prominent, pre- venting humans from easily identifying the an- swer at first glance. 3. Each annotation is cross-checked by at least two professional mul- timodal researchers to ensure accuracy and pre- vent annotation errors caused by human bias.
Table 2: Comparison of benchmarks. MME- Real is the largest fully human-annotater dataset, featuring the highest average resolution and the most challenging tasks.
Benchmark
QA--Pair
Fully
Aunstation
Reveutinnn
LLAVA-13-7R
Performance
5000
123 1224
2011
765
1596-961
MMSS::
1500
512x3355
50.3
ScienceOAA
32719
$4D
MM Ver
218
12000-075
31.1
Boad-Berch
19242
1024-031
16.1
2330
1124-546
36.8
32325
2365x377
49.5
S19:4446
908
997:903
ValT-Becch
BLINK
1907
620- 1924
37.1
CV-Barch
2618
1024-766
5734
955-776
58.2
MME
2574
1161 1101-880
7000
MMBeanch
3217
512:-270
64.3
The comparison of benchmarks is shown Tab. 2. The maximal resolution of MMF-Rea/World 42, 177, 408 pixels, with dimensions of 5304x 7952. The average resolution is 3, 007, 695 pixels, cquivalent to an image size of approximately 2000 1500. This resolution is significantly higher than that of existing benchmarks. For instance, the highest benchmark, MME, has an average res- olution of 975, 240 pixels, corresponding to an image size of about 1161 x840. The exceptional image quality and our strict, fully human annotation process make tasks the most challenging among all benchmarks. This is evidenced by the baseline model LLAVA-1,5-71 achieving an ac- curacy of just 24.9%, significantly lower than on other benchmarks. Although some benchmarks may approach our level of difficulty, this is primarily due to the inherent complexity of their tasks. For instance, Math Vista focuses on pure mathematical problems, and MM-Ve Vet involves multi-step ceasoning-boote of which are naturally challenging and result in lower baseline performance. How- ever, the majority of our tasks are centered on real-world perception problems. This means that, current MLLMs still struggle to effectively address human-level perceptual challenges.
EXPERIMENTS
We evaluate a total of 24 open-source MLLMs, including Qwen-VL-Chie VL-Chat (Bai et al., 2023a),
LLaVA, LLaVA-Next (Li et al., 2024a), TextMonkey (Liu et al., 2024b), mPLUG-DocOwl 1.5 (Hu et al., 2024a), ShareGPT4V (Chen al., 2023b), MiniGPT-v2 (Chen et al., 2023a), Monkey (Li et al., 2023f), OtterHD (Li et al., 2023a), Cambrian-1 (Tong et al., 2024), Mini-Gemini-HD (Li et al., 2024d), MiniCPM-V 2.5 (Hu et al., 2024b), DeepSeek- VL (Lu et al., 2024a), YI-VL-34B', (Zhang et al., 2023), InternVL-
SliME (Zhang et al., 2024), VLM2 IntemL.MXXCompore22 Chat V1-5, Intern (Chen et al., 2023c), and Qwen2-VLL VL', as well as 4 close-source MLLMs, including, GPT-4o", GPT-40-mini, Gemini 1.5 pro (Team et al., 2023), and Claude 3.5 Sonnet'.
3.1 RESULTS ON MME-REALWORLD
3. PERCEPTION
Tab. 3 presents the perception capabilities of different models across domains. Overall, Qwen2-VL. demonstrates the strongest perception abilities, outperforming other closed-source models. How- ever, the performance varies across different tasks, with some key observations as follows:

---

Published as a conference paper at ICLR 2025
Table Experimental results on the perception tasks. Models are ranked according their aver- age performance. Rows corresponding to proprictary models are highlighted in gray for distinction. "OCR". "RS", "DT", "MO", and "AD" each indicate specific task domain: Optical Character Recognition in the Wild, Remote Sensing, Diagram and Table, Monitoring, and Autonomous Driv- ing, respectively. "Avg" and "Avg-C" indicate the weighted average accuracy and the unweighted average accuracy across subtasks in each domain.
Method
LLM
Perception
Task Split
OCR
RS
DT
MO
AD
Avg
Avg-C
# QA pairs
5740
3738
5433
2196
3660
20767
20767
Qwen2-VL
Qwen2-7B
81,38
44.81
70.18
37.30
34,62
58.96
53.66
Intern VL-2
InternLM2.5-7B-Chhl
73.92
39.35
62.80
53.19
35.46
55.82
52.94
Claude 3.5 Sonnet
72.47
25.74
67.44
32.19
40.77
52.90
47.72
DDemLLM-XCCoposr22..
InterLLM2-7B
69.25
36.12
63.92
39.48
33.63
52.47
48.48
IntemVL--ChatVill Vh.t
M
71.51
33.55
55.83
51.16
31.42
51.36
48.69
Mini-Gemini-34B-ID
Nous- Hermes-2-Yi--44B
69.55
40.40
44.36
39.61
32.70
48.05
45.32
MiniCPM-V2. 2.5
Llama3-8B
66.79
27.69
52.81
38.70
34.15
47.37
44.03
Cambrian-1-34B
Nous-Hermes-2-1 Hermess2--Y--34B
66.45
38.63
40.44
45.98
33.61
46.68
45.00 45.02
GPT-4o
28.92
46.68
33,93
22.43
46.43
BogVLM2-llama3-Cme
Llama3-8B
69.97
28.76
47.51
33.74
30.22
45.84
42,04
Cambrian-1-88
Jama3-8B-Instruu
58.68
40.05
32.73
47.68
38.52
43.82
43.53
SliME-8B
Llama3-8B
53.45
42.27
29.34
40.62
33.66
40.29
39.87
Gemini-1.55 1.Spro
67.62
13.99
39.90
31.11
26.64
39.63
35.85
GPT-4o-mini
44.23
26.50
24.18
37.12
32.82
Monkey
Qwen-7B
32.51
28.01
29.67
36.30
33.96
mPLUG-DDeOOl 1.5
Llama-7B
51.15
23.71
29.34
24.97
28.28
33.71
31.49
DeepSeek-Vi VL
PerpSeek-LLLM-b--leeee
49.55
25.49
23.38
26.97
33.39
33.14
31.76
SliME-13B
Vicuna-130
50.58
25.82
20.93
24.73
27.16
31.50
29.84
Mini-Gemini-7B-HD
Vicuna-7B-vl..
42,02
31.30
22.31
34.15
24.81
31.07
30.92
YI-VL-34B
Yi-34B-Chai
34.85
28.31
30.97
31.14
LLaVA-Next
Llama3-8B
47.94
25.42
26.63
19.46
18.66
30.14
27.62
LLaVA-Next
72B
37.07
29.13
27.68
29.37
17.98
29.01
28.25
LLaVA1.5-13B
Vicuna- 13B
44.10
23.27
20.17
20.45
26.12
28.42
26.82
ShareGPT4V-13B
Vicuna- 13B
44.55
23.06
20.17
19.26
26.12
28.38
26.63
MiniGPT-v2
Llama 2-7B-Chat
39.02
23.33
20.41
19.26
25.96
26.94
25.60
ShareGPT4V.7B
Vicuna-7B
39.39
22.10
20.08
19.13
26.04
26.73
25.35
LLaVAI.5-7B
Vicuna-7B
38.69
22.12
20.08
19.13
26.04
26.54
25.21
Qwen-VL-Chat
Qwen-7B
32.37
15.14
15.59
22.13
15.08
20.75
20.06
TextMonkey
Qwen-7B
37.30
11.69
5.93
16.14
14.26
18.18
17.06
GPT-4o performs well in real-world OCR tasks, achieving 77% accuracy, which is sec- ond only Qwen2-VL. However, its performance significantly drops in more challenging tasks, lagging behind other top-ranked models. This trend is also observed in other closed- source models, such as Gemini-1.5-4 1-5-Po and GPT-4o-mini, which perform well in OCR tasks but struggle significantly in other real-world tasks. There are three possible reasons: Close- source models often have limitations on the maximum image size and resolution when uploading local images. For example, Claude 3.5 Sonnet has a maximum resolution limit SK and a maximum image quality of 5MB, while GPT-4o and Gemini-pro allow up 20MB. This restricts the input of some high-quality images, the images for upload. 2) Close-source models tend to be more conservative. We observe that the proportion of responses, where closed-source models output "E" indicating that the object in question is not present in the image, is high. This suggests that these models may adopt conservative response strategy avoid hallucinations or provide safer answers. 3) Closed-source models sometimes refuse to answer certain questions. Due to different input/output filtering strategies, some samples are considered involve privacy or harmful content and are therefore not answered.
have to compress
Models allowing higher resolution input, such as Mini-Gemini-HD and SliME, demon- strate significant advantage over models directly using vision encoders like CLIP, such as ShareGPTTV and 1... At the same model size, these models consistently improve across different subtasks. This highlights the critical importance of high-resolution image processing for addressing complex real-world tasks.
There are also notable trends across different domains. Remote sensing tasks involve processing extremely large images, demanding a deeper comprehension of image details. Models that focus high-resolution input, such Cambrian-1, Mini-Geminii-HD. and

---

Published as a conference paper at ICLR 2025
Table Experimental results on the reasoning tasks. Models are ranked according to their aver- age performance. Rows corresponding to proprictary models are highlighted in gray for distinction. "OCR". "RS", "DT", "MO", and "AD" each indicate specific task domain: Optical Character Recognition in the Wild, Remote Sensing, Diagram and Table, Monitoring, and Autonomous Driv- ing, respectively. "Avg" and "Avg-C" indicate the weighted average accuracy and the unweighted average accuracy across subtasks in each domain.
Method
L.M
Reasoning
Task Split QA pairs
OCR
DT
MO
AD
Avg
Avg-C
500
500
498
1334
2832
2832
Claude 3.5 Sonnet
61.90
61.20
41.79
31.92
44.12
49.20
Qwen2-VL
Qwen2-7B
63.40
48.60
33.13
31.47
40.39
44.15
Intern VL-2
IntemLLM2.5-7--Chm
57.40
39.00
43.57
29.84
38.
42.45
GPT-4o
61.40
44,80
36.51
26.41
37.61
42.28
DogVLMM-llamua-CCe
Llama3-8B
54.00
32.80
41.16
31.18
37.25
39.79
VL-ert -h-hatt--S
56.80
35.40
37.35
28.94
36.48
39.62
Cambrian-I-8B
Llama3-8B-Instrn
53.20
27.40
42.37
30.73
36.16
38.43
SliEE-BB
Llama3-81
53.20
29.40
36.14
31.55
35.80
37.57
MiniCPM-V22.5
Llama3-8B
44.00
31.80
36.95
31.03
34.50
35.95
SliME-13B
Vicuna-13B
41.00
39.00
33.13
30.80
34.46
35.98
Incmm.M-XComppoer2.55
IntemL.M2-7B
53.40
41.00
17.67
29.99
33,90
35.52
GPT-4o-mini
47.00
39.80
25.81
26.79
32.48
34.85
YI-VL-34B
Yi-34B-Chat
42.40
26.00
31.33
31.55
32.45
32.82
LLaVA-Next
Llama3-8B
55.20
23.40
21.08
30.73
32.06
32.60
Seii-Gemini-4BBHHD
Nou-HHemes-2-Yi-3443
59.20
39.20
20.48
22.84
31.73
35.43
Gemini-1.5 1.5-pro
$2.70
33.20
28.33
19.20
29.19
33.36
Monkey
Qwen-7B
27.20
20.80
27.31
33.04
28.84
27.09
DeepSeek-V VL
DeepSeek-LLMM-7bbbe
45.20
23.80
16.67
27.31
27.98
28.25
LLaVA-Next
Qwen-7218
17.20
34.20
27.31
29.69
27.86
27.10
Cambrian-1-34 1-34B
Nous-Hermes:22Yi-B33
55.00
36.00
19.48
16.07
27.06
31.64
mPLUG-DecOwl 1.5
Llama-7B
42.60
19.80
20.48
26.04
26.88
27.23
Mini-Gemini-7B-111
Vicuna-7B-v1.5
35.40
24,60
25.90
23,29
26.12
27.30
ILaVAI.5-13B
Vicuna- 13B
30.20
20.80
27.51
24.78
25.51
25.82
Shar:GPT4V-13B
Vicuna-135
26.00
20.80
27.31
24.55
24.63
24.67
LLaVAL,5-7B
Vicuna-78 7B
26.00
20.60
25.90
24.18
24.17
24.17
ShareGPT4V-7B
Vicuna-7B
24.15
20.60
26.10
24.18
23.88
23.76
MiniGPT-v2
Llama 2-7R-Chat
30.00
20.40
16.87
23.66
23.01
22.73
Ewen-VL-Chan
Qwen-7B
28.60
13.60
16.47
24.63
21.95
20.83
TextMonkey
Qwen-7B
30.40
2.20
4.42
20.01
15.96
14.26
SliME, outperform other models in these tasks. Additionally, models trained on large amounts chart data exhibit improved perception capabilities for complex charts. For instance, SliME and VLa have limited and relatively simple chart data in their train- ing sets, resulting in inferior performance in this category compared to more recent models.
3.1.2 REASONING
Experimental results on the reasoning tasks are shown in Tab. 4. terms of reasoning ability, Claude 3.5 Sonnet distinguishes itself as the top performer across most domains, particularly outpacing the second-place Qwen2-VI. by 12.6% in chart-related tasks. The closed-source model GPT-40 also performs well, trailing slightly behind the third-place InternVI.-2 but even outperforming it in several domains. Most open-source models perform poorly, with traditional baseline methods such VA and QWen-VL-Chat yielding results close to random guessing. Furthermore, reasoning tasks are more challenging than perception tasks. Even the top-ranked model fails to achieve an average accuracy above 45%, with blass-based accuracy not exceeding 50%. This indicates that current models still have significant gap to bridge to reach human-level reasoning capabilities.
3.2 RESULTS ON MME-REALWORLD--
Results of perception tasks and reasoning tasks are presented Tab. 5 and Tab. 12, respectively. The models show different performances compared to the Weallld English version.
1) Qwen2- VL and Intern VL-2 significantly outperform existing models in both perception and rea- soning tasks in the Chinese version. The performance of these two models even surpasses their
8

---

Published as a conference paper at ICLR 2025
Experimental results on the perception tasks of MME-RealWorld-CN, Models are ranked according to their average performance. Rows corresponding to proprietary models are high- lighted in gray for distinction. "OCR", "RS", "DT", "MO", and "AD" each indicate a specific task domain: Optical Character Recognition in the Wild, Remote Sensing, Diagram and Table, Moni- toring, and Autonomous Driving, respectively. "Avg" and "Avg-C" indicate the weighted average accuracy and the unweighted average accuracy across subtasks in each domain.
Table 5:
Method
L.M
Perception
Task Split
OCR
RS
DT
MO
AD
Avg
Avg-C
# QA pairs
1908
300
602
500
700
4010
4010
Inter IntemVL-22.3
75.79
49.33
80.40
29.20
35.86
61.72
54.12
Qwen2-VL
Qwen-7B
70.28
38.33
89.20
29.40
36.86
59.80
52.81
IntemVT.-2
69.92
41.33
71.63
39.3
34.14
57.97
51.26
Intern VL.-Cht Th--Cht-VV-5
60.59
32.00
60.12
32.40
32.14
49.90
43.45
Claude 3.5 Sonnet
54.44
32.67
74.09
25.00
32.43
48.25
43.73
SliME-8B
Llama3-8B
53.93
41.33
58.25
29.20
31.29
46.60
42.80
GPT-4o
$5.90
23.67
54.86
25.20
21.14
43,44
36.15
Y1-VL-34B
Yi-34B-Chat
51.41
34.33
49.52
25.20
27.71
42.45
37 63
SliME-13B
Vicuna-13B
50.63
17.33
48.49
17.80
33.23
40.69
33.50
Camhrian-1-34B
Houu-
48.11
33.79
44.34
26.43
40.13
36.05
CogVLMM-llama--Che
Llama3-8B
46.12
22.00
39.48
24.80
34.14
38.57
33.31
Nous- H-rmeese---2
41.82
38.28
40.60
27.80
34.29
38.31
36.56
I.LaVA-Nes
Llama3-8B
40.62
31.67
37.49
35.40
27.29
36.50
34.49
Gemini-1.5-pa
48.32
12.33
39.78
25.20
17.57
36.10
28.64
Monkey
Qwen-7B
40.46
26.55
41.12
19.20
35.86
36.07
32.64
IntemL M-Coopppss2222.2555
I22ernl.M2-7B
39.26
38.33
38.88
19.40
33.57
35.66
33.89
Mini-Gemiini-B-HDD
Vicuna-7B-v1.5
39,66
17.24
39.29
16.80
28.29
33.09
28.26
Cambrian-1-88
Llama3-8B-Insttun
32.71
35.86
30.28
27.60
35.57
32.44
32.40
mPPLUG-DooOwl 1.5
LLama-7B
33.33
18.62
31.83
25.60
28.43
30.19
27.56
LLaVA-Next
Qwen-72B
32.76
23.67
28.69
34.60
23.14
30.02
28.57
MiniCPM-V 2.5
Llama3-8B
33,23
16.67
31.67
20.40
26.00
28.89
25.59
DecpSock-VL.
DecpScck-LLMM-bb-hass
27.10
25.44
26.02
21.60
35.71
27.63
27.17
TextMonkey
Qwen-7B
31.24
11.38
30.76
19.60
26.71
27.44
23.94
GPT-4o-mini
29.56
7.33
31.79
22.00
24.00
26.32
22.94
Qwen-VL-Chat
Qwen-7B
27.36
15.00
27.89
24.29
27.36
26.13
24.38
ShareGPT4V-13B
Vicuna-13B
27.94
17.59
27.57
16.80
28.14
25.75
23.61
LLaVA1.5-13B
Vicuna-13B
27.52
17.33
26.25
17.00
28.66
25.45
23.35
MiniGPT-v2
Llama 2-7B-Chat
26.78
19.31
27.05
14.40
25.18
23.39
ShareGPT4V.7B
Vicuna-7B
26.73
17.24
25.75
16.60
28.14
24.86
22.89
LLaVA1.5-7B
Vicuna-7B
26.36
16.67
25.75
16.60
28.14
24.64
22.70
performance on the English version of AME-RealWorldd, indicating that they jave been specifically optimized for Chinese data.
2) There is substantial difference in how models handle Chinese and English data, with some models performing much worse in Chinese scenarios, particularly in reasoning tasks. For instance, GPT-4o and GPT-4o-miní show performance drop of nearly 10%. However, some models seem to excel in Chinese-related tasks. Notably, models based on L. 3. 3-ma generally achieve strong results in both Chinese perception and reasoning tasks, such SliME and Cog VLMVLM2. This suggests that L 3-8B may be an effective LLM backbone for Chinese tasks.
3.3 FINE-GRAINED ANALYSIS AND FINDINGS
Existing Models Still Lacking in Image Detail Perception. Fig. 3 displays the frequency with which various models choose "E" as their answer. We compare 4 close-source models with the top-level open-source model, Intern During our annotation process, the frequency of "E" answers does not exceed 5% of the overall data, meaning represents only a small portion of the total QA pairs. However, nearly all models show much higher frequency of "E" outputs than the actual number of "E" instances present our benchmark. This indicates that most models" visual perception modules fail to identify the objects in the images corresponding to our questions.
Computation Efficiency. There is significant disparity in computation efficiency among different models when processing high-resolun righrresolution images. For example, using models similar to LLMs (e.g., Vicuna- 13B), the computational requirements for handling images exceeding 1024x (024x1024 resolution follows: VA1.5 requires 16.37 TFLOPS, SliME requires 40.82 TFLOPS, while LLaVA-

---

Published as a conference paper at ICLR 2025
6000
Model
5000
Total QA
Claude 3.5 Sonnet
4000
# Samples
TiPT-4o-mini
3000
Gemini- 1.5
Intern
QA pairs with answer "E"
1000
OCR (P)
RS
DT (P)
MO
AD
OCR (R)
DT (R)
MO (R)
AD (R)
Figure 3: Frequency of outputting answer "E" for different models across various domains. The notation in parentheses indicates the task type: P for perception and R for reasoning. The total QA pairs and those with answer "E" are also presented for comparison.
Next and Wini-Gemini-HD require 78.37 and 87.59 TFLOPS, respectively. LLa VA-Nxxt and SliME employ dynamic chunking and encoding of images, while Mini-Gemini-HD uses higher-resolution vision encoder and significantly increases the number of vision tokens, resulting in computation cost approximately 5 times that LLaVA1.5. Additionally, existing methods have inherent lim- itations in handling high-resolution images. For example, Mini-Gemini-HH resizes images larger than 672 this size, causing a loss of more details. Moreover, we observe interesting phe- nomena in closed-source models regarding image resolution. For instance, GPT-4o-min uses over 10, 000 tokens for some large images, which is about times more than other closed-source mod- els, although its performance does not significantly surpass other models. Overall, we currently lack methods that can efficiently handle higher resolution images with lower computational overhead.
Appendix Overview: Supplementary Analyses and Detailed Results. Due to the space con- straints of this paper, we are unable to include some detailed discussions several important anal- yses, which summarize here. First, explore simple tricks mitigate the challenges posed by high-resolution images, specifically by segmenting the images into smaller sub-images and querying each individually. The responses are then aggregated using majority vote. While this method seems intuitive, leads to a significant decline in performance, mainly due to the loss positional context and the risk of splitting fine-grained objects across sub-images. Additionally, we conduct an analysis of the distribution of incorrect choices across various models, revealing distinct response patterns. Larger models show tendency to select safer options, while smaller models often lean towards the first choice. Finally, we observe that the instruction instruction-follooing capabilities of some cur- rent open-source models are still insufficient, indicating substantial room for improvement. These analyses, along with the performance results for each subtask, are presented in the Appendix Sec. D Sec. G due to space constraints the main text.
4 CONCLUSION
In this paper, we have introduced dress key limitations in existing evaluations of MLLMs, such as data scale, annotation quality, and task difficulty. As the largest purely aunan-andotated dataset with the highest resolution to date, MME-RealWorld benefits from the participation of 32 annotators, ensuring high data quality and minimal individual bias. Most QA pairs focus on real-world scenarios, such as autonomous driv- ing and video surveillance, which have significant applicability. Furthermore, Real Reallorld-CNy a benchmark specifically focused on Chinese scenarios, ensuring that all images and questions relevant to Chinese contexts. Our evaluation of wide range of models reveals signifi- cant performance gaps, highlighting the current models' shortcomings complex image perception and underscoring, and the need for further advancements.
comprehensive benchmark designed to ad-
propose MME-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by National Key R&D Program of China (20222D0117901) and National Natural Science Foundation of China (Grant No. 62236010, 62373355)
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In CVPR, 2017.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01 009000900.0 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint Xr3:: /22.300....2..... 2023a.
Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, and Jingren Zhou Touchstone: Evaluating vision-language models by language models. arXiv preprint arXiv:2308. 210223888.6800 2023b.
Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gard- ner, Rohan Taori, and Ludwig Schimdt. Visit- Visit-bench: A benchmark for vision- language instruction following inspired by real-world use. arXiv preprint arXiv:2308.06559 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020.
Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09477 2023a.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12799 2023b.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language 2024.
models? arXiv preprint 2rXiv:2403.20333
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qing- long Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. In- ternvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint 2w..:::231221222555 2023c.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https//vvicuna Imsys. org (accessed 14 April 2023), 2023.
11

---

Published as a conference paper at ICLR 2025
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. JMLR, 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. NeurIPS, 2024.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multi- modal language model. arXiv preprint Stt.:23303.03775. 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 1..04 13394, 2023a.
Chaoyou Fu, Renrui Zhang, Haojia Lin, Zihan Wang, Timin Gao, Yongdong Luo, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, et al. A challenger to gpt-4v? early explorations of gemini in visual expertise. arXiv preprint wiXi::2312.124360 2023b.
Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, et al. Vita: Towards open-source interactive omni multimodal Ilm. arXiv preprint 2rXi::2408.052/1 2024a.
Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, al. Mme-survey: A comprehensive survey on evaluation of multimodal llms. arxiv preprint arXiv:241 1,1529,,,, 2024b.
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long. Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.0199 2025.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng. Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404. 12399, 2024c.
Haoxiang Gao, Yaqian Li, Kaiwen Long, Ming Yang. and Yiqing Shen. A survey for foundation models in autonomous driving. arXiv preprint arXiv:2402.00 105, 2024.
Chunjiang Ge, Sijie Cheng, Ziming Wang, Jiale Yuan, Yuan Gao, Jun Song, Shiji Song, Gao Huang, and Bo Zheng. Convllava: Hierarchical backbones as visual encoder for large multimodal models. arXiv preprint arXiv:2405.1 15738, 2024.
Xiaotian Han, Quanzeng You, Yongfei Liu, Wentao Chen, Huangjie Zheng, Khalil Mrini, Xudong Lin, Yiqi Wang, Bohan Zhai, Jianbo Yuan, Heng Wang, and Hongxia Yang. Infimm-eval: Com- plex open-ended reasoning evaluation for multi-modal large language models, 2023.
Dongyang Hou, Zelang Miao, Huaqiao Xing, and Hao Wu. V-rsir: An open access web-based image annotation tool for remote sensing image retrieval. IEEE Access, 2019.
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. hoplug-docow 1.5: Unified structure learning for ocr-free document understand- ing. arXiv preprint arXiv:2403. 128.. 2024a.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06:: 063955 2024b.
Xinyu Jia, Chuang Zhu, Minzhen Li, Wengi Tang, and Wenli Zhou. Llvip: A visible-infrared paired dataset for low- vision. In ICCV, 2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310006825, 2023.
12

---

Published as a conference paper at ICLR 2025
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. NeurIPS, 2024.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03722 2023a.
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal ca- pabilities in the wild, May 2024a. URL nttps://llave vl. rr.ggihub... io/blog/
1-----------aaa-eeeeeeee- www.-eeeoooeee-eeeeee 1lms/.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed- bench-2: Benchmarking multimodal large language models. arXiv preprint arXiv:2311.17092, 2023b.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Iims with generative comprehension. arXiv preprint arXiv:2307.16125, 2023c.
Bohao Li, Yuying Ge. Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. feed-bench-2-plee Benchmarking multimodal large language models with text-rich visual comprchension. arXiv preprint arXiv:2404. 16790, 2024b.
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, and Furu Wei. Imagine while reasoning in space: Multimodal visualization-of-thllee arXiv preprint 2011v:2501.07542, 2025.
Hongyang Li, Yang Li, Huijie Wang, Jia Zeng. Pinlong Cai, Huilin Xu, Dahua Lin, Junchi Yan, Feng Xu, Lu Xiong, et al. Open-sourced data ecosystem in autonomous driving: the present and future. arXiv preprint 1rXiv:2312.03-400 2023d.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023e.
Kaican Li, Kai Chen, Haoyu Wang. Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Din Yeung, et al. Coda: A real-world road corner case dataset for object detection in autonomous driving. In ECCV, 2022.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multi- modal arxiv: A dataset for improving scientific comprehension of large vision-language models, 2024c.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 18814, 2024d.
Zhang Li, Biao Yang. Qiang Liu, Zhiyin Ma. Shuo Zhang, Jingxu Yang. Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311.0m 1.0607, 2023f.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, arXiv preprini urXiv:2310.03744 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304 3004508844000 2023b.
Haotian Liu, Chunyuan Li. Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae
Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024a. URL https:
//1lava-v1, git.hui. httpb//////2/01--11//
J. Liu, D. Liu, W. Yang, S. Xia, X. Zhang, and Y. Dai. A comprehensive benchmark for single image compression artifacts reduction. In arXiv, 2019.
13

---

Published as a conference paper at ICLR 2025
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023c.
Yuliang Liu, Biao Yang. Qiang Liu, Zhang Li. Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473, 2024b.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vi: towards real-world vision-language understanding. arXiv preprint arXin:2403.05525, 2024a.
Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wild- vision: Evaluating vision-language models in the wild with human preferences. arXiv preprini arXiv:2406. 11069, 2024b.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, 2021.
Brandon McKinzie, Zhe Gan, Phan-Philippp Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre-training, arXiv preprint 24X 2024.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual gen- eralization through multitask finetuning. arXiv preprint arXiv:2211.0/786 2022.
OpenAl. Gpt-4 technical report. 2023.
Enna Sachdeva, Nakul Agarwal, Suhas Chundi, Sean Roelofs, Jiachen Li, Mykel Kochenderfer, Chiho Choi, and Behzad Dariush. Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning. WACV, 2024.
Yunhang Shen, Chaoyou Fu, Shaoqi Dong, Xiong Wang, Peixian Chen, Mengdan Zhang, Haoyu Cao, Ke Li, Xiawu Zheng, Yan Zhang, al. Long-vita: Scaling large multi-modal models million tokens with leading short-context accuray. arXiv preprint arXiv:2502.05177. 2025.
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. arXiv preprint arXiv:2312. 14150, 2023.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019.
Xian Sun, Peijin Wang, Zhiyuan Yan, F. Xu, Ruiping Wang, W. Diao, Jin Chen, Jihao Li, Yingchao Feng, Tao Xu, M. Weinmann, S. Hinz, Cheng Wang, and K. Fu. Fair Im: A benchmark dataset for fine-grat imn-ngrined object recognition in high-resolution remote sensing imagery. ISPRS, 2021.
Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, et al. Mtvqa: Benchmarking multilingual text-centric visual question answering. arXiv preprint arXiv:2405.11988 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followe llama model, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud. Yonghui Wu, Japt-sae Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint ar /8805 2023.
14

---

Published as a conference paper at ICLR 2025
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal 1lms. arXiv preprint 2rX1v:2406.16680 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. LLaVA-UHD: an Imm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403. 1703, 2024.
Yibo Yan, Shen Wang, Jiahao Huo, Hang Li, Boyan Li, Jiamin Su, Xiong Gao. Yi-Fan Zhang, Tianlong Xu, Zhendong Chu, et al. Errorradar: Benchmarking complex mathematical reasoning of multimodal large language models via error detection. arXiv preprint arXhv:2410.04500 2024.
Qinhong Yang, Dongdong Chen, Zhentao Tan, Qiankun Liu, Qi Chu, Jianmin Bao, Lu Yuan, Gang Hua, and Nenghai Yu. Hq-50k: A large-scale, high-quality dataset for image restoration. arXiv preprint 003.00930000 2023a.
Zhenjie Yang, Xiaosong Jia, Hongyang Li, and Junchi Yan. LIm4drive: A survey of large language models for autonomous driving. arXiv e-prints, pp. arXiv-2311, 2023b.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint S.Ii::200..41778 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun. Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang. Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi, 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In ICMI,, 2024.
Kaihao Zhang, Dongxu Li, Wenhan Luo, Wenqi Ren, Björn Stenger, Wei Liu, Hongdong Li, and Ming-Hsuan Yang. Benchmarking ulrra-bigh-definition image super-resolution. In ICCV, 2021.
Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuan- grui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. -ntemonn-coompoer: A vision- language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309. 12 2023.
Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond Ilava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406.088 084887 2024.
Yi-Fan Zhang, Tao Yu. Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng. Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, Xue Wang, Yibo Hu, Bin Wen, Fan Yang. Zhang Zhang, Tingting Gao, Di Zhang, Liang Wang. Rong Jin, and Tieniu Tan. Mm-rlhf. The next step forward in multimodal Ilm alignment, 2025. URL https: / //axxvvv..... org/ abs 22002.5222 10391.
Yunsong Zhou, Linyan Huang, Qingwen Bu, Jia Zeng, Tianyu Li, Hang Qiu, Hongzi Zhu, Minyi Guo, Yu Qiao, and Hongyang Li. Embodied understanding of driving scenarios. arXiv preprint arXiv:2403.0444 045993, 2024.
15

---

Published as a conference paper at ICLR 2025
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing asion-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Heng Fan, Qinghua Hu, and Haibin Ling, De- tection and tracking meet drones challenge. T-PAMI, 2021.
16

---

Published as a conference paper at ICLR 2025
MMElRealWorld
Appendix
CONTENTS
Introduction
MME-RealWorld
2.1
Instruction and Criterion
2.2
Data Collection and Annotation
2.3
MME-RealWorld-CN
2.4
Quality Control and Analysis
3
Experiments
3.1
Results on MME-RealWorld
3.1.1
Perception
3.1.2
Reasoning....
3.2
Results HIME-ReaWorld-CN
3.3
Fine-grained Analysis and Findings
Conclusion
10
Related Work
19
Data Collection and Task Split
20
B.1 OCR in the Wild
20
B.2 Diagram and Table
................................
22
B.3
Remote sensing
22
B.4 Autonomous Driving
................................
23
B.5
Monitoring
26
Discussion on Data Diversity, Scalability, and Limitations
28
C.I Advancing Dataset
Diversity and Scalability: Challenges and Model-Assisted
Strategies
28
C.2 Why We Select Domains Such as Remote Sensing and Autonomous Driving Instead of Others Like Embodied AI or Natural Images in COCO
29
C.3 Existing Limitations and Plans for Extension
29
C,4 Exploring Model-Assisted Approaches to Enhance Dataset Scalability
30
Experimental Results on All Task Splits
31
Experimental Results All Task Splits of MMME-RealWorId-CN
36
17

---

Published as a conference paper at ICLR 2025
F
MME-RealWorld-lte
36
Other Analysis
37
18

---

Published as a conference paper at ICLR 2025
A RELATED WORK
Multimodal Benchmark. With the development of MLLMs, a number of benchmarks have been built. For instance, MME (Fu et al., 2023a) constructs comprehensive evaluation benchmark that includes a total of 14 perception and cogmition tasks. All QA pairs in MME are manually designed to avoid data leakage, and the binary choice format makes it easy to quantify. MMBench (Liu et al., 2023c) contains over 3, 000 multiple-choice questions covering 20 different ability dimensions, such as object localization and social reasoning. It introduces GPT-4-based choice matching to address the MLLM's lack of instruction-follooinng capability and a novel circular evaluation strategy to im- prove the evaluation robustness. Bred-Bench (Li et al., 2023c) is similar to MME and MMBench but consists of 19, 000 multiple-choice questions. The larger sample size allows it to cover more abil- ity aspects and achieve more robust results. SEED-Bench-2 (Li et al., 2023b) expands the dataset size to 24, 371 QA pairs, encompassing 27 evaluation dimensions and further supporting the eval- uation of image generation. MMT-Bench (Ying et al., 2024) scales up the dataset even further, including 31, 325 QA pairs from various scenarios such as autonomous driving and embodied AI. It encompasses evaluations of model capabilities such as visual recognition, localization, reasoning. and planning. Additionally, other benchmarks focus on real-world usage scenarios (Fu et al., 2024c; Lu et al., 2024b; Bitton et al., 2023) and reasoning capabilities (Yu al., 2024; Bai al., 2023b; Han et al., 2023). There are also some benchmarks that focus on safety or MLLM reward model evaluation (Zhang et al., 2025) and mathematical problems ( (Yan et al., 2024). However, there are widespread issues, such as data scale, annotation quality, and task difficulty, in these benchmarks, making it hard to assess the challenges that MLLMs face in the real world.
This field has undergone significant evolution (Yin et al., 2023; Fu et al., 2023b), initially rooted in BERT-based language decoders and later incorporating advancements in LLMs. MLLMS exhibit enhanced capabilities and performance, particularly through end-to-end training techniques, by leveraging advanced LLMs such as GPTss (OpenAL., 2023; Brown et al., 2020), LLaMA (Touvron al., 2023a;b), Alpaca (Taori al., 2023), PaLM (Chowdhery et al., 2023; Anil al., 2023), BLOOM (Muennighoff et al., 2022), Mistral (Jiang et al., 2023), and Vicuna (Chiang et al., 2023). Recent model developments, including Flamingo (Awadalla et al., 2023). PaLI (Laurençon et al., 2024), PaLM-E (Driess et al., 2023), BLIP-2 (Li et al.. 2023c). InstructBLIP (Dai et al., 2024), Otter (Li et al., 2023a), MiniGPT-4 (Zhu et al., 2023), mPLUG-Owl (Ye et al., 2023), LLaVA (Liu et al., 2023b), Qwen- VL (Bai et al., 2023a),, and VITA series Fu et al. (2025; 2024a); Shen et al. (2025) bring unique perspectives to challenges such as scaling pre-training, enhancing instruction- following capabilities, and overcoming alignment issues. However, the performance of these models in the face of real scenarios has often not been revealed.
MLLMs.
High-resolution MILMss Empirical studies have shown that employing higher resolution is an effective solution for many tasks (Bai et al., 2023a; Liu et al., 2023a; Li et al., 2023f; McKinzie et al., 2024). Approaches like LLaVA-Next (Liu et al., 2024a) segment high-resolution images into multiple patches, encoding each one independently before concatenating all local patch tokens with the original global image tokens, albeit at an escalated computational cost. Other models, such as Monkey (Li et al., 2023r) and LLaVA- (Xu et al., 2024), also split images into patches but sub- sequently compress them to avoid redundant tokens. Mini-Genimi (Li al., 2024d) comprises twin encoders, one for high-resolution images and the other for low-resolution visual embedding. They work in an attention mechanism, where the low-resolution encoder generates visual queries, and the high-resolution counterpart provides candidate keys and values for reference. Conv-LlaVA (Ge et al., 2024) employs Cen instead of ViT as the vision encoder. Cambrian (Tong et al., 2024) uses a set of learnable latent queries that interact with multiple vision features via cross-attention layers. Additionally, SliME (Zhang et al., 2024) stresses the importance of global features, com- pressing the local image patches twice but preserving all the global context. Although many of these models focus on improving resolution, none have been tested in a rigorous high-resolution benchmark, often providing only intuitive examples that lack informativeness and convincing sults. Our proposed benchmark offers a rigorous evaluation to test the capabilities in understanding high-resolution images.
19

---

Published as a conference paper at ICLR 2025
B DATA COLLECTION AND TASK SPLIT
B. OCR IN THE WILD
Data Characteristics.
The data is from real-world street scenes and high-resolution images of product advertisements. Text is dense difficult detect and requires careful observation to be identified.
B.1.1 DATA SOURCES AND ANNOTATION PROCESS
Data Sources. We manually select images with complex scenes and recognizable text information from existing high-resolution datasets for our test images. The open-source datasets used include DIV2K and Flickr2K (Agustsson & Timofte, 2017), which offer paired high-resolution RGB images and their corresponding downscaled low-resolution RGB images by factor of two. In our approach, we exclusively utilize high-resolution images, selecting and preserving images with complex scenes and contexts. Additionally, images with resolutions of at least 3K, most ranging from 4K to 6K. This dataset provides abundant materials for testing and evaluating performance on 4K/SK display devices, featuring diverse and complex low-level signal distributions and backgrounds. We also incorporate two large-scale Ultra- High-Definition datasets, UHD4K and UHD&K (Zhang al.. 2021), which collectively contain 23, 000 images. These datasets cater to various low-level image enhancement tasks, including image super-resolution (SR), image deraining (Derain), low-light image enhancement (LLIE), and image reflection removal (IRR). Finally, use HQ-50K (Yang et al., 2023a), lcage-scale, high-quality image restoration dataset containing 50, 000 high-quality images. HQ-50K stands out for its large scale, high resolution, varying compression rates, rich texture details, and semantic diversity.
include the LIU4K (Liu et al., 2019) dataset, which contains 2, 000
Annotation. 20 volunteers annotate the question and answer pairs. 3 experts are tasked with check- ing and correcting the annotations ensure quality.
B.1.2 EVALUATION DIMENSIONS AND BENCHMARK STATISTICS
The evaluation of models in real-world complex scenes involves their ability to recognize and un- derstand text, enabling us to ascertain their capacity to comprehend and process textual information within visual content, thereby enhancing the overall practicality and reliability of intelligent systems. Specifically, Optical Character Recognition (OCR) in complex contexts comprises five perception tasks and two reasoning tasks. For perception tasks,
Contact information and addresses (Fig. 4(a)). Recognizing telephone numbers, names of countries/citiessstreedt,, and buildings (469 images and 577 QA pairs).
Product and Advertisement Perception (Fig. 4(b).. Identifying product names/prices or adver- tisements of shops brands (803 images and 1, 588 QA pairs).
Identity Information Perception (Fig. 4(c)), Recognizing license numbers or ID cards of cars/humans (852 QA pairs).
Other kind of Small Text on Signals or Indicators Perception (Fig, 4(d)). Recognizing small text indicators, signals, and similar objects (626 images and 1, 198 QA pairs).
Book, Map and Poster Perception (Fig. 4()).. Recognizing dialogues/information on posters and specific locations involving country/region maps (785 images and 555 QA pairs).
Additionally, our two reasoning tasks include:
1. Scene Recognition (Fig. 5(a)). Understanding the meaning of scenes in images, such as predict- ing the outcome of a game based on the scoreboard or what might happen in the future based on the scene, inferring the time by looking at clock, calculating object prices (250 images and 250 QA pairs).
2. Characters Understanding (Fig. 5(b)). Understanding the pertinent characteristics of characters in a poster or comic, including their relationships, emotions, intentions, or quantities (250 images and 250 QA pairs).
20

---

Published as a conference paper at ICLR 2025
Morea
DRAAGANTTAVGNNNN SNTEE
(a) Phone and Address Perception.
(b) Product and Advertisement.
Contios
Prantee whet to
(7) Deman
Teringg Aer
(c) Human/Car License.
Other kind of small Text Signals Indicators.
SOTTUT BAT THE
DON' WAIT FOR WETLE FIND
OWNN RARBET H0M
LSEE YOU BROUGHT WILLOW LIFGOOO
WIT2I TOU
DHE NAPAL ES MARIAN UIEEFS)
CAN MANAGE.
The anagy festur contrnt
Book, Map and Poster.
Figure 4: Data Examples for Perception Tasks in OCR in the Wild
Jhat eight be she cnstins of chanarin)))
(i can
Rhs Cler
(8) Jngn:
Nayy:
2:50
Say!
fetvre
(a) Scene Recognition.
(b) Characters Understanding.
Figure 5: Data Examples for Reasoning Tasks in OCR in the Wild
Note that although we have 3, 293 unique images, some tasks use overlapping image sets, so the total number of images listed in all the tasks is not exactly 3, 293.
21

---

Published as a conference paper at ICLR 2025
B.2 DIAGRAM AND TABLE
Data Characteristics. Diagrams and tables with rich content present significant challenges for rapid localization and analysis, even for human researchers. These tasks demand a high level of perceptual capability.
B.2.1 DATA SOURCES AND ANNOTATION PROCESS
Although there are existing datasets for evaluating diagrams and tables, such as ChartQA (Masry al., 2022) and some open-source scientific chart data like Arxiv QA (Li al., 2024c), we observe that these datasets often have relatively low image resolutions and limited content richness. Con- sequently, they are relatively easy for humans to interpret quickly, which does not align with the design goals of our benchmark. To address this, source complex diagram data from the internet, such as detailed financial reports with large charts. Analyzing these large charts poses significant perceptual challenges, even for humans, and thus better aligns with the objectives of our benchmark.
Annotation. 20 volunteers are involved to generate question and answer pairs for the perception task. Additionally, expert researcher is responsible for generating reasoning annotations. To ensure high-quality annotations, three experts are assigned to review and correct the annotations.
B.2.2 EVALUATION DIMENSIONS AND BENCHMARK STATISTICS
The ability of multimodal models perceive and understand diagram and table data has long been focus of research. In our Diagram and Table domain, we have elevated the difficulty level to point where even humans find it challenging to solve easily. We have collected 2, 570 images and 5, 933 annotations, categorizing the annotations into the following four types:
1. Table Perception (Fig. 6(a)). Identifying specific elements within a table by using the given table name, horizontal axis coordinates, and related location information to determine the value of elements in specific positions (4, 018 QA pairs).
2. Diagram Perception (Fig. 6(b).. Identifying specific elements within diagram by using the provided legend or title, along with specific location information, to determine the value of elements or the intervals they belong (1, 415 QA pairs).
Additionally, our two reasoning tasks include:
1. Table Reasoning (Fig. 6(c)). This involves tasks that go beyond simple perception, such as comparing the values of two elements in specific positions within a table, filtering the table based on given conditions, determining the maximum and minimum values (174 QA pairs).
Diagram Reasoning (Fig. 6(d)). Similar to table reasoning, but reasoning with diagrams involves distinguishing specific colors in the legend and assessing the height of curves bars (326 QA pairs).
B.3 REMOTE SENSING
Data Characteristics. From real remote sensing data, some images have extremely high quality, with individual image sizes reaching up 139MB and containing rich details.
B.3.1 DATA SOURCES AND ANNOTATION PROCESS
We select high- reshuresooltoon images from public remote sensing datasets with rich information. For example, the FAIRIM dataset (Sun et al., 2021) focuses on fine-grained object recognition and detection using high-resolution (0.3 0.8m) RGB images from Gaogen (GF) satellites extracted via Google Harth. It contains 15, 000 images annotated with rotated bounding boxes across 5 main categories (ships, vehicles, airplanes, courts, and roads), further divided into 37 sub-categories. The Potsdam dataset" dataset includes 38 patches of true orthophotos (TOP) extracted from larger mosaies. VGoogle (Hou et al., 2019), VBiig (Hou et al., 2019), and VAreGIS (Hou et al., 2019) datasets, derived from Google Earth, Bing World Imagery, and AreGIS World Imagery respectively,
22

---

Published as a conference paper at ICLR 2025
(a) Table Perception.
(b) Diagram Perception.
Thes Prccy LREIE
Table Reasoning.
(d) Diagram Reasoing
Figure 6: Data Examples for Diagram and Table Tasks
each feature 38 classes with total of approximately 59, 000 images per dataset. Each class contains least 500 images, with spatial resolutions ranging from 0.07 to 38.22 meters.
Annotation. For all the questions in this subsection, 20 volunteers manually create the questions and answers, while another expert reviews the quality of the questions to ensure they meet the required standards.
B.3.2 EVALUATION DIMENSIONS AND BENCHMARK STATISTICS
Remote sensing images have wide range of applications in real-world scenarios. During the con- struction of our dataset, we observe that many tasks are challenging for humans. For example, counting the number of airplanes in Fig. 7(a) requires careful observation and counting by human annotators. Automating this process with multimodal large models would be highly valuable for re- mote sensing applications. We select total of 298 high-quality images and design three specific tasks tailored for remote sensing images:
1. Object Counting (Fig. 7())). Task involves counting specific objects such as airplanes, ships, or buildings within a given image (1, 255 QA pairs).
2. Color Recognition (Fig. 7()).. Task involves identifying and describing the colors of specific objects in the image (1, 226 QA pairs).
3.Spatial Relationship Understanding (Fig. 7(c)). Understanding both the absolute spatial rela- tionships and relative spatial relationships between objects in the images 257 QA pairs).
B.4 AUTONOMOUS DRIVING
Data Characteristics. The front-view driving datas are recorded using onboard cameras with var- ious sensor configurations. The images encompass diverse weather conditions (e.g., sunny, night, rainy, etc.), geographic locations (c.g., US, SG, CN), and complex traffic scenarios (c.g., urban, highway, etc.)
23

---

Published as a conference paper at ICLR 2025
How many
atrongft are
there she
picture?
(4)
a Object Counting.
miille area of the
plenre?
(A) Yallon:
(5) The image does
not feanure the cofor
(b) Color Recognition.
Whore pieture?
(4) the sypwe lef the pieture. upper areas picture. the lover right the pienure the orea picture. The image doer not feature the position.
(C)
(c) Spatial Relationship Understanding.
Figure 7: Data Examples for Perception Tasks in Remote Sensing
B.4.1 DATA SOURCES AND ANNOTATION PROCESS
Data Sources. We select high-quality images from large open-source driving datasets, each with distinct advantages. The Rank2Tell dataset (Sachdeva et al., 2024) ranks the importance level sur- rounding objects for driving safety. Additionally, it provides dense annotations of semantic, spatial,
24

---

Published as a conference paper at ICLR 2025
and relational attributes with bounding boxes for approximately 2, 600 frames captured intersec- tions, and it stitches images from three cameras to deliver a wide field of view (FOV). ) To enhance the reliability of autonomous driving systems, the CODA dataset (Li et al., 2022) collects 1, 500 driving scenes, each containing object-level corner cases, and labels more than 30 novel categories garbage bag, concrete block, etc.). It focuses on evaluating performance of perception systems in detecting out-of-distributiin (OOD) objects compared to common traffic elements. The nuScenes dataset (Caesar et al., 2020), one of the most popular real-world autonomous driving datasets, pro- vides abundant 3D perception annotations with semantic map and CAN bus expansion (Li et al., 2023d). Based on nuScenes (Caesar et al., 2020), DriveLM-nuSceent proximately 4, 800 key frames with driving behaviors and motions by formulating 3P reasoning (perception, prediction, planning) as a series of rich question-answer pairs in a directed graph.
(Sima et al., 2023) links ap-
Annotation. For all the questions in this subsection, a professional researcher manually generates the questions and answers based on the source datasets' labels, achieving their non-ambiguity, chal- lenge and complexity. Another expert reviews the quality of the questions to ensure they meet the required standards.
B.4.2 EVALUATION DIMENSIONS AND BENCHMARK STATISTICS
Vision-centric autonomous driving is one of the most significant applications of artificial intelli- gence. However, unresolved issues remain, including both object-level and task-level corner cases, well as safe-critical and human-like planning (Yang et al., 2023b). MILMS with general knowl- edge and the ability of driving scenarios embodied understanding (Gao et al., 2024; Zhou et al., 2024) are seen as a promising solution achieve Level autonomous driving. Specifically, we have designed three main perception tasks and three main reasoning tasks, which are further subdivided into a total of fifteen sub-tasks. It is worth noting that, as traditional detection tasks in autonomous driving have largely been addressed by modern perception models, our focus is shifting towards perception challenges involving small or distant objects, specifically those that occupy less than 1/100 of the total image area. Meanwhile, LLMs must possess extensive driving expertise and even a deep understanding of 3D spatial concepts in order to effectively address the complex reasoning challenges. For perception tasks:
I. Object Identification (Fig. 8(a)). Describing the main traffic elements in front of the ego car including their categories and corresponding quantities (1, 101 images and 1, 101 QA pairs).
2. Object Attribute Identification. Task involves identifying the attribute of a specific object according to its appearance and location (a total of 454 images and 523 QA pairs), and describing the attributes of all objects within a specific category (a total of 1, 167 images and 1, 1,315 QA pairs) in traffic scenarios. In terms of sub-tasks, the former includes the visual attribute of traffic signal (Fig. 8(c), 157 images and 201 QA pairs) and the motion attribute of pedestrian (Fig. 8(f), 152 images and 164 QA pairs) or a vehicle (145 images and 158 QA pairs), and the latter includes the motion attributes of multiple pedestrians (Fig. 8(c), 493 images and 492 QA pairs) or vehicles (Fig. 8(d), 674 images and 823 QA pairs).
3. Object Counting (Fig, 8(b)). Counting special traffic elements in the given image, such as cars, trucks, traffic signals, etc., especially some novel objects compared traditional autonomous driving tasks such as garbage bags, dogs, concrete blocks, etc. (647 images and 720 QA pairs).
Furthermore, reasoning tasks are as follows:
1. Intention Prediction. Task involves predicting the intention of a designated traffic agent in the given image (a total of 582 images and 614 QA pairs). In terms of sub-tasks, it contains fine- ane-grained behavior prediction of the ego vehicle (Fig. 9a), 304 images and 304 QA pairs) and future intention of a pedestrian (95 images and 103 QA pairs) or a vehicle (Fig. 9b), 183 images and 207 QA pairs).
2. Interaction Relation Understanding. Task involves reasoning the interaction relation between two specific traffic elements (a total of 444 images and 513 QA pairs). In terms of sub-tasks, contains the ego vehicle's 's reaction to a specific object (Fig. 9e)), which is further categorized into three categories: pedestrian (102 images and 106 QA pairs), vehicle (95 images and 101 QA pairs), and traffic signal (81 images and 105 QA pairs). Additionally, another aub-task is predicting the interactions between the aforementioned objects, excluding the ego vehicle (Fig. 9()), 166 images and 201 QA pairs).
25

---

Published as a conference paper at ICLR 2025
the the ofo Poore
indice
nnd dig
Image
Object Identification.
(b) Object Counting.
Then
(c) Motion Attribute Identification of Multiple Pedestri-(d) Motion Attribute Identification of Multiple Vehicles. ans.
fir dockadig bufte Aghua in dh Agl (bed Aaa
aepering dljict
(c) Visual Attribute Identification a Specific Traffic Signal
sonrnyy Aet byyns Aeoigh
Atreding Waath
Aazeysshaaa monckgg
valle wec.legg-dee foenoo
(l) Motion Attribute Identification of a Specific Pedestrian
Figure 8: Data Examples for Perception Tasks in Autonomous Driving
Driver Attention Understanding (Fig. 9c)). Reasoning the traffic signal that the driver should pay attention to in the given front view image, such as yellow light, speed limit sign, no parking sign, etc. (217 images and 217 QA pairs).
B.5 MONITORING
B.5.1 DATA SOURCES AND ANNOTATION PROCESS
Data Characteristics. Monitoring images are captured from different cameras (e.g., drone- equipped cameras. fixed surveillance cameras, infrared cameras), viewpoints (arbitrary and fixed viewpoints), scene complexities (e.g.. streets, shopping malls, intersections, campus, etc.), and en- vironmental factors (day and night).
26

---

Published as a conference paper at ICLR 2025
(6) The Peraneeiie
sufting ciry
Intention Prediction of Ego Vchicle.
(b) Intention Prediction of Specific Vehicle.
K-/NG
(c) Driver Attention Understanding
(d) Interaction Relation between Traffic Elements
(2)
sappng Athming
We skjett
(c) Interaction Relation between Ego Vehicle and Traffic Elements
Figure 9: Data Examples for Reasoning Tasks in Autonomous Driving
We select high-resolution images from public monitoring image datasets with many real-world chal- lenges. For example, the VisDrone dataset (Zhu et al., 2021) brings several challenges, e.g., view- point variations, scale variations, and out-of-view, etc. Additionally, its dataset contains 263 video clips with 179, 264 frames and 10, 209 static images, which are captured via various drone-equipped cameras across various categories (e.g., pedestrian, people, bicycle, car, van, truck, tricycle, awning- tricycle, bus, and motor), density (e.g.. sparse and crowded scenes) and environments (c.g.. urban and rural regions). Additionally. The second dataset is collected in diverse environments (c.g., street, mall, clevator, etc.) for crowd density prediction task, features 3, 000 images with only per- son category, captured by fixed surveillance cameras. This dataset is highly diverse from the camera viewpoints (low altitude, high altitude, fisheye, etc.), scale size, and scene complexities. The LLVIP dataset (Jia et al., 2021), which 30, 976 images taken in binocular cameras, and contains large number of pedestrians. We only select manually infrared images from it to test the model's robustness on different modals.
visible-infrared paired dataset for low-light vision, contains
Annotation. For all the questions in this subscction, two professional researchers manually create the questions and answers, and another expert reviews the quality of the questions to ensure they meet the required standards.
This dataset is publicly hups://aistudio.tt baidu, com//taasseterui/22883..
accessible through the AT Studio
website
27

---

Published as a conference paper at ICLR 2025
B.5.2 EVALUATION DIMENSIONS AND BENCHMARK STATISTICS
Monitoring images are widely applied in real-world scenarios to increase public safety. Analyzing the monitoring images accurately with MLLMs would be highly valuable for public safety and crowd management. Specifically, we have designed three main tasks for monitoring images:
1. Object Counting (Fig. 10(a)). Task involves counting specific objects such as pedestrians, cars, or trucks in the given monitoring images (a total of 1, 600 images and 1, 600 QA pairs). Noted that, when the count of specific object is equal to zero, this object counting task well into the object existence task (Fig. 10(b), for judging whether a specific object exists in the given images. Thus, the object existence task be regarded as special case of the counting task. Additionally, we categorize this task into two sub-tasks for vehicle counting (608 images and 608 QA pairs) and person counting (992 images and 992 QA pairs), respectively.
be transformed
2. Object Location (Fig. 10(c).. Task involves judging the location of the specific vehicles, like cars, trucks in the given monitoring images total of 136 images and 136 QA pairs).
3. Attribute Recognition. Task involves identifying and describing the attributes of specific objects, e.g., color recognition (Fig. 10(d) and orientation perception (Fig. 10(e)), the monitoring images (a total of 460 images and 460 QA pairs). Additionally, we categorize this task into two sub-tasks for the vehicle (352 images and 352 QA pairs) and person attribute recognition tasks (108 images and 108 QA pairs), respectively.
In addition, there are three seasoning tasks described follows.
1. Calculate the Sum of Different Objects (Fig. 11())). Counting various objects and calculating their total number accurately (300 images and 300 QA pairs).
2. Intention Reasoning (Fig, 11(b)). Reasoning the next route and turn of the specific object (98 images and 98 QA pairs).
3. Attribute Reasoning (Fig. 11(c).. Reasoning the specific materials and functions of the given objects, such as inferring the function of the dustbin via its appearance (100 images and 100 QA pairs).
C DISCUSSION ON DATA DIVERSITY, SCALABILITY, AND LIMITATIONS
The creation of a diverse and scalable dataset poses significant challenges, particularly in the con- text of high-resolution, real-world scenarios. This section discusses our rationale for domain se- lection, identifies current limitations in dataset diversity and scalability, and explores the potential for leveraging models to enhance scalability. By addressing these key aspects, we aim to provide comprehensive overview of our approach and outline possible avenues for improvement in the future.
C.I ADVANCING DATASET DIVERSITY AND SCALABILITY: CHALLENGES AND MODEL- STRATEGIES
The creation of a high-resolution. diverse, and scalable dataset is a multifaceted challenge that in- volves balancing domain-spenn speciffc requirements, practical constraints, and innovative approaches to enhance efficiency. In this section, we summarize the key aspects of our work, including the ratio- nale behind our domain selection, existing limitations in dataset construction, and the potential for model-assisted strategies to improve scalability.
Rationale for Domain Selection (Appendix. C.2): We prioritized domains such as remote sens- ing, surveillance, and autonomous driving for their practical value and unique challenges, focusing on high- high-rcsolution imagery with complex scenarios. These domains are better suited for testing nuanced perception and reasoning compared to simpler datasets like COCO, which lack scene com- plexity and high resolution.
2. Current Limitations and Plans for Extension (Appendix. C.3): Our dataset faces challenges in task diversity and scalability. There is a lack of high-resolution natural scene data and underrepresen- tation of domains like indoor scenes, healthcare, and AR/VR. Additionally, the dataset construction
28

---

Published as a conference paper at ICLR 2025
process, requiring significant human effort, limits scalability. Future plans include capturing natural images, expanding to more domains, and exploring strategies to reduce manual effort.
3. Exploring Model-Assisted Approaches to Enhance Scalability (Appendix. C.4): We trialed MLLMs for data filtering and question generation. While models like GPT-4o effectively filtered images, their performance in generating complex QA pairs was suboptimal, often producing lower task difficulty and higher error rates compared to manual annotation. This suggests that while model- assisted pipelines can reduce workload, further refinement is needed match the quality of manual processes.
C.2 WHY WE SELECT DOMAINS SUCH AS REMOTE SENSING AND AUTONOMOUS DRIVING INSTEAD OF OTHERS LIKE EMBODIED AI OR NATURAL. IMAGES IN COCO
Practical Value: Instead of opting for data sources like COCO, our primary objective is to enable the understanding of high-resolution, real-world scenarios that are closely tied to domains such as autonomous driving and surveillance. Recent studies have demonstrated a growing application of MLLMs in these realistic fields, which pose unique challenges to their perception and reasoning abilities. The selected domains specifically demand high image resolution, enabling MLLMs to showcase progress in these settings. As result, we prioritize areas where MLLMs are currently poised to make significant impact.
Benchmark Objective: Our benchmark is not limited to autonomous driving surveillance appli- cations. The core goal is to evaluate MILMs' fundamental perception and reasoning capabilities within these domains. The tasks we design include various scenarios that researchers have identified as appropriate for assessment through multiple-choice QA formats. We deliberately avoid more spe- cific or complex tasks, such as robotic arm trajectory prediction, maintain a focus on fundamental capabilities.
Task Difficulty: We aim to emphasize high- high-resolutiin imagery and focus on challenging and subtle objects. In contrast, embodied AI scenarios often feature large, static objects, which pose relatively limited challenges perception tasks. Similarly, while COCO indeed provides large number of samples, its images are relatively small resolution, with insufficient complexity in the scenes. For instance, many COCO images feature only a single object, making it difficult to create sufficiently challenging questions.
This limitation is one of the primary reasons did not select COCO as major data source. Cover- ing all the scenes and categories in COCO would require capturing images ourselves using cameras in complex environments to generate sufficiently challenging, high-resolution perception tasks. This approach would undoubtedly be resource-intensive and difficult to scale, making it less feasible for our benchmark objectives.
C.3 EXISTING LIMITATIONS AND PLANS FOR EXTENSION
At present, the main limitations of work are concentrated in the diversity of tasks and the scala- bility of the dataset.
1. Perception of Natural Scenes: Although we discussed in Section C.2 some data sources that were excluded during the dataset construction process, such as COCO, which were deemed too simple for high-resolution perception tasks, there remain challenges in finding sufficiently complex, high-resolution natural images containing small objects on the internet. This limitation has resulted in lack of natural scene data in our dataset, which may hinder the evaluation of models' question- answering capabilities related to natural landscapes. To address this issue, we plan to collect natural scene data through self-captured images or generate complex high-resolution images using genera- tive models in future work, thereby compensating for this shortcoming.
Extension to Other Domains: Beyond domains like remote sensing, surveillance, and au- tonomous driving, which have high-resolution requirements and demand understanding of complex scenes, there are other practical applications such as indoor scene understanding, commerce, health- care, robotics, and AR/VR that also share similar needs. Due cost constraints, current work does not encompass all these domains, leading to potential bias in the dataset. For example, surveil- lance and autonomous driving data primarily human-centric, which might lack evaluations of
29

---

Published as a conference paper at ICLR 2025
other critical factors. In future work, aim to further explore these mentioned domains, identify suitable images, and annotate them to enhance the diversity of our dataset.
3. Weak Scalability: Our dataset construction involves two key stages requiring human involve- ment.
Stage 1: Image Selection and Filtering: During this stage, ensure: The images contain challenging or valuable small objects. The images are of high quality, free from glare or noise. The scenes are clear enough to minimize ambiguity in QA tasks (e.g., if question asks about person in a blue shirt in the upper right cornrr, there should be only one such person).
By ensuring high-quality and challenging images, we provide annotators with images more likely yield usable QA pairs, enhancing annotation efficiency.
Human Annotation: This involves crafting questions and answers for the complex images. The human cost throughout this process is relatively high, which makes it challenging to scale to other languages data scenarios. To mitigate this issue, we have discussed in Section C.4 some promising alternative strategies, such as leveraging MLLMs to reduce the reliance on human labor.
Stage 2:
C.4 EXPLORING MODEL-ASSISTED APPROACHES TO ENHANCE DATASET SCALABILITY
To explore whether models assist in improving the scalability of our dataset construction process, we conducted a small-scale trial using the following approach:
We implemented model-based strategy to streamline the data filtering process. minimum resolution threshold of 1 1024 x 1024 was set to ensure the images were sufficiently large. We employed a MLLM to remove images that were noisy or unclear. The MLLM was prompted with the following instruction:
1. Data Filtering:
"We provide an image; please score the scene from two aspects on scale of [1,
2, ..., 10] and provide Objects with Observational Challenges:
1. Complexity: Assess whether the image contains large number of elements, such as various objects. If the scene features only one prominent object and the foreground/backkroumn is relatively simple, the score should be Conversely, if the image includes multitude of elements and the scene is complex, the score should be higher.
2. Object Salience: Evaluate whether the image contains small objects that are difficult to observe clearly, specifically objects occupying less than 1/20 of the total pixel area. If all objects the image can be observed very clearly or are relatively large, the score should be O.
3. Objects with Observational Challenges: If there are objects in the image that are difficult observe or smaller targets, please list these objects and their locations."
Using this scoring strategy, annotators could easily reference the model's 'ss assessments of image com- plexity and the presence of small objects. Currently, we are utilizing GPT-4oo as the scoring model. In the future, voting mechanism involving multiple models could be employed for improved qual- ity assurance. This approach has successfully reduced the workload for annotators, enabling us to quickly filter out simple images.
We explored using GPT-4o for question generation and answer construc- tion. However, the results were suboptimal for high-resolution scenarios, especially in reasoning tasks. The generated samples were either too easy or contained errors, rendering them unusable.
Manual Annotation:
To address this, we tested multiple GPT-4o. Each model independently generated three questions and answers for a single image, fol- lowing the construction standards outlined in article. Predefined prompts were provided to ensure consistency. Human experts then reviewed and retained the most challenging and reasonable questions from the model outputs.
Llava-ov-72b,
and
30

---

Published as a conference paper at ICLR 2025
Rhst the mumker ofordigg
CaT Nange!
Fil Soltee
(a) Object Counting in Monitoring Images.
(b) Object Existence in Monitoring Images.
apter
Tie nige
Siher amagg Ser
age Joude
(c) Object Location Monitoring Images.
(d) Color Recognition in Monitoring Images.
1 I1
Prat the orientiio of the
tricycle in the image?
The right the inaage
of imuge
(C) The left the image
The the image imaay
tricycle.
(c) Orientation Perception in Monitoring Images.
Figure 10: Data Examples for Perception Tasks in Monitoring Images
While this method allowed to obtain some high-quality QA pairs, small-scale experiments revealed that: The task difficulty generated by models was generally lower than that of purely manual annotation. The rate in model-generated samples was significantly higher. -Mauually reviewing multiple model outputs did not substantially reduce the time required compared to direct manual annotation.
Observations and Future Directions: Our findings suggest that, for -utra-high-resolution percep- tion tasks, existing model-assisted pipelines are not yet optimal. The limitations include reduced task complexity and higher error rates compared to purely manual processes. Additionally, the time efficiency of manual review was not significantly improved. We believe this represents a valuable direction for future research.
D EXPERIMENTAL RESULTS ON ALL TASK SPLITS
OCR in The Wild Performance. Tab. displays the performance various models on real-world OCR tasks. Generally speaking, when image resolution is high, the more advanced models still
31

---

Published as a conference paper at ICLR 2025
Phar the toial number motors smage?
(R)
F33
The imagge not festuure objects.
(a) Reasoning Task of Calculating the Sum of Different Objects in Monitoring Im- ages.
image?
(4) Stopping
(8) Keep montng
Turm left
right
Thk Dmuge does not feuture
abject.
(b) Reasoning task Intention Special Object Monitoring Images.
Plas the yellow trach the
image?
Exprees delivrry
(8)
Bane supply
Food impply
meppy
The mage does not foature object.
(c) Reasoning task of Attribute of the Special Object in Monitoring Images.
Figure 11: Data Examples for Reasoning Tasks in Monitoring Images
demonstrate commendable OCR capabilities. However, this does not imply that task is of low difficulty. The average accuracy rates of Qwen-Ve and the basic LLaVA model on perception tasks only slightly better than random guessing. In this task, the gap between open-source and closed- source models not significant. GPT-44o ranks first in overall performance, while Claude 3.5 Sonnet leads in reasoning tasks.
Diagram and Table. Tab. presents the results for the diagram domain, where some of the more advanced models perform relatively well, with four models achieving average accuracy of
32

---

Published as a conference paper at ICLR 2025
6 to
(4) 175893714.34
6263
175893714
(8) 48
175893714.35 ( 175893714.33
(C)
(D) 36
#6
(8) #5
Figure 12: Data Examples for Perception Tasks in
Table 6: Experimental results on the OCR in the Wild tasks are categorized as follows: "Product" represents products and advertisements: "B & M & p" represents books, maps, and posters; "Contact" denotes contact information and addresses; "Identity" pertains to identity infor- mation; and "Signage" refers signage and other text. Models are ranked according to their average performance on perception tasks, from highest to lowest. Rows corresponding to proprietary models highlighted in gray for distinction.
Perception
Reasming
Ideatity
Signage
Avg.C
Avg-C
$7.64
hacm/V1-2
Incnn 7---Cht
73.47
Clank 3.5
83.67
69:70
71121
InsenVL VL-Char V1-5
75.56
73.36
67.05
71.51
$7.60
56,00
56.80
56.50
CogVLm2 Hama3. Chut
1Lama3 3B
66.52
76.41
67.05
69:97
70.92
58,80
49.20
54.00
$4.00
Gemini 34B
Nous Hermes 34B
79.04
55.16
66.61
67.45
60.30
59.20
59.20
InteLLM XCompoee225
ImemLM2
71.83
78.29
69.1
50.40
$3.40
$3.40
Gemint
66.37
70.19
66.35
68.65
36.60
48.
52.70
$2.70
MinCWMF-V 2.5
LLam33-8
69.3
71.58
68
62.19
65.79
67.35
48.30
39,
46.00
64.00
Combrin-1--5 --34
Noos-HHerees-3 Vi-3411
67.65
77 30
62.22
40.47
64
65.45
66.37
$4.00
56.00
$5.00
$5.00
GPT--4o-mini
02.31
68
S4.1
62.56
58.51
67.51
61.37
$2.80
41.200
47.00
47.00
Combionn-l.d :38
50.15
67.22
S5 96
$2.93
$2.25
$3.68
$7.52
$2.50
51 60
$3.20
$3.30
Maskey
Qwen-78
55.55
$4.47
$3.38
99.98
50.42
$4.63
$4.77
32.40
22,00
27.20
27.50
SEME-SD
1Lam33 3B
$7.30
53.65
$1.90
55.60
50,
53.20
$3.20
mPL. DPP.LG--DeDOel 1.5
HLMA-18
$4.62
$2.20
10
63.0
$2.99
$1.15
$2.39
46.80
18 40
42.60
42.60
SEMF-
Vizun--133 138
$2.25
46. 50
53.76
$3.17
SU.E
50.53
45.60
36.40
41.00
41.00
ScckSSco-VV..
DeepSeck-E.1 M-Th-base
53.23
55.06
31.72
44.13
49-42
40.55
46.81
48.50
41.
45.20
45.30
1 LLaVA-NNeat
11 112mm3-38
90.77
49.45
38,30
36.73
$3.51
47.04
46.15
$8.00
$3.40
55.20
55.30
YLVL--3B
Yi-34B-Cla
48.33
50.55
33.07
37.91
43.57
44.95
42.57
46.50
38,00
42.40
43.40
StucoGPTAV-1338
Vicuna-138
46.92
40.51
40.38
46.01
47.66
44.55
44.30
31.20
30.50
26.00
36.00
LLaVALS-133
Vicuna-133 13B
45.51
40.39
39.69
4T.54
46.74
44.10
43.97
36,50
23,60
30.20
30.20
Mini Gemini 7B HD
Vicuna 7B v1.5
40.15
44.44
42.95
44.37
39.32
42,02
42.25
38,00
32,50
35.40
35,40
ShazeGPT4Y 7B
Vicuna 7B
40.50
35.43
37.61
42.14
41.99
39.39
39.55
25.00
22.70
25.15
24.15
MiniGPT-2
Liama 2:70 Chat
40.15
14.73
38.99
41.08
41.82
39.02
39.35
36.40
23.50
30.00
30/00
LLaVAL5-70 3-76
Vicuna-716
39.67
14.92
37.26
40.6
41.00
$8,09
36.00
30.30
21.20
26.00
36.00
Teathenney
Owen- 78
38.12
31 96
44.89
45.19
33.89
$7.90
38.81
36.00
24.90
30.40
30:40
HaVA--Net
Owin-22l
43.58
45.73
30.6
14.91
41.24
31.07
33.15
17.40
16.90
17.20
15:20
Owan-VI-Clol
Own-78
32.73
37,62
27.38
33.22
36.88
32.57
31:57
36:40
20.90
78.60
28.00
33

---

Published as a conference paper at ICLR 2025
Table Experimental results on the Diagram and Table tasks. Models are ranked accord- ing to their average performance on perception tasks. Rows corresponding to proprietary models are highlighted in gray for distinction.
Perception
Reasoning
Method
LLM
Diagram
Table
Avg
Avg-C
Diagram
Table
Avg
Avg-C
Qwen2-Vi VL.
Qwen2-77 BB
74.70
68.59
70.18
71.65
47.
49.39
48.60
48.37
Claude 3.5 Sonnet
71.31
66.08
67.44
68.70
60.92
61.35
61.20
61.14
InternL MCM.MM--om
InterL.
69.05
62.12
63.92
65.59
43.10
39.88
41.00
41.49
I1termV1.22
M2.5-7B--Chu
62.80
42.53
37.12
39.00
IntemVL-Chaa-V1s
InemmLM2-Chat-208
61.55
53,81
55.83
57.68
37.36
34,36
35,40
35,86
MiniCPM-V 2.5
57.81
51.05
52.81
54.43
26.44
34.66
31.80
30.55
CogVLmm2-lamu--Chat
LLama3-8B
51.52
46.09
47.51
48.81
31.61
33.44
32.80
32.53
GPT-4o
47.35
46.44
46.68
46.90
44.25
45.09
44,80
44.67
Mini Seni--Gmmini34BB
Nous HereHeeee-eeeee Yi-BB34B
43.21
44.36
39.57
39.20
39.04
GPT-to-mini
46,22
43.54
44,23
44.88
38.51
40.49
39.80
39.50
Cambrian- 1-34B
Nous-Hermes-2 Yi-34B
43.67
39,30
40.44
41.49
37.93
34.97
36.00
36.45
Gemini 1,5
41.41
39.37
39.90
40.39
35.63
31.90
33,20
33.77
Cambrian-1 1-8B
LLama3-8B-Instee
36.25
31.48
32.73
33.87
27.39
27.30
27. 27.40
27 45
Monkey
Qwen-7B
18.39
20.80
mPI H.PLUG-DecOOi 1,5
1.1 a.mam-78
30.74
28.85
29.34
29.80
18.39
201.55
19.80
19.47
SliME-BB
LLama3-8B
29.75
29.19
29.34
29.47
89
29.40
29.52
LLAVA-Next
Qwen-72B
27.77
27.65
27.68
27.71
36.78
32.82
34,20
34,80
LLaVA-Next
LLama3-8B
26.64
26.63
26.63
26.64
22,99
23.62
23.40
23.31
DeepScek-VL
DeopSckk-LMM-7bbbuuee
23.67
23.27
23.38
23.47
22.99
24.23
23.80
23.61
Mini Geminn-7B-HD
Vicuna-7B
21.20
22.70
22.31
21.95
27.01
23.31
24.60
25.16
SliME-138
Vicuna-131
20,93
MiniGPT-v2
Liama 2-7B-Chat
18.59
21.06
20.41
19.83
20.69
20.25
20.40
20.47
LLaVA1.5-13B
Vicuna 13B
18.30
20.83
20.17
19.57
22.41
19.94
20.80
21.18
ShareGPT4V-138
Vicuna-13B
18.37
20.81
20.17
19.59
21.84
20.25
20.80
21.05
L55V
Vicuna-7B
18.30
20.71
20.08
19.51
21.84
19.94
20.60
20.89
ShareGPT4V.7B
Vicuna-7B
20.08
19.94
YI-VL-34B
Yi-34B-Chat
15,90
16.03
15.99
15.97
23.56
27.30
26.00
25.43
Qwen-VL-Chat
Qwen-7B
18.42
14.56
15.59
16.49
14.94
12.88
13.60
13.91
TextMonkey
Qwen-7B
6.71
5.65
5.93
3.45
1.53
2.20
2.49
60%. Reasoning tasks, however, have proved to be more challenging. Only Claude 3.5 Sonnet manage to exceed 60% accuracy, standing out significantly, with the second-ranked Qwen2- VL trailing by 10%. Additionally, models like LLaVA-Next. which have performed well on existing benchmarks like chartQA, show noticeably weaker performance on our dataset, underscoring the higher difficulty of the our benchmark.
Remote Sensing. Tab. presents the performance various models on remote sensing tasks. is evident that models performing well on remote sensing data typically either employ special han- dling for high-resolution images (c.g.. Mini-Gemini- HD, SliME, Cambrian) or have vision encoders designed to support high-resolution inputs (e.g.. InternVL), Among these, SliME achieves the high- est performance due to its support for the largest resolution. However, even the top pop-perfrmiig model, SliME-SB, shows poor performance counting tasks with extremely large images, with only 30% accuracy. Some closed-source models perform even worse, with GPT-4o-mini achieving only 2% accuracy. This highlights the high demands of remote sensing data on resolution and detail perception.
Autonomous Driving. Tab. and Tab. 10 show the perception and reasoning performance of var- ious models in autonomous driving scenarios. As a critical application area, autonomous driving remains challenge for MLLMs, with no model currently capable of reliably addressing tasks such as intent prediction, traffic light recognition, and object counting solely through text. Only Claude 3.5 Sonnet achieve an average perception accuracy exceeding 40%. Reasoning tasks are even more difficult, with even the most advanced models achieving only around 30% accuracy. Autonomous driving inherently a high-risk domain that demands very high accuracy for practical deployment. This indicates that more powerful multimodal models with 3D spatial prediction and understanding ability, or specialized fine-tuning on -opeiii datasets for driving expertise, are necessary before MLLMs be effectively applied in this field.
Monitoring Performance. Tab. 11 presents the performance of various models under monitoring scenarios. As can be observed, the monitoring task poses high degree of difficulty. Traditional models like Qwen-V VL and LLaVA have an accuracy rate of around 20%, which is nearly equiva- lent random guessing. Open-source models significantly outperform closed-source models. For
34

---

Published as a conference paper at ICLR 2025
Table Experimental results on the Remote Sensing tasks. Models are ranked according to their average performance. Rows corresponding proprietary models are highlighted in gray for distinction.
Method
LLM
Color
Count
Position
Avg
Avg-C
Qwen2-VL VL.
Qwen2-7B
50.92
21.21
61.73
44,81
44.62
SIiME-BB
LLama3-8B
45.66
28.63
52.19
42.27
42.16
Mini-Gemini-34B-HD
Nou--ermes-2-Yii-ae
41.12
21.29
58.31
40.40
40.24
Cambrian-1-88
I.I.ma3 88 Instruct
38.01
20.55
61.10
40.05
39.89
InterVV--2
IaternLM2.5-7B-- BB-Chtt
47.41
25.69
44.63
39.35
39.24
Cambrian-1-384
Nous-Hermess22 2---34BB
37.05
22.76
55.69
38.63
38.50
Internl MaanllMM-Compere//
Internl M2-78 M2-771
45.34
17.62
44.95
36.12
35.97
InternVL-Chat- V1-5
M0ermLM2-Chat-2000
34.10
17.86
48.29
33.55
33,42
YI-VL-34B
Yi-34B-Chat
34.02
19.00
41.53
31.62
31.52
Mini- Mini-Gemiini7B-HHD
Vicma-7B-v1.5
37.29
22.43
33.97
31.30
31.23
JLaVA-Nex:
Qwen-
33.86
23.08
30.31
29.13
29.08
GPT-4o
34.18
15.17
37.07
28.92
28.81
CogVLm2-Ilamaa--00
LLama3-8B
37.69
18.35
29.99
28.76
28.68
MiniCPM-V P.CCP--V...5
LLama3 8B
37.69
11.50
33.49
27.69
27.56
SliME-13B
Vicuma-1
30.04
18.43
28.80
25.82
25.76
Claude 3.5 Sonnet
31,87
18.11
27.95
25.74
25.98
DeepSock-VL
DeepSeck-LLM-7b-B5
29.40
7.34
39.30
25.49
25.35
LLaVA-Next
LLamu3-8B
30.04
20.55
25.74
25.42
25,44
Monkey
Qwen-7B
22.07
16.97
35.72
24.99
24.92
mPLUG-DacOwl 1,5
LLaMA-7B
27.81
16.39
26.81
23.71
23.67
MiniGPT-v2
Llama 2-7B-Chat
23.35
20.15
26.41
23.33
23.30
1.aVA1,5-13H
Vicuma- 13B
26.22
16.88
26.57
23.27
23.22
ShareGPT4y--13
Vicuna- 13B
25.58
16.97
26.49
23.06
23.01
LLaVA115-7B
Vicuna-7B
23.11
16.88
26.25
22.12
22,08
ShareGPTTV-7B
Vicuma-7B
23.03
16.88
26.25
22.10
22.05
Qwen- VL.-Chat
Qwen- 7B
16.97
11.50
16.87
15.14
15.11
Gemini-L.S-pm
13.39
8.32
20.13
13.99
13,95
TextMonkey
Qwen-7B
6.93
2.04
25.86
11.69
11.61
GPT-4o-mini
5.82
2.61
11.54
6.69
6.66
Table 9: Experimental results on the Autonomous Driving perception tasks. Models are ranked according their average performance. Rows corresponding to proprietary models are high- lighted in gray for distinction.
Motion
Method
LLM
Identity
Veblcle
Muultivebicie
Pedestrain
Mutti-pedostral
Traffic Sienal
Object Count
Ang
Arg.C
Clande Sennor
55.50
18:45
15.
11.12
$1.54
19
40.73
49.73
Cambrrinn 13B
LLann 5B fastruct
56.77
50.00
33.29
32.52
14,20
25.82
33.06
35.52
47.65
IntemVV..2
Inseml 5-78--C--h
Qwmn2Vl
Qwnn2-78
MiniFPM
30.8
M2-77
39.03
Cambnon- 1-141l
Nous- Hormes-- Vi-141
3861
33.5
12.52
16.6%
DorpSock-VI.
Deeppook-111 M-/hiaase
44.05
61.70
35
56.50
13.05
22.30
14
16.73
Mini Gemai 34B HD
Nous Hemmes 2 343
39.78
42.41
36,09
31.10
15.53
17.41
31.53
32.70
36.24
EnenVL Chas V1.5
IemmLM2 Chat 200
00.42
39.87
26.97
27.44
13.56
24
30.04
31.42
35.92
Llams3388
33.15
36.71
20.77
31.71
19.27
43.1
30.23
31.69
Monkey
Qwen 71
33.97
60.76
24.30
37.20
18.26
32.34
24.72
29.57
32.83
YE-NL-3441
41.77
1910
31.11
20.03
19.11
26.33
DrcDww
LLams-7B
22.72
27.51
SIME
5-mm
LLaNA15
Vieuna
24.69
ShartPPay
Vieuna
24.1
31.10
24.69
LL
Vicume7B
23.25
31.01
34.1
31.10
25.06
35.32
26.81
26.04
24.5
ShamGP1ay-711
Vicuna-1 7M
25.25
31.01
26.1
31.10
25,90
15.57
26.81
25.04
24.55
Minit/Fe>2
Liama 2-7p--hht
23.11
53.16
12.96
23.6
20.00
35.97
20
25.5 96
26384
Mini Gemani-B 7B-HD
Vieun 7B.5
27.25
60.76
23.57
26.53
14.11
36.32
17.78
24.51
26.03
GPT to mini
19.07
45.57
24.67
22.75
11.36
40.30
31.11
24.18
21.63
1685
22.36
20005
Uwen-726
VL-CCal
15.3
12.17
TeuMonkky
22.72
11.40
35

---

Published as a conference paper at ICLR 2025
Table 10: Experimental results on the Autonomous Driving reasoning tasks. Models are ranked according to their average performance, from highest to lowest. Rows corresponding to proprietary models are highlighted in gray for distinction.
Mothod
LLM
Intention
Relation
Attention
Avg-C.
Fgo
Pedlestrian
Verhiele
Fggo1P
Ego2T
Ego2V
020
Signal
Monkey
Qwen-7B
Clande 3.5 Scomet
26.32
24.64
23.58
25.71
20.79
24.38
31.92
30.50
YL-VL-34B
Xc-34B-Chat
28.26
46.60
33.33
21.70
24.76
31,68
15.42
49.77
31.55
31.45
SBME-8B
LLama3-SB
28.29
39.81
33.33
24.53
19.05
22.77
10.45
63.59
31.55
30.37
Qwen2-VL
Quen2-7B
19.08
43.69
35.75
13:21
29.52
22.77
18.41
64.98
51.47
30,99
CogVLm2-llana Chat
LLama
30.26
25.24
25.60
35.85
20.95
26.71
18.41
31.18
30.27
MiniCPM-V2
SBiME-13B
Vicuna-13B 13B
25.00
41.75
28.
28.30
21,90
24.75
25 25.87
30,80
30.64
LLaVA-Nest
LLamaš-SB 8BB
32.89
49.51
33.82
28.30
25.71
24.75
7.96
43.32
30.73
30.78
Cambrian- SB
LLamaš- SB Instricttttt
25.00
41.75
35,2
23.58
25,81
16.83
11:44
60.37
30.73
29.56
InsemV1122
Intern1.22.5-77 1-22.5-7B--0 Chal
24.01
43.69
32.85
22.64
.7.57
21,78
21.89
43.78
29
29.89
LLaVA-Nest
Qwen-72B
35.27
27.62
29
InserVVL Chat- VL-5
M2
9.45
DeepSeck-VL.
DeepSeck-Lan MLLSscckLLLM--b--- 7b-bbue
30.26
17.48
27.05
22.64
25.71
24.75
6.97
$1.15
27.31
25.92
GPT-4o-min
11.51
19:42
24 64
22.64
26.57
17,82
31.34
54.84
26 79
26.40
GPT-4o
12.11
19:42
21.54
15.09
20.00
22.77
16.92
60.83
26.41
25.12
mPLUG- CPLLUG-De<OwlL..
LLama-78 7B
20.72
26.21
30.43
19.81
31,43
25.74
12.94
41.91
26.01
26.14
LLaVAI,5-138
13B
Qwen- ViL-Chut
Qwen
20.39
20.77
16.04
23.81
17,82
16.92
50.60
24
23,60
SturoCPTaV.J3B
Vicuna: 13B
23.36
17.48
26.09
25.47
27.63
22.77
25.37
26.27
24.55
24.33
Cambrian-1 1134BB
Hous-Heemes- 2- 2-YB-BBB
14,14
24.27
22.71
24.53
30.48
31,68
10.95
46.54
24.40
25.52
LLaVAI 5-78
Vicuns78 7B
23.36
17:48
26.09
25:47
27.62
22.77
25.37
23.96
24.
24.03
ShuveGPTeV-7B
Vicuna: 7B
23.36
17.48
26.09
25.47
27.82
22.77
25.37
23.96
24.18
24.03
InternLM C.ompoeer22
MnierLLM277B
MiniGiT-v2
Liaaa 2-78-Chat
23.68
23.24
28.02
28.30
22,86
21.78
2,49
37.33
23,66
23.71
Mini-Gemini-R--B
heuma-78-v1.5
21.05
6,80
14.49
19.81
15.24
25.74
1542
$4.38
23.20
21,80
Mini- Gemini- HID
Nous Hemmes-2 2-BB-34B
14.14
22.33
21.74
23.58
31.43
30,69
10.45
47.00
22.81
21.91
TexiMunkey
Quen-7B
9.54
24.27
23.67
24.53
17.14
20.79
11.44
35.94
20.01
20.81
Geminni-1 5-pro
instance, Intern VL-2 has an average accuracy rate of 53.19 on perception tasks, greatly surpassing GPT-4o's 33.93. We notice that closed-source models such as GPT-4o have a high frequency of answering "E", with over 35% of responses choosing "E". This suggests that closed-source models may be more inclined to refrain from responding when the answer is uncertain. Furthermore, we find that while most models perform reasonably well counting tasks, they struggle with tasks related to spatial relationship judgment and attribute recognition. Moreover, related reasoning tasks also pose high level of difficulty, with model achieving accuracy rate over 40% to date. In combination with the results from autonomous driving tasks, we observe that MLLMs exhibit sig- nificant deficiencies in understanding, predicting, and reasoning about the dynamic information of objects in 2D or 3D space. Although the input to these models is a single frame image rather than temporal sequence of video frames, there remains considerable gap between their performance and that of humans. For humans, who possess rich experiential knowledge of dynamics, it is not dif- ficult to infer the future states of objects from single image in unambiguous situations. Therefore, modern MLLMs are still far from having the capability to function as world models.
E EXPERIMENTAL RESULTS ON ALL TASK SPLITS MME-REALWORLD-CN
The detailed results on MME-RealWorld-CN can be found Tab. 13, Tab. 15, Tab. 14, Tab. 16, Tab. 17, and Tab. 18.
F
To facilitate the evaluation of high-resolution perception abilities in multimodal large language mod- els, we plan to sample 50 examples from each task within each domain, resulting in a mini-test set.
Tab. 19 summarizes model performance on the original dataset due to the sampling process, the overall task difficulty has increased. We reduced the number of samples for simpler tasks, such OCR and basic reasoning, making the test set more demanding. For instance, Qwen2-VIL VL's performance decreased from 56.4 on the full dataset 46.7 the reduced test set, underscoring the increased difficulty.
initial tests with several models. While it is challenging to fully preserve
36

---

Published as a conference paper at ICLR 2025
Table 11: Experimental results on the Monitoring fasks. Models are ranked according their average performance on perception tasks. Rows corresponding to proprietary models are highlighted in gray for distinction.
Perceplian
Reasoning
Method
Velicle Lecation
Pedestrin
Arg-C
Caleolate
Intention
Property
Avg-C
Counting
Alleitute
Counting
Altribulc
IreeVV--Cha V1.5
Iroen MD 30B
13.35
37:37
7.41
51.16
39.52
35.95
Canisian 1-50
LLamal SB Instrat
02.01
29:41
20.45
55.44
7.41
47.58
31.87
45.00
29:33 10.59
44.00
42.57
39.56
Canictts 1:34B
Neus Bernce2 2-1-5-2---1-4-
$1.32
33.00
26.14
$5.14
12.95
45.98
37.44
11:33
18:37
45.00
19.48
24.00
SUMI--4
00:33
11.33
18.08
11-48
11:48
40.63
13.31
lovel M-XComppunt
fromm ME5-711
5361
13.34
17.61
11.9%
19:48
73 1B
1360
13:09
54000
17.07
30.51
MisiCPSS V
LLaraa SB
02.65
1691
22:73
35.40
4.63
38,70
30.35
30,00
35.71
41.00
56.05
$7.57
Qweuz VL
Qwead 75
35.59
19.55
25.00
35.60
13.03
37.30
29:45
30:33
19.39
55,00
35.15
34.91
Y5-H1B--hal
12:47
5.56
25,00
44.00
Mini-amm--1e
40
19:50
0.400
33.95
35.76
Con'Lan2 danaa Chuc
LLarra SB
48.9
26.47
32.16
32.36
12.04
33.74
25.16
-0.00
-0.33
45.00
41.16
41.96
Clinde Sornet
$059
35:77
11.95
33.37
8.33
32:19
25:43
34.37
18.37
44.00
12.25
Gemi-
9.56
1000
31.11
24:21
11:67
13:27
17.31
Qwen- 22
Menloy
7B
42.76
40:44
21:37
9:36
38.01
27:21
43
27.51
98.99
DaepBook VI,
MeopSod--11M-Thhbe
43:41
7:35
17:35
36.70
9:36
76.97
31.59
5.75
19.39
48.00
16.67
24. 54.54
GPP 4o-7-
44.57
8.82
8.53
25:31
3:70
25.30
19:30
7:35
5.36
19.00
11.50
11.50
LLaMA-7B
6.48
10:00
SIMME-18
338
22.79
23.30
24.73
21.25
26.33
33.18
Owen- Vi VloChor
Oswm-71s
77.65
16:30
31.88
14.33
13.04
12.15
30.35
1440
17:35
21.00
16:02
12.67
Vizame- 105
14:47
32.06
31.99
34.39
17.55
30.45
19:10
27.00
23:47
$100
27-51
37 18
LL Nest
LLarza 4B
46.71
0.00
22.16
4.13
23.15
19.46
19.27
1367
46.44
18.00
21
26.20
StaneciPTa
Vicuno
12.04
17.85
24449
27.31
2-18-Chai
LLaVAl 5-18
Viamma-BB
14.31
22.06
1862
33.79
11.11
19.15
17.67
22.31
23.47
24.00
34.95
Viamma-8
14:31
22.66
15.82
23.99
11.33
19.15
17.67
21.33
23:47
25.00
36.00
15.27
TecMMokke
Qween TB
39.47
6.02
7.10
8.17
0000
16.14
13.92
0.67
4.08
16.00
4.42
6.92
This compact test set offers researchers practical yet robust tool for small-scale, focused evalua- tions and model comparisons. It maintains a balance across diverse tasks, ensuring that evaluations remain insightful while being accessible the broader research community.
G OTHER ANALYSIS
Extended Metries In addition to the multiple-choice option, we also plan to release a performance evaluation version of the dataset (EM), which retains only the question without providing choices. The evaluation will be conducted using exact match or GPT-match" to assess the final result. This approach prevents models from relying on information from the choices. We selected 50 samples from each task for testing, using the prompt "Please respond to the question with a single word or phrase," prompting the model to generate a direct response. The generated response then compared to the correct answer to check for consistency. The input-output format of EM compared to naive MCQ is shown in Fig. 16.
In our experiments (Tab. 23), all models showed noticeable performance drop when choices were removed (EM vs. MCQ). Additionally, using GPT-4 to align the model's response with the intended meaning of both the response and the correct option (line 10 line 9) showed some performance improvement; however, the results still lagged behind those with choices provided. Consequently, Machine Match is key evaluation strategy we plan to prioritize in future assessments. Under this evaluation strategy, GPT-4 achieves around 30% accuracy, highlighting the increased difficulty level of our tasks. This suggests that the challenge level of tasks based our benchmark data could be further enhanced.
Simple Tricks Cannot Solve the Problem of Large Image Perception Effectively. A straight- forward strategy was attempted to alleviate the visual perception challenges posed by large images. Intuitively, when humans observe high-resolution remote sensing images, they often zoom in for a closer look. Additionally, questions and answers often include directional cues, such as "the up- per right corner of the image." "above the image," or "below the image." Hence, it would be more reasonable to segment the corresponding image before processing.
To implement this, each image is uniformly divided into 9 sub-images, and questions are posed for each sub-image, obtaining a corresponding answer. Each image has multiple-choice result; if any
'We prompt GPT-40 with the following to determine consistency between the model output and the correct answer: "Please determine whether the following two responses are consistent. If they are, output 1; otherwise, output Answer A: (answer1 ); Answer B: (answerH), The oulput number (0 or 1) is:"
37

---

Published as a conference paper at ICLR 2025
Experimental results on the reasoning tasks of MME-RealWorId-CN, Models are ranked according to their average performance. Rows corresponding to proprictary models are high- lighted in gray for distinction. "OCR", "DT", "MO", and "AD" each indicate a specific task domain: Optical Character Recognition in the Wild, Diagram and Table, Monitoring and Autonomous Driv- ing, respectively. "Avg" and "Avg-C" indicate the weighted average accuracy and the unweighted average accuracy across subtasks in each domain.
Table 12:
Method
LLM
Reasoning
Task Split
OCR
DT
MO
AD
Avg
Avg-C
QA pairs
207
602
298
800
1907
1907
V1.2.5
M2.55-22
53.14
73.59
30.20
31.25
46.83
47.05
Intem
IntmmLM2.5-7B-Chhm
44.93
74.92
38.14
29.00
46.65
46.75
Qwen2-VL
Qwen2-7B
38.16
72.92
57.00
33.37
46.46
50.36
Claude 3.5 Sonnet
74.44
65.79
31.54
25.12
44.31
49.22
SliME-8B
LLama3-8B
44.44
70.93
30.54
29.13
44.21
43.76
Vncm/L-Chhtt-i
V1-5
IntemLM2-Chha-220
48.79
67.11
30.20
29.88
43.74
44.00
CogVLMM-Ilama3-CC
LLama3-8B
33.81
65.24
37.25
29.00
42.25
41.33
YI-VI.334
Yi-34B-Chat
37.68
61.46
33.22
29.75
41.16
40.53
Monkey
Qwen-7B
43.96
56.81
32.55
28.38
39.70
40.43
Mini-Genin Mini-Gemini-7B-HD
Vicuna-7B-vI.5
28.50
68.6
25.50
24.00
38.80
36.65
Mini
nus-Hermes-2-Yi-Mae
29.95
60.47
25.50
29.63
38.75
36.39
LLaVA-Next
LLLama-8BB
14.93
18.14
29.53
28.25
36.44
32.71
Cambrian-1-88
-Lamm3-8B-Instenn
27.54
47.51
31.21
31.25
35.97
34.38
SliME-13B
Vicuna-11 13B
44.93
49.17
30.54
23.87
35.18
37.13
V.LaV--Nett
Qwen-72B
24.64
40.87
27.52
28.12
31.67
30.29
DiommLM-XComposeee
IntemLM2-77B
18.36
40.70
16.78
30.00
30.05
26.46
GPT-40
33.81
39.87
20.81
22.75
29.05
29.31
DeepSock-VL.
DecpScck-LLM--bbbbans
36.23
23.59
25.50
29.25
27.63
28.64
Cambrian-1-340
Hermes2-Yi-34B
21.74
31.40
23.49
25.12
26.48
25.44
LLaVAI.5-13B
Vicuna-138
36.23
27.08
25.84
23.25
26.27
28.10
ShareGPT4V-13B
Vicuna-13 1BB
35.75
27.91
24.83
22.88
26.17
27.84
MiniCPM-V
LLama3-8B
36.23
29.90
16.44
23.87
25.95
26.61
LLaVA1,5-7B
Vicuna-7IB 7B
33.33
25.91
25.17
23.25
25.48
26.92
ShareGPT4V-7B
Vicuna-
33.33
25.91
24.83
23.25
25.43
26.83
Qwcn-VL-Chat
Qwen-7B
30.92
36.41
13.42
19.88
25.29
25.16
GPT-4o-mini
27.88
27.08
14.77
26.87
25.16
24.15
MiniGPT-v2
Llama 2-7B-Chat
34,30
28.57
19.80
22.13
25.12
26.20
mPLLUG-DocOwl 1.5
LLama-7B
37.68
24.42
19.80
22.38
24.28
26.07
TextMonkey
Qwen-7B
27.53
31.07
12.08
22.50
24.12
23.29
Gemini-1.5-pp
5.30
5.32
14.77
15.67
11.14
10.26
choice result is non-E, the image assigned a result based on the majority vote. Otherwise, the result is E. The results are shown in Tab. 20, indicating noticeable decline in perception performance across various datasets. There could be several reasons for this decline:
1. The images are very large, and the queried objects very fine-grained. After segmenting these images, the task remains highly challenging.
2. Positional words the questions lose their original meaning when the image is segmented without further processing, leading to misleading interpretations.
3. Image segmentation cannot guarantee that the queried objects are not split into different sub-images, resulting the loss of correct answers.
Thus, simple image segmentation tricks are unlikely to significantly improve the perception of com- plex images. Instead, the inherent capabilities of the models themselves should be relied upon and improved.
Analyzing Incorrect Choices. We investigate the distribution of incorrect choices across range of models, shown in Fig. 13. We can see that MILMS show different response strategies when dealing with questions imbued with uncertainty. Larger models generally adopt a more conserva- tive approach, often opting for the safer response "E", as illustrated from Fig. 13(a) to 13(c). In contrast, smaller MLLMs often lean towards the first option-usually option "A"-in similar sit- uations, as shown in Fig. 13(d) and 13(e). Notably, Intern InternV--2 presents unique distribution of incorrect choices that is remarkably uniform, which may account for its exceptional performance in evaluation.
38

---

Published as a conference paper at ICLR 2025
Table 13: Experimental results on the OCR in the Wild tasks of MMMERRalWorrdd-N are categorized follows: "Product" represents products and advertisements: "B & M & P" represents books, maps, and posters; "Contact" denotes contact information and addresses; "Identity" pertains identity information; and "Signage" refers signage and other text. Models are ranked according to their average performance on perception tasks, from highest to lowest. Rows corresponding to proprietary models are highlighted in gray for distinction.
Method
LLM
Perception
Reasoning
Product
& M
Contact
Ideacity
Signage
Arg
Arg
Scene
Choracter
Avg
Avg-C
Owan2-VT
Qwanl-TR
59.04
68:57
Incel ChT
44.97
Intenl M2-
57.68
51.81
59.15
50.00
48.79
47.63
32.61
32.00
33.81
Clunde Sonnet
53.73
74.
74.00
SEME-SR
1-8m11888
51.33
12.18
YI-VI-334R
Vi-34B-Clal
48.71
SBME-13B
Visau-1338
50.00
47.66
44.83
Gemini-1 Sqpur
5.32
Cumbrian- 1-34B
Henmes2- Yi.34B
21.78
CogYLm2
8B
33.81
33.76
Nous
17.76
30.38
37.50
17.66
14.83
Mc.Compour25
M3-7B
PP DilDDeDDdl 1.5
11 Hamss-7
MinCPPA--
$2.55
LLa/VA-Nest
Qwen-728
27.
2.41
27.74
34.64
Cumbria--1 1-48
LLamu33-8B Eastruct
33.82
10.84
29.39
26.00
27.54
27.49
TextMonkey
Qwen-7B
32.37
16.87
28:85
21,00
27.32
GPT
26.56
27.15
22.27
33.89
27.88
28.08
ShireGPTY 139
Vicuna
27.62
28.63
24.10
27.91
27.29
37.38
34,00
35.75
38,09
1LMAA1 5-138
Visums-138
27.62
27.
24.10
26.96
36.16
Qwan-V Vl-Cull
26.19
24.90
10.84
24.5%
35,00
31,05
DespSeck-VV
DeapSeck LLM-Tbbuse
28.30
28.33
22.89
26.52
35.00
36.19
MinGPP-42
2:7B Chut
27.56
26.14
22.89
26.11
34.30
34.23
StueGPT4V
Vicuna 7B
25.44
26.50
27.94
27.39
24.10
26.35
37.38
29.00
33.35
33.19
1LaVA1.5
Vicuna 7B
25.04
26.50
26.14
22.89
25.8
37.38
33.35
33.19
Table 14: Experimental results on the Diagram and Table tasks of MME-RealWoridd-CN Models are ranked according their average performance on perception tasks. Rows corresponding to proprietary models are highlighted in gray for distinction.
Method
LLM
Perception
Reasoning
Diagram
Avg
Avg-C.
Dingram
Table
Avg
Avg-C.
Owen2-VL Claude 3.5 Sonnet
Qwen2-74 BB
86.38
92.03
89.20
89.21
74.42
71.43
72.92
72.93
84.39
63.79
74.09
74.09
68.11
63.46
65.79
65.79
Intern
92.36
71.63
77.08
72.76
74.92
74.92
VnternVL.-CMe Vi.Chiii Chat-VV155
Intern Chat 20B
Cha
68.11
49.17
60.12
58.64
65.45
68.77
67.11
67.11
SliME-8B
LLama3-8B
88,70
55.15
58.25
71.93
64.45
77.41
70.93
70.93
GPT-4o
41,86
29.24
54,86
35.55
56.48
23.26
39.87
39.87
YL-VI-34B
Yi-34B-Chat
47.51
39.52
49.52
43.52
59.80
63.1
61.46
61 .6
SliME-13B
Vicuna- 13B
47.84
35.55
48.49
41.70
43.85
54,49
49.17
49.17
Cambrian-1-88
Sermes2-Yii344
29.57
35.22
44,34
32.40
9.30
$3.49
31.40
31.40
Monkey
Qwen-7B
$4,
59.47
56.81
Mini-Gemini-34B--4
Nous- Cous-Hermes-2-Yiie
30.90
42.52
40.60
36.71
50.50
70.43
60.47
60.47
Gemini- 1.5-pro
13.29
7.97
39.78
10.63
6.98
5.65
6.32
6.32
CogVi.m2 wogVL_m2-llama3-cc 3lama3-hhtt
I.Lama3-88 8B
8.64
28.24
39.48
18.44
67.11
63.37
65.24
65.24
Mini--eemini-BBHHH
Vicuna-7B-v1.55
37.54
30,56
39.29
34.05
60.13
77.06 08.08
68.61
68.61
Internl.M XComposer2.5
IntemL-M2-78
40.70
40.70
LLaVA-Next
LLama3-8B
12.61
39.53
37.49
26.07
51.50
64.78
58. 14
58.14
mPLUG-DecOwl 1.5
LLama-7B
22.26
31.89
31.83
27.08
14.62
34.22
24.42
24.42
GPT-4o-mini
45.41
32.23
31.79
38.82
20:27
33.89
27.08
27.08
MiniCPM-V 2.5
LLama3-8B
25.25
28.24
31.67
26.75
29.57
30.23
29.90
29.90 90
TextMonkey
Qwen-7B
41.86
30.76
36.74
36.88
25.25
3
31.07
Cambrian-1 1-8B
LLama3 8B Instruct
24.58
20.60
30.28
22.59
40.53
$4.49
47.51
47.51
V.
Qwen-72R
28,90
28.69
19.27
40.87
Swen-VL--Cann
Qwen-7B
21.26
37.87
27.89
29.57
30.90
41.91
36.41
36.41
ShareGPT4Y-133
Vicuna 13B
25.58
27.24
27.57
26.41
24.92
30.90
27.91
27.91
MiniGPT-v2
Llama 2-7B-Chat
29.24
26.58
27.05
27.91
26.91
30.23
28.57
28.57
LLaVAI.5-13B
Vicuna- 13B
25.25
27.24
26.25
26.25
25.25
28.90
27.08
27.08
DeepSeek-VL.
DeepScek-LLA M.MBkckk-LLMM-b-beeee
20.93
24.25
26.02
28.24
18.94
23.59
23.59
LLaVAI,5-7D
Vicuna-7B
24.25
27.24
25.75
25.75
24.58
27.24
25.91
25.91
SharoGPT4V-7B
Vicuna-7B
24.25
27.24
25.75
25.75
24.58
27.24
25.91
25.91
39

---

Published as a conference paper at ICLR 2025
Table 15: Experimental results on the Remote Sensing tasks of MME-ReelWorrddNN, Mod- els are ranked according to their average performance. Rows corresponding to proprietary models are highlighted in gray for distinction.
Method
LLM
Color
Count
Position
Avg
Avg-C
SLIME-8B
LLama3-8B
45.00
21.00
58.00
41.33
41.33
In-emVL-22
InternLM2.5-7B-Chat
52.00
23.00
49.00
41.33
41.33
Qwwn2-VVL.
Qwen2-7B
44.00
18,00
53.00
38.33
41.33
InternLM-XCompent
XComposer2.5
InternLLM2-7B
48.00
17.00
50.00
38.33
38.33
Mini- Ge---emmnii--HHHD
Nous-Hormes-2-Va5
Y---344
50.00
7.00
60.00
38.28
38.82
Cambrian-1-8B
LLuma3-8BIInstie
40.00
7.00
63.33
35.86
36.55
YI-VI-34B
Yi-34B- Chat
34.33
34,33
Cambrian-1-22 1-34B
Hermes2-Yi-s Yi-34B
31,00
11.00
62.22
33.79
34,50
Claude 3.5 Sonnet
38.00
19.00
41,00
32.67
32.67
-
Internl M2-Cmat--20000
53.00
32.00
LLaVA-Next
LLama3-8B
45.00
16.00
34.00
31.67
31.67
Monkey
Qwen-7B
33.00
21.00
25.56
26.55
26.53
DeepSeek-VV
DeepSeek-L..M-75
8.00
34.00
25.36
LLaVA-Nest
Qwen-72B
21.00
11,00
39.00
23.67
23.67
GPT-4o
19.00
11.00
41.00
23.67
23.67
CogVLm2-Ilamaa-Ch
LL
15.00
17.00
22.00
22.00
MiniGPT-v2
Llama 2-7B- Ch-h
25.00
20.00
12.22
19.31
19.13
mPLUG-DecOwl 1.5
LLama-7B
21.00
16.00
18.89
18.62
18.63
ShareGPT4V-13B
Vicuna-13B
25.00
15.00
12.22
17.59
17.45
1JaVA1.5-138
Vicuna- 138
26.00
1.5000
11.00
17.33
17.33
SliME-13B
Vicuna- 13B
27.00
11.00
14,00
17.33
17.33
ShareGPT4V-7B
Vicuna-7B
24.00
15.00
12.22
17.24
17.12
Mini-Gemini--BBHHD
Vicuma-7B-vI..
24.00
15,00
12.22
17.24
17.12
LLaVA115-78
Vicuna-78 7B
24.00
15.00
11.00
16.67
16.67
MiniCPM-V225
LLama3-8B
11.00
6.00
33,00
16.67
16.67
Owen-VL-Chan
Qwen-7B
21.00
9.00
15.00
15,00
15.00
Gemini-1.5 1.5-pm pro
9.00
8.00
20.00
12.33
12.33
TextMonkey
Qwen-7B
11.00
1.00
23.33
11.38
11.68
GPT-4o-mini
3.00
5.00
14.00
7.33
7.33
Table 16: Experimental results on the Autonomous Driving perception tasks of MME- RealWorId-CN, Models are ranked according to their average performance. Rows corresponding to proprietary models are highlighted in gray for distinction.
Mathod
LLM
Identity
Vehicde
Multi-chiile
Polistrain
Mulfi-eedstrain
Traffic Signal
Objjact Count
Asg-C
Owen2 VL
Qwen2-7B
20.00
47.00
35,00
27.00
17.30
00.00
25,30
35,56
26335
Monkey
Owen 73
35.00
59,00
22.00
26.00
41,50
50.00
18.00
33.56
15.35
DaspSook-VV.
Brsppack--11 M-Thhsss
90.00
61:00
32:00
30.00
15.00
46.00
26.00
35.71
35.71
Cambrin
LLamu3 5B instruct
51,00
51.00
44.00
(3)
10.00
37.00
29,00
35.57
35.57
Mini Gemmi- HD
Nous Nous Homes Y1-44
29.00
$2,00
43.00
31.00
21,00
28000
25,00
34.29
10.29
M-XCumposr225
SIME 13B
Viguna
20.00
30.43
Clande 3.5 Sonnec
38.00
32.14
MiemmL-Chae
Intena Intem1 M2-Chat-206
41.00
$3.00
32.00
34.00
15.00
21 2100
30.00
32.1 14
$2.14
SUME: sg
LLamz3 8J
44,00)
25.00
34.10
31.00
15,00
30.00
57.00
31.24
31.29
MiniiPPa22
Liama 2-7 2-7p-Chhr Chat
21:00
22.00
20100
30/00
34.00
36
(1)
23.83
79
LLaV 13B
Vicuna 13B
23.00
38.00
20.00
33,00
27.00
30.00
23.00
25.66
28.85
mPLUG DoeDhel
LLamu-7B
22.00
60.00
27.00
33.00
25,00
17.00
12.00
25.43
28.43
Gemmii-B-HH
Visuns-
30.00
18
LLNW115
Vicuna
20.00
27.00
SharGGP
Vicuno
20.00
27.00
VE-VL-334
Yi-34B-Chal
36.00
25.00
27.71
Qwen-Vi- VL--hot
Qwin
2600
79100
16/00
19.00
10/00
26.29
LLa Nest
LLamn33 SB
27.00
49.00
30.00
27.00
21,00
29.00
5,00
27.29
27.29
TeuDSonkky
Qwen 73
32.00
36,00
15.00
31.00
29.00
32.00
9,00
25.71
26.71
Camitrian 1-34B
Hemooc--i--44
33.00
30.00
35.00
31.00
5.00
11000
$4.00
36.43
76-43
Min V
LLam3 33
40.00
29.00
39.00
25.00
17:00
39.00
19.00
25.00
29.57
GPT do mini
15:00
14.00
21 100
14.00
9100
15.00
32.00
24.00
24:00
VA-Nsi
Qwan-728
25.00
21.14
Aypm
40

---

Published as a conference paper at ICLR 2025
Table 17: Experimental results on the Autonomous Driving reasoning tasks of MME- RealWorld Models are ranked according to their average performance, from highest to lowest. Rows corresponding to proprietary models are highlighted in gray for distinction.
Method
LLM
Intention
Relation
Attention
Avg.
Avg-C
Ego
Pedestrian
Verhicle
Ego2I'
Egu2T
Equ2V
020
Signal
Owen2-VL VL
Qwen2-715
25,00
37,00
36,00
23.00
35.01
25,00
16.00
73.00
33.37
33.38
Cambrian-
LLama3-8B-lnstee
17.00
31.25
Insernl M-XCCoopoee225
Internl. M2-7B
19.00
28.00
35.00
22.00
37.00
27.00)
18.00
56.00
50.00
InterVLL Lieemn- -h--hht-VV--
MBcentLM--Chha-200
30.00
25.00
35.00
26.00
25.00
24.00
9.00
61.00
29.58
29.88
YI-VL-34B
Yi-34B-Chol
30.00
37.00
32.00
24.00
26.00
30.00
15.00
42.00
29.75
29.75
Mini--Gmmini-Bn HD
Wous-Hemess2-o 21--344
22,00
13.00
26.00
26.00
35.00
36.00
16.00
63.00
29.63
29.63
DeepSeck-VL
DeepSeck-LLN Mb-bbau
30.00
37.00
34,00
23.00
29.00
20.00
10.00
51.00
29.25
29.25
SliME-8B
31
9.00
CogVi M2-T-mm--aaa
Mama3-88
24.00
29
21.00
27.00
19.00
51.00
29.00
29.00
InternVL-2
Internl Maternl-M..-78-CChh
25.00
33.00
30.00
26.00
35.00)
26.00
11.00
48.00
29.00
29:00
Monkey
Qwen-7B
22,00
46,00
31,00
21.00
23.00
21.00
9.00
$4.00
28.38
28.38
LLaVA-Nest
LLama3-88
27.00
43.00
34.0 00
26.00
24.00
23.00
11.00
38.00
28:25
28.25
LLaVA-Noxt
Qwen-72B
30.00
8.00
OPT-4o-min
11.00
26.87
Cambrian-1-33 1-34B
Hermax2-Yi- Yi-34B
11.00
10.00
24. 00
22.00
38,00
29.00
10.00
$7.00
25.12
25.13
Clande 3.5 Someet
34.00
20:00
25.00
16.00
14.00
12.00
15.00
65.00
25.12
25.13
Mini-Geminnl-B-Bi
Vicuna-78.v13
19.00
21.00
22.00
27.00
27.00
22,00
25,00
20.00
24.00
24.00
MiniCPM-V 2.5
LLama3-BB
22,00
28.00
28.00
20.00
28.00
17.00
12,00
36,00
23.87
23,88
SBMMF-138
Vieuna-130
24.00
15.00
25.00
23.87
LLaVAI.5-78
Vicuns-7R
19.00
27.00
22.00
25.00
24.00
23.25
LLaVAI 5-13B
Vicum= 13B
19.00
21.00
22.00
00
26.00
21,00
25.00
24.00
23.25
23:25
StoreCPTTV--7
Vicums-88
19.00
21.00
22.00
26.00
27.00
22.00
25.00
24.00
23.25
23.25
Store(iPTaV-138
Vieuna-1 13B
19.00
21:00
22.00
25.00
27.00
20.00
25.00
24.00
22.88
22.88
GPT-40
26,00
16.00
19.00
14,00
10,00
54.00
22:75
TexiMonkey
Qwen-7B
22.50
mPI. mPI.IGG-Deeehl
Mama-7B
17.00
20.000
20.00
33.00
21,00
10.00
36.00
22.38
MiniGPT-42
Llama 2-78-Chit
21.00
23.00
22.00
22.00
31,00
16,00
10.00
32.00
22.13
22.13
QweenVLL-Che
Qwen
18.00
13.00
8.00
22.00
27.00
22.00
14,00
35.00
19.88
19.88
Gemini-- 1eiimii-1.5-poo
17.00
12.00
27,00
5.00
8:00
10.00
14.00
32.00
15.67
15.63
Table 18: Experimental results on the Monitoring tasks of MME- ReME--eeW Models are ranked according to their average performance perception tasks. Rows corresponding to proprietary models are highlighted in gray for distinction.
Persalion
Reawoning
Methad
LLM
Calenlate
Intention
Property
Counting
Location
Attribate
Counting
Attribute
InooVii-
Inomm 5-18-Chot
2007
08
10.48
13.04
9::00
51.07
21:41
LLaYN
728
11.00
26.00
29.00
34.50
34,60
13:27
27.52
27.42
MemnVV-CCaatt
112 Chor-28
02.00
22.00
20.00
47.00
11.50
32.40
31.40
25.00
11:00
54.00
30.13
30.00
Owant-VV
Queax2-78
63.00
18:00
15.00
43.00
8.00
29.40
25:40
27:00
17:35
$3.00
55.89
13.78
SiME-48
LLamml-301
5500
18:00
17.00
40.00
15.00
20.30
25:20
24,00
31:63
Nexs
Contitinn 1:40
ap
22.00
13.00
6.30)
42.00
31.31
Cambrian 1:348
Hermest S4-
02.00
13.00
7.00
39.00
7.00
(3.50
31.60
11:00
15.3 15.31
33.00
19.50
19.77
LLarxa-B 78
3680
19:00
20.00
24.00
0..00
25,50
25.60
4.00
17:35
35.00
19.80
19.75
VENL-34B
Y5-34B-Cha
57.00
7.00
11.00
24.30
24.00
35.21
33.17
Gemini
6100
13.00
9.00
18.00
5.00
9:00
75.00
14:70
15.75
Clande 3.5 Sunnet
62.00
10.00
10.00
34.00
9.00
25.00
25.00
23.00
22
49.00
31.54
31.49
Oog'l VLa2 Hanta Chut
LLaraa-33 4B
32,00
25.00
22.00
33.00
13.00
24.50
24.80
2700
37.26
47.00
17.25
37.25
32350
23:00
M-7hhez
MMn V 2.5
Llama1-30 -30
$5.00
15.00
11.00
17.00
4.00
20.40
20.40
11.00
11:23
21.00
16.41
Quen VL. Char
Qwee-
$5.00
11.50
3100
11.0
24.50
20:30
20.30
7.00
12.00
21.00
13.34
13.33
ToxMMoske
Oven 76
57.00
2.00
15.00
24.00
10.00
19:50
19.03
1.00
12:34
23,00
12.08
12.06
Incem IncmmL McCoooeur225
6009
5.00
3.00
24.00
0.00
19:40
19:40
0.15
18.75
73
SBMME-18 20
Vicuna .38
$7.00
21.00
18.00
23.00
12.00
17.50
13.50
26,00
27.35
35.00
30.54
30.52
LLaVAT 5-158
Vicina 303
16,00
31.00
17.00
70,00
11.00
17.00
13.00
26.00
23-47
25.00
25.61
2553
Micana 100
15.00
31.00
12.00
70.00
11.00
16
16.33
26,00
23447
25.00
24.5
36
Vicura
21.00
20.00
15.50
15:80
26.00
LLaNAl 5-18
7B
23.47
25.17
25.16
StareCPP--
Vicuma-2 7B
15600
21.00
16.00
20.00
11.00
15.50
15.59
26000
23.47
25.00
34.53
24.52
MinttiFFx2
Liams 2-15-Chor
9.00
15.00
16/00
35.00
11.00
14:40
14.40
16400
18:37
25.00
90
19.9
Table 19: Performance comparison of various MILMs across perception and reasoning tasks on MeEE Metrics include task-specific scores (e.g., OCR, RS, DT, MO, AD) and
their averages.
Perception DT
Reasoning
Method
LLM
Overall
OCR
RS
MO
AD
Avg
OCR
DT
MO
AD
Avg
GPT-4o
46.4
80.8
45.3
65.0
33.9
37.4
49.
72.0
50.0
42.0
32.8
42.1
GPT--o-mini
37.4
69.9
23.3
62.0
19.4
34,3
38.8
$7.0
39.0
18.7
35.0
35.2
Intern-LLa VL-2.5
47.1
81.2
46.0
79.0
29.5
38.9
49.7
68.0
53.0
40.7
35.0
42.9
Qwen2-VV VL
Qwen2-7B
46.7
85.6
40.0
74.0
27.9
36.0
48.2
73.0
46.0
47.3
35.8
44.4
LLaVA-OV
Qwen2-78 7B
48.5
82.0
50.7
64.0
34,2
44.9
$2.8
71.0
43.0
44.7
34.8
42.7
Slime
Llama3-8B
37.1
57.6
36.0
51.0
29.2
32.9
37.7
51.0
27.0
40.7
33.5
36.4
41

---

Published as a conference paper at ICLR 2025
Table 20: Exploring the Limitations of Simple Image Segmentation for Large Image Percep-
tion. The results are evaluated on the perception tasks of
Perception
Method
LLM
OCR
RS
DT
MO
AT
Avg
Avg-C C
SliME-8B
LI -L8
53.45
42.27
29.34
40.62
33.66
40.29
39.87
Split Img
LLama3-8B
53.66
35.26
31.11
39.07
30.57
38.84
37.93
Qwen2-VL
Qwen2-7B
81.38
44.81
70.18
37.3
34.62
58.96
53.66
Split Img
Qwen2-7B
67.8
36.3
49.33
34.97
34.21
47.91
44.52
Intern VL-2
73.92
39.35
62.80
53.19
35,46
55,82
52.94
Split Img
IntemLLM2.557B-CMM
61.43
33.09
44.19
34.88
31.58
43.75
41.03
Table 21: Performance of different models on intention prediction tasks.
Eng
CN
JD-Intention-Eng
AD-Intention-CN
Method
Avg
Monitoring
Monitoring
Ego
Pedestrian
Verhicle
Ego
Pedestrian
Verhicle
GPT-4a
13.27
8.16
17.11
19.42
27.54
26.00
16.00
23.00
18.81
Claude 3.5 Sonnel
18.37
22.45
26.32
32.04
24.64
34.00
20.00
25.00
25.35
Intern
21.43
21.43
24.01
43.69
32.85
25.00
33.00
30.00
28.93
Qwen2-Vr 19.39
17.35
19.08
43.69
35.75
25.00
37.00
36.00
29.16
Limitations of MLLMS in Understanding Dynamic Information. In combination with the in- tention prediction results from autonomous driving and monitoring tasks (Tab. 21), we observe that MLLMs exhibit significant deficiencies in understanding, predicting, and reasoning about the dy- namic information of objects, such as predicting the steering of a car. Although the input to these models is single frame image rather than video, there remains considerable gap between their performance and that of humans. Therefore, it seems that these MLLMs are still far from having the capability to be world models.
Chain-of-Thought Reasoning does not enhance the model's high-resolution perception capa- bilities. As shown in Fig. 16, we further evaluate the impact of CoT reasoning on high-resolution tasks (for both MCQ and the EM mentioned above). Tab. 23 demonstrates that while CoT reason- ing can assist the model in reasoning tasks, especially in chart-related reasoning tasks, provides minimal benefit for perception tasks (line 6 vs. line 7 and line 8 vs. line 11). In fact, for models like GPT-4o-mini, perception performance even declines. This finding suggests that high-resolution perception tasks are already highly challenging on their own. Recent work MVoT (Li et al., 2025) investigates multimodal COT, opening new avenues for future research. However, the model can- not effectively receive and understand visual inputs, then even the strongest reasoning capabilities of an LLM will only lead to limited improvements. Therefore, enhancing MILM's comprehension of visual inputs remains critical focus.
What matters for ultra-resolution image perception: LLM capability or visual cognition capa- bility? It is indeed challenging completely separate model's capability process input images from its inherent perceptual ability, as these two aspects are closely coupled. However, in the case of high-resolution images, the model's capacity process inputs seems especially crucial. For exam- ple, as shown in the Tab. 24, Mini-Gemini-7B--HH and LLaVA1.5-7B use similar LLM architectures and have comparable training data, yet Mini-Gemini-7B-EE exhibits far superior high-resolution perception capabilities. This demonstrates the critical importance of handling higher-resolution data effectively. As result, most modern MILMs have incorporated various image-splitting strategies to accommodate larger maximum resolutions. Nevertheless, simply supporting higher resolution is not a complete solution to high-resolution perception challenges. For instance, while Intern- V1.2 has higher input resolution limit than Qwen2-VL. its overall performance is slightly lower. This implies that the ability to handle larger image resolutions alone is insufficient for robust high-resolution per- ception. The model's inherent capabilities (such information extraction and comprehension) also play a vital role.
Instruction Following Abilities. As described in Sec. 2.1, prompts specify that the model should directly select and output a single answer. In this regard, closed-source models generally perform better, with outputs being more concise and directly aligned with the instructions. However,
42

---

Published as a conference paper at ICLR 2025
1598
389
2012
2272
Genaa Truh
Grnal Trah
1148
Groad
750
1006
1100
4917
1:99
124
1951
675
1599
500
39
250
48
41
33
32
15
Labcls
Productod
Preduced
(a) Claude 3.5 Sonnet
(b) GPT-4o
(c) Cambrian- 1-344B
2306
1053
315
SG1
10:1
235
456
M477
4694
1M144
1000
2000
Grvns Thih
2151
1247
276
148
2140
Grond Thuh
2523
325
231
568
1164
704
524
565
2644
1194
256
1340
2543
11:44
226
134
902
181
524
9-8
400
2012
1092
1175
102
1000
2214
535
0
445
D
811
725
700
173
516
200
354
Predited Lanels
Predited Labch
Predecsd Labeh
(d) Monkey
(c) mPLUG-DacOwl 1.5
(f) Intern
Figure 13: Distribution of incorrec choices. The matrix reveals distinct response behaviors among different MILMs. Larger models tend to select the safer option "E", while smaller models exhibit bias toward the first option "A". Intern however, shows a unique uniform error distribution.
we have observed that some open-source models do not strictly adhere to our queries and generate significant amount of additional analysis. Sometimes, they even produce outputs that are excessively verbose, continuing until the token count reaches the predefined maximum limit. This indicates that the open-source models have lot of room for optimization in the ability of instruction following.
Preparatory Models have better AI security. We designed three metrics to assess the frequency and performance models when predicting "E" (indicating refusal to answer):
Correct E (%): This metric measures the percentage of times model correctly predicts "E" when the ground truth is "E," helping evaluate whether the model can recognize ques- tions that genuinely require a refusal to respond.
E Ratio in Wrong Predictions (%): This indicates the proportion of wrong answers la- beled as "E" among all incorrect predictions, offering insights into the model's tendency to choose "E" when uncertain.
# Predicted E: This counts the total number of times model predicts "E" within the current split, providing a straightforward view of "E" predictions overall.
Finally, Error Ratio (%) reflects the model's 's perceptual accuracy within the domain. Below are the summarized results from Table. 22, from which draw several key insights:
(a) Less capable models, such as SliME-8B, rarely opt for "E" and instead tend to select an- swers they consider correct, leading low rates of both correct "E" predictions and "E" frequency across domains.
(b) More advanced open-source models, like InternVL, show "E" frequencies comparable to closed-source models simpler tasks, such OCR Diagram and Table interpretation. In these easier tasks, most models are confident in their answers, so the frequency of se- lecting "E" remains low.
(c) On more challenging tasks, such Remote Sensing and Monitoring, all models have higher emro rates. Notably, proprictary models like GPT-4o and Claude35 exhibit both higher accuracy in predicting "E" and greater frequency of selecting "E" compared to
43

---

Published as a conference paper at ICLR 2025
Table 22:
In-depth comparison of metrics associated with different models' tendencies select
response 'E'.
Remote sensing
Model
Correct E (%)
E Ratio in Wrong Prediction Predicted
Error Ratio (%)
GPT-4o 50.00%
56.19%
1500
71.08%
Claude35
71.43%
63.00%
1759
74.26%
SliME
7.14%
9.13%
198
57.73%
Intern VL
21.43%
18.31 %
418
60.65%
Monitoring
Model
Correct E (%)
E Ratio in Wrong Prediction (%)
# Predicted E
Error Ratio
GPT-4o
94. 98%
47.39%
T138
67.50%
Claude35
97.26%
48.96%
1206
69.89%
SliME
85. 85.96%
18. 18.23%
571
65. 1 1%
Intern VL.
91.44%
31.71 %
800
62.39%
OCR
Model
Correct E (%)
E Ratio in Wrong Prediction (%
Predicted E
Error Ratio (%
GPT-4o
15.07%
18.57%
284
23.56%
Claude35
13.70%
38.02%
686
28.49%
SliME
0.00%
3.86%
50.32%
InternVI.
21.92%
28.19%
498
27.40%
Diagram and Table
Model
Correct E (%)
E Ratio in Wrong Prediction (%)
Predicted E
Error Ratio (%)
GPT-4o 11.76%
24.84%
763
53.65%
Claude35
5.88%
4.10%
32.92%
SliME 0.00%
9.30%
39
70.65%
VL
23.53%
29.58%
692
39.15%
Table 23: Performance of different models under various evaluation metrics, with input-output formats for each metric shown Figure 16. Removing choices (EM) significantly reduces model performance; while using GPT-4o for matching model responses helps somewhat, overall accuracy remains low. CoT reasoning benefits reasoning tasks but has minimal impact on perception tasks.
Perception DT
Reasoning MO
Line
Metric
Method
OCR
RS
MO
AD
Avg
OCR
DT
AD
Avg
MCQ
SliME
58
36
33
37.7
36.4
EM
Exact Match
13
2
11
11.3
2.3
MCQ
LLaVA-OV
82
51
64
34
45
52,8
71
43
45
35
42.7
EM
Exact Match
34
15
25
13
13
18.8
4
20
13
5.8
MCQ
GPT-4o-mini
70
23
62
34
38.8
57
19
35.2
EM
Exact Match
39
11
30
14
11.3
11.9
CoT-MCQ
67
22
57
36.7
34
39.1
8
MCQ
GPT--4o
81
45
65
34
37
49.1
72
50
42
33
42.1
EM
Exact Match
43
20
29
9
20
22.8
28
40
5
15.1
10
EM
Machine match
53
31
46
16
32
33.1
61
45
22
27
33.0
11
CoT
CoT-MCQ
83
53
63
39
40
49.3
75
66
41
41
49.3
12
C6T-FM
Machine match
55
30
45
25
29
35.0
67
52
28
29
36.8
InternVL, which maintains relatively low "E" frequency (10%-35%) even difficult tasks.
In summary, for simpler multimodal tasks, the safety profiles of advanced MLLMs and proprictary models are comparable. However, on more complex tasks, proprietary models demonstrate signif- icantly higher level of safety by opting for "E" when uncertain, aligning better with human values by avoiding misleading answers. Given that open-source models currently undergo limited alignment with human preferences, this presents an important direction for future research and development.
44

---

Published as a conference paper at ICLR 2025
Scavestie
Favencaa Cl Charrs
Seidding
Patly Grows
Large Charrss
Reewole
Pndrostiin
Seming
Biiaa Farter
Dvno
Monitorina
Video
Valic
Autt Mntotennesss Driving
General
Gecnarios
Nteet View
DOGI ERONET
Figure 14: Diverse data sources encompassing 15 basic image categories, including text- dense scenes, autonomous driving, video surveillance, natural environments, and more, highlighting wide range of scene diversity.
45

---

Published as a conference paper at ICLR 2025
1000
Prodect
File
119P
Sepee tille Trxt
201
101
OCU Wad
Onjint
1157
Briatienlly R0P
Coteenmen Driviag
Spslii Reiideddi
105
1255
Auttnekk (331)
Romote Sontin
Coter Rengultoe
(Ajer
Reasuming
Poreption
Dizgrars
56
Poodiciin
Viãeo
Table
4018
Muts-Chan Gronding
Fiminy Purrephon
Dosgrem &
Astosoneess Drivie
200
1392
OCR lo The
Video Mestborkee
Mase Voaele Stolie
Tale
158
525
Batorns anstber
Tralfe
250
Neow
10s
HAMD
2312
Figure 15: Detailed information on task distributions across different domains.
TUN
DAAEE
2015
301
3933
2014
Multi-Choice QA (MCQ)
(20) Mi1 vay Ma1 WO MIP" 'rhy 2070", luage Setnet the but the abore mallgaic- qucation hune! the vaage. Ruraond witl
vod the loter 61 ir oummec uplios.
Exact/Machine
(EM)
An whic dety revena usoee 10,00 aoctming the
198 BRARETEN DEALS' diggnnn? chosoo beinn.
'sd) 2821". "180 2012", TC3 2013" 103y 2014" TEU fhr bwage doer ve Swavee masder Fleare
*2002* 1.27% *12022e9 Revense
2202g* Rosasv 1202244 Revesse
Though MCD
fially relect hust axcrer
ceveusr for crendd 10A00 *200pee anrronggoo i lnvert NY3
In sluch your does the resemue firt crcred H6600) according to the BRAKEYEN DEALS diagros?? Thrchorco ure fornd behre:
deterntter when revetu evendd 10,00 accorboy
disoram iala.
up tlee hars reprrsentin cevemae
2025" "O0 MiDe" *Mt) The luagy doar fouoeme custic stop mi feulll snuvee question divectly.
1(4) 2011" "Wy
Though EM
217 2011 1.7tt aus (iis 3835 199s 3004 11,00 elennnr firs cucmnd 10,000
20241x33e lest 2024.
Figure 16: Illustration of input-output formats for various experimental setups. MCQ represents the default multiple-choice question format in the dataset. EM removes the choices to prevent model from relying choice information, instead directly matching the final answer. CoT (Chain of Thought) prompts the model to first perform reasoning before providing the final answer.
46

---

Published as a conference paper at ICLR 2025
Table 24: Maximum Acceptable Resolution of Different MLLMs.
Model
LLM
Max Resolution
Qwen- VL-Chat
Qwen
448
LLaVAI 1.5-BB
Vicuna-7B
336
LLaVA 1.55-3B
Vicuna-1
336
LLaVA-Next
LLama3-8B
672
LLaVA-Next
Qwen-72B
672
m
mPLUG-DocOWl 1.5
LLama-7B
448
ShareGPT4V-7B
Vicuna-7B 7B
336
ShareGPT4V-13B
Vicuna- 13B
336
MiniGPT-v2
Llama 2-7B-Chat
448
Monkey
Qwen-7B
896 1334
Cambrian-1-8B
-Lama3-8B-Instrue
1024
Cambrian-1-333 1-34B
Yermes2-Yi-34B
1024
DeepSeek-VE VI.
DeepSeek-LLLMM-bbbbaaee
1024
YI-VL-34B
Yi-34B-Chat
448
MiniCPM-V 2.5
LLama3-8B
1344
InternLM--Cooppsee2.5
InternL.M27BB
4096
Cog CogVLm2-llama3-Chat
LLama3-8B
1344
Mini- Simi-Gemini-7B-HE
Vicuna-7B-v/
672
Nous-Hermes- Hermes 2- 2-Yi-34B
672
SliME-13B
Vicuna-133
2016
SliME-8B
LLama3-8B
2016
Intern VL-Chat-V1-5
IntemL
4096
InterVVL-2
IntemL.M2.5-7B-Chan
4096
Qwen2-VL
Qwen2-7B
3584
GPT-4o
Private
GPT-4o-mini
Private
Gemini-1.5-pp
Private
Claude 3.5 Sonnet
Private
47


Reviews:
[{"review_id": "ctx9GFcagV", "reviewer": "Reviewer_a7Si", "summary": "This paper introduces MME-RealWorld, a large-scale, fully manually annotated benchmark for multimodal large language models across diverse real-world tasks. The MME-RealWorld has 13366 images with 29429 question-answer pairs annotated by humans. It covers 43 subtasks across 5 real-world scenarios. MME-RealWorld is the largest manually annotated benchmark to date, featuring the highest resolution and a targeted focus on real-world applications. It reports comprehensive results with various leading LVLMs.", "questions": "Overall, the proposed MME-RealWorld is a solid paper. There are some additional comments in Weakness section.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. MME-RealWorld is the largest manually annotated benchmark for LVLMs across 43 subtasks across 5 real-world scenarios.\n\n2. MME-RealWorld has the highest resolution among manually annotated benchmarks. \n\n3. It reveals several insights for current LVLMs. For example, existing models still lack abilities for image detail perception and dynamic information understanding.", "weaknesses": "Overall, the proposed MME-RealWorld is a solid paper. There are some additional comments: \n1. The holistic MME-RealWorld is too large for developing LVLMs. The author should formulate a mini set of MME-RealWorld for community convenience. I suggest maintaining a balance of task types and difficulty levels while reducing the overall size\n2. There exists a clear preference for preparatory APIs for selecting E or refusing to answer the question. Does this mean the preparatory APIs are better aligned with human values and have better AI security? The author may discuss if it is possible to compare preparatory APIs with open-sourced models in a better way. Moreover, I suggest that the authors conduct a detailed analysis of when and why models choose option E or refuse to answer.\n3. MME-RealWorld has a large scale of images and question-answer pairs. The author should discuss if MME-RealWorld is sufficiently diverse or just collecting several similar images and QAs. More comparisons on the distribution of image types, question types, and answer types between MME-RealWorld and other benchmarks are welcomed. \n4. It is better to have a discussion of knowledge leakage LVLMs on MME-RealWorld benchmark, as in [1]. \n\n[1] Are We on the Right Way for Evaluating Large Vision-Language Models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cLk1aK5KR0", "reviewer": "Reviewer_oyYW", "summary": "This paper analyzes the limitations of existing MLLM benchmarks and observes three significant challenges (i.e., small data scale, restricted data quality of model-based annotations, and insufficient task difficulty especially caused by the limited image resolution). To address these issues, this paper constructs a new benchmark named MME-RealWorld that is the largest manually annotated benchmark to date and features the highest resolution and a targeted focus on real-world applications. The authors benchmark 29 advanced MLLMs on the introduced MME-RealWorld, revealing the limitations of existing MLLMs (none of them reach 60% accuracy).", "questions": "Please refer the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The overall paper is well organized and easy to follow. The motivation of constructing the dataset is clear.\n+ This paper contributes a large-scale, large-resolution and manually annotated VQA dataset, which contains 13,366 high-quality images and 29,429 question-answer pairs covering 43 subtasks across 5 real-world scenarios.\n+ Large number of MLLMs are benchmarked on the introduced benchmark, i.e., 4 close-source and 25 open-source MLLMs in total.", "weaknesses": "- Although the dataset contribution is great, it is hard to find some insightful analysis from this paper. The authors only summarize the benchmark results. It would be better to provide more in-depth analysis and highlight the future direction.\n- In Lines 479-480, the authors claim that ‘This indicates that most models’ visual perception modules fail to identify the objects in the images corresponding to our questions.’ This is not clear and convincing. I am wondering how do the authors attribute the higher frequency of ‘E’ outputs to the limited image detail perception of MLLMs? \n- In Lines 481-485, the authors analyze the ‘limitations of MLLMs in understanding dynamic information’ without providing any qualitative or quantitative result. It would be more convincing to provide some evidences to support such statement. \n- Figure 2a shows the different domains and tasks. It would be nice to add a figure to show the distributions of different domains and tasks for better readability.\n- In Lines 93- 94, the authors claim ‘we collect a total of 13, 366 high-resolution images from more than 300K public and internet sources.’ I have doubts about the number of the data sources. 300K is a huge number, and how long does it take to collect the images?\n- For better readability, it would be better to add a paragraph title for the last paragraph in Section 3.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jw5yFyg7Pd", "reviewer": "Reviewer_4Fv1", "summary": "This paper presents a new evaluation benchmark for Multimodal Large Language Models (MLLMs), dubbed MME-RealWorld, which focuses on challenges that models face in the real world. Specifically, MME-RealWorld covers 29,429 question-answer pairs across 5 real-world scenarios. Experimental results on MME-RealWorld show that even the most advanced models still struggled in real-life scenarios. Besides, the authors have also conducted detailed analyses to explain the unsatisfying performance of MLLMs.", "questions": "Do the authors have plans to expand or adapt MME-RealWorld to include new tasks or modalities as MLLMs capabilities evolve?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The perspective of evaluating MLLMs in real-life scenarios, such as OCR in the Wild, Video Monitoring and Autonomous Driving is new and of significant value for practical deployment of MLLMs.\n- The authors have conducted detailed comparisons with existing benchmark in Tab.2, which helps better capture the unique characteristics of MME-RealWorld.\n- The authors have evaluated 24 open-sourced MLLMs and 4 closed-sourced MLLMs, which provide a comprehensive evaluation of current MLLMs.", "weaknesses": "- The evaluation on MME-RealWorld seems to require lots of computation resources, which may limit the accessibility for researchers with fewer resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PUnPBd95ja", "reviewer": "Reviewer_qNEP", "summary": "In this paper, the authors present the MME-RealWorld dataset which stands out for a few key reasons. First, it’s the largest human-annotated benchmark for real-world scenarios with nearly 30,000 QA pairs created by 32 volunteers. The data is high quality, featuring high-resolution images that capture important details and every annotation was double-checked by a professional team. The tasks in this dataset are challenging, reflecting real-world needs that even top models struggle to handle accurately. The authors also include a Chinese section in their dataset with 5,917 QA pairs to avoid translation and cultural issues.\n\nIn addition, the authors evaluate a total of 24 open-source MLLMs (some of which are public APIs) on QA pairs that emphasize perception capabilities, reasoning, and a focus on Chinese. They share insights into the strengths and limitations of current models, showing that even the most advanced ones struggle with these benchmarks, with the top ones achieving high 50s%.", "questions": "Do authors have plans or see it feasible to increase diversity of data sources and tasks? Is the pipeline on data collection, labeling and evaluation scalable? can you easily extend to other languages and data sources? from the current version it looks very manual and not scalable. It would be good to share more insight as extending the dataset to more image sources and languages, even in future, would be great. (please refer to my comments in the previous section)\n\nCan we add more open ended evaluation methods? Moving beyond fixed options, or providing more open-ended tasks without predefined answers, would allow for a deeper evaluation of model understanding.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The dataset is a comprehensive and high-quality collection of realworld QA pairs manually annotated to capture complex details and ensure accuracy. It features high resolution images essential for interpreting information in certain domains like MO, with annotations rigorously cross-checked by small group of professionals. The dataset includes particularly challenging, real-world tasks, where top models struggle to handle, with performance generally falling below 60% accuracy. Additionally, a dedicated Chinese section addresses translation and cultural gaps often seen in other datasets. Finally, an evaluation of numerous open-source models highlights current limitations in handling such complex, real-world scenarios.", "weaknesses": "The dataset is certainly valuable and a step forward compared to existing QA-focused benchmarks. However, the contribution may be somewhat limited for publication in this conference due to a few areas. For instance, the diversity of data sources appears limited, with an overemphasis on specific tasks like autonomous driving (If the goal is to capture embodied understanding, related areas in robotics could also be included to broaden the dataset’s scope) Similarly, the monitoring section could benefit from a more varied range of examples and so on. There is not a discussion on why these areas are chosen over others. While tackling everything can be challenging, my worry is that this dataset will introduce bias toward certain domains while trying to address limitations of other benchmarks. \n\nThe use of multiple-choice challenges, while popular, remains a somewhat limited way to assess model capabilities. A more open-ended evaluation method would provide a richer assessment of model understanding, especially on this complex benchmark. The addition of a fifth option in multiple-choice questions is a positive step, but more could be done to move beyond predefined answers.\n\nAdditionally, the reliance on full human annotation restricts scalability. It would be beneficial if the authors leveraged this benchmark as a foundation to extend to larger datasets and broader domains. \n\nFinally, while English and Chinese are the primary languages in current benchmarks, including more languages would strengthen the dataset’s multilingual utility.\n\nI clearly see the values of the work, however, I think it might be better suited for a dataset track or even a relevant workshop focused on dataset, evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XzLD0QMAML", "reviewer": "Reviewer_FpJY", "summary": "This paper introduces a new benchmark dataset, MME-RealWorld, designed to evaluate the capabilities of Multimodal Large Language Models (MLLMs) in handling high-resolution, real-world scenarios. The dataset addresses limitations in existing benchmarks, such as small data scale, reliance on model-based annotations, and insufficient task difficulty. MME-RealWorld contains over 13,000 high-quality, high-resolution images annotated with 29,429 question-answer pairs across 43 subtasks and 5 real-world scenarios, making it the largest manually annotated benchmark to date. The paper reports the performance of 29 prominent MLLMs on this benchmark, revealing that even the most advanced models struggle to achieve 60% accuracy, indicating significant challenges in perceiving high-resolution images and understanding complex real-world scenarios.", "questions": "- 1) In line 315, the paper says \"InternVL-2 demonstrates the strongest perception abilities, outperforming other closed-source models.\", but Tab.3 shows that Qwen2-VL is the best performing model. So is there a typo here? Please point out if I've misunderstood.\n\n- 2) It is recommended to add the max resolution supported by the baseline model to the table to enhance readability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- 1) The MME RealWorld proposed in this paper has significant advantages in terms of data scale, annotation quality, visual content resolution, language types, task types, and task domain diversity, filling the gaps in existing work. It emphasizes the relevance of benchmarks and the real world, providing a strong and persuasive benchmark for evaluating the visual-language abilities of multimodal agents in real-world application-related scenarios.\n\n- 2) The experimental section of this paper is solid. The authors present rich test results and analysis, providing various statistical indicators to reveal the limitations of existing VLMs in fine-grained image perception and dynamic information understanding, as well as biases of models from different sources in visual-language tasks. This offers valuable insights for improving the performance of VLMs in various application scenarios.", "weaknesses": "- 1) The main challenges of MME RealWorld stem from high-resolution images and complex content. However, the corresponding questions are only focused on image content recognition and simple single-step reasoning, showing limitations in task difficulty and the requirement for understanding capabilities of large models.\n\n- 2) Some methods displayed on the leaderboard are restricted by fixed input resolutions. In high-resolution scenarios, directly resizing input images may result in the loss of information needed to answer questions. Therefore, is the model's error due to the inability to find the correct information from complex image content or because the necessary information was not provided at the input stage? Supplementing such discussions can further enhance the persuasiveness of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)