MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks
Authors: Jiacheng Chen
Tianhao Liang
Sherman Siu
Zhengqing Wang
Kai Wang
Yubo Wang
Yuansheng Ni
Ziyan Jiang
Wang Zhu
Bohan Lyu
Dongfu Jiang
Xuan He
Yuan Liu
Hexiang Hu
Xiang Yue
Wenhu Chen
Venue: ICLR 2025 Poster (2025)
Keywords: evaluation of multimodal large language models
Primary area: datasets and benchmarks

Abstract: We present MEGA-Bench, an evaluation suite that scales multimodal evaluation to over 500 real-world tasks, to address the highly heterogeneous daily use cases of end users.
Our objective is to optimize for a set of high-quality data samples that cover a highly diverse and rich set of multimodal tasks, while enabling cost-effective and accurate model evaluation.
In particular, we collected 505 realistic tasks encompassing over 8,000 samples from 16 expert annotators to extensively cover the multimodal task space. Instead of unifying these problems into standard multi-choice questions (like MMMU, MM-Bench, and MMT-Bench), we embrace a wide range of output formats like numbers, phrases, code, \LaTeX, coordinates, JSON, free-form, etc. To accommodate these formats, we developed over 40 metrics to evaluate these tasks. 
Unlike existing benchmarks, MEGA-Bench offers a fine-grained capability report across multiple dimensions (e.g., application, input type, output format, skill), allowing users to interact with and visualize model capabilities in depth. We evaluate a wide variety of frontier vision-language models on MEGA-Bench to understand their capabilities across these dimensions.

Published as a conference paper at ICLR 2025
MEGA-BENCH SCALING MULTIMODAL EVALUATION TO OVER 500 HEAL-WORLD TASKS
Jiacheng Chen') Tianhao Liang, Sherman Siu, Zhengqing Wang. Kai Wang, Yubo Wang, Yuansheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, Dongfu Jiang, Xuan He, Yuan Liu, Hexiang Hu, Xiang Yue Wenhu Chen"
MEGA-Benche Team
https://tiger-ai- lab habp Bench/
BUUTO
What the main toplo of the article mentloned on the webpage?
the region.
extract the following Information: Movie Title
Identity the the orchestra
Mathematics
using image, the (lastl name the composer.
Information Extraction
Knowledge
Q: List all the needed
Identify country, atate, municipality the photos taken.
Planning
image Image
Science
(A/?
Coding
Generate declarer strategy, taking into account all possible scenarios
Metrics
Python program Images.
Q: Recogniza ongans can appropriately describe the marked areas
Perception
Q: Convert the math
why image
Treck the car and the bounding coordinates each mubssequent frames
G: Can you write story containing
Idantify the posltions keypoints providing thair normliizdd coordinates.
the both of the animale appearing the Individual Images?
Figure 1: MEGA-BENCH contains 505 multimodal tasks with diverse data sources, input/output formats, and skill requirements. The taxonomy tree guides and calibrates the annotation process.
ABSTRACT
We present MEGA-BENCHA evaluation suite that scales multimodal evaluation to over 500 real-world tasks, to address the highly heterogeneous daily use cases of end users. Our objective is optimize for set of high-quality data samples that cover highly diverse and rich set of multimodal tasks, while enabling cost- effective and accurate model evaluation. In particular, tasks encompassing over 8,000 samples from 16 expert annotators to extensively cover the multimodal task space. Instead of unifying these problems into standard multi-choice questions (like MMMU, MMBench, and MMTT-Bench,, we embrace wide range of output formats like numbers, phrases, code, 19TEX, coordinates, JSON, free-form, etc. To accommodate these formats, we developed over 40 met- rics to evaluate these tasks. Unlike existing benchmarks, MEGA-BENT offers a
collected 505 realistic
Core Contributors, Contributed equally. See the Author Contribution Statement for details. x
jca348 @ cfu wenhuchen @ uwaterloo.ca

---

Published as a conference paper at ICLR 2025
fine tne-grained capability report across multiple dimensions (e.g.. application, input type, output format, skill), allowing users to interact with and visualize model ca- pabilities in depth. We evaluate a wide variety of frontier vision-language models on MEGA BAIGGA-BENHHH to understand their capabilities across these dimensions.
INTRODUCTION
Large foundation models (OpenAI, 2023; 2024a; Anthropic, 2024a; Google, 2023; Meta, 2024; Al- ibaba, 2024) have dramatically transformed the landscape of artificial intelligence by showcasing exceptional capabiliities across various tasks and domains. Originating in the realm of natural lan- guage processing, these models have progressively expanded to perceive and interpret multimodal information, including single images, multiple images, and videos. Previously, multimodal models were mainly used for standardized tasks like image captioning (Lin et al., 2014), video caption- ing (Wang et al., 2019), and visual question answering (Antol al., 2015: Goyal et al., 2017; Xiao et al., 2021). With the recent progress on multimodal alignment, these models have shown great potential to solve many diverse and complex tasks with well-designed prompts. As result, people have applied them to assist with many realistic tasks like "web navigation" (Koh et al.. 2024). "game playing" (Valevski et al., 2024). "travel planning" (Xie et al., 2024), "visual navigation" (Wang et al., 2023a), "sports analysis" (Xia et al., 2024), "visual entity recognition" (Hu et al., 2023), "vi- sual quality assessment" (Ku et al., 2024), and more. These efforts have significantly increased the utility of multimodal models.
An important challenge is identifying how to accurately gauge the abilities of these vision-language models (VLMs) across wide range of tasks. Most existing benchmarks are designed to cover only one or a few similar tasks, making them inadequate for evaluating the models' overall capabilities. The status quo is to evaluate the model many cxisting benchmarks to showcase their all-round abilities. For example, Qwen2-VL. was evaluated on 27 image and video benchmarks in total. Al- though this massive evaluation effort provides valuable insights into how well these models handle specialized tasks, it also introduces a significant overhead and several challenges:
Limited Output Diversity: The existing multi-task benchmarks like MMMU (Yue et al., 2024a), MMT-Bench (Ying et al., 2024) rely heavily on multiple-choice questions to lower the burden of evaluation. This fails to evaluate the generative abilities of these multimodal models.
Lack of Task Coverage: The existing benchmarks are often sporadic and lack a systematic design to cover the multimodal task space. Certain abilities are not well covered in the current ecosystem. Consequently, even exhaustively testing all the available benchmarks would not be sufficient.
Expensive Inference Cost: The full evaluation process is expensive regarding computation cost/- time API expense. Since many examples or tasks are similar in the capabilities they assess (e.g.. VQA Mathew et al. (2021) alone has thousands of examples for examining doc understanding and OCR-related abilities), overly repetitive evaluation at a large scale leads to resource waste.
- Unmanageable Setups: Each benchmark has complexities when setting up the evaluation. For example, VQA (Goyal et al., 2017) alone has four splits (val, dev-test, std-test, and test). It is hard to track the exact setup of different baseline models to ensure a fair comparison.
To address these challenges, we advocate for unified protocol that scales up multi-modad evaluation to maximize the task coverage and the diversity in model outputs while optimizing the inference cost. As initial attempt, we propose MEGA-BENCH which is designed to provide comprehensive and systematic assessment of multimodal foundation models.
To build MEGA -BENCH,, we first construct task laxonomy tree that organizes different multimodal tasks based on the application type (Figure 1), with significant effort spent adjusting and refining the taxonomy tree to ensure sufficient coverage and diversity. The task taxonomy tree then serves the guiding principle to ensure all relevant tasks and skills are covered and appropriately balanced. To help the annotators create their tasks, we build an annotation GUI to simplify the process of creating the task JSON files and web tool to visualize the results of the VLM'' responses alongside the ground truth. We also review each task contribution when it is first submitted, after evaluating the models on the new tasks, and periodically throughout the annotation process to ensure that all of the tasks are novel and high-quality. This collaborative effort resulted the compilation of 505 realistic
M/Qwen2-VI.
2

---

Published as a conference paper at ICLR 2025
tasks, effectively covering (almost) the entire multimodal capabiliiy space a manageable inference cost. To facilitate nuanced and precise evaluation, we also developed 45 highty-customized metrics tailored to these tasks during the annotation process.
Unlike existing benchmarks that often provide a single score, MEGA- BENCH offers a fine-grained capability report based multiple dimensions such as the input type, input format, output format, and required skills. This interactive and visualizable report enables users to identify the models' performance across several orthogonal dimensions, uncovering strengths and weaknesses that might be obscured in aggregate scores. Such detailed analysis is invaluable for researchers and developers aiming to enhance foundation models and optimize them for specific downstream applications.
Using MEGA-BENCH: we conducted comprchensive studies of popular flagship and efficiency models (with both open-source software and proprictary APIs) and identified some findings below:
1. Among flagship models, Claude 3.5 Sonnet (1022) and GPT-4o (0513) currently lead in perfor- mance across wide range of multimodal tasks, with less than 0.1% difference in their overall scores. Our detailed breakdown shows that Claude 3.5 Sonnet excels in planning and math with its latest upgrade bringing clear boosts in processing Ul/Infographies inputs, while GPT-4o leads in information extraction and knowledge-intensivy tasks.
2. Among open-sourced models, Qwen2-VL performs the best, with its performance near the top close-sourced flagship models, and outperforms the second best open-soures model by =10%
3. Among efficiency models, Gemini 1.5 Flash is the strongest model overall, except for the tasks related to handling User Interfaces and Documents.
4. Proprietary models can effectively leverage Chain-of-Thonn Thought (CoT) prompting to improve their performance, while open-source models hardly produce helpful reasoning processes. In our evalua- tion results, 10 of 13 open-source models get worse results with CoT prompting.
RELATED WORK
Multimodal benchmarks. Benchmarking in vision-language models has been long-standing research problem. Before the era of large multimodal models, most benchmarks were designed for specific tasks or skills. Some benchmarks like VQA (Antol et al., 2015), GQA (Hudson & Manning, 2019), and ViswizVQA (Gurari et al., 2018) focus on photograph or natural images. ChartQA (Masry et al., 2022), InfoVQA (Mathew et al., 2022), Doc (Mathew et al., 2021), and OCR-VQA (Mishra et al., 2019) focus more on documents, infographics, and other similar me- dia. Later on, there was a trend to build more well-rounded benchmarks to cover a wider range of skills topies, such as ScienceQA (Lu et al., 2022), MMBench (Liu et al., 2023b), MMMU (Yue et al., 2024a;b), MMT-Bench (Ying et al., 2024), and more. However, due to the diversity of these different tasks, most benchmarks use multiple-choice questions for all problems. Therefore, these benchmarks cannot fully reflect the generational abilities of multimodal models. Complementary to this, LMsys arena (Chiang al., 2024) and WildVision arena (Lu et al., 2024) have proposed to user voting and Elo-ranking to benchmark multimodal models. Our benchmark is the first to scale up the tasks by significant magnitude. Furthermore, our benchmark provides a breakdown report to analyze multimodal models across multiple dimensions.
Sensitivity of large model leaderboards to input format. Creating reliable leaderboards poses a substantial challenge for evaluating large models. Previous studies have noted that LLMs exhibit sensitivity to minor input modifications, including prompts and in-context examples in few-shot settings (Sclar et al., 2024: Chang & Jia, 2023). To mitigate input sensitivities, researchers have de- veloped specialized prompt design and prompting- -aasd training approaches (Liu et al., 2023a; Jain et al., 2024b). Nonetheless, for benchmarks that only allow multiple-choiee format (Wang et al., 2024d), studies by Zheng et al. (2024) and Robinson et al. (2023) find the option sequencing can significantly alter model rankings on the leaderboard. Recently, Alzahrani et al. (2024) explores the advantage of a hybrid scoring method to stabilize models' leaderboard rankings over input format. Though MEGA-BENCH does not include hybrid scoring for each individual task, the overall use of diverse and hy brid scoring methods and output formats across more than 500 tasks demonstrates the robustness of the benchmark.
3

---

Published as a conference paper at ICLR 2025
4-5
Ethical Safety
Photographs 28%
Multiple- choice 17%
Planning & Decision
Open-ended 16%
6-8
10%
7%
8%
Commonsense Social
Interface 18%
2%
6% Videos 8%
Contextual Formatted 20%
Numerical 10%
9+
4%
Domain Knowledge
Math Logical
109
OCR
137
Exact Structured 22%
Single Image
Spatial & Temporal
152
Documents 16%
Infographics 20%
62%
Language Parsing & Gen.
Scene Understanding
154
Object Recognition
(1) Input Format
(2) Output Format
Visual Input Number
(4) Skills
four keyword dimensions and the task-level statistics. The diversity along various dimensions enables fine-grained capability analysis.
Figure 2:
MEGA-BENCH
MEGA-BENCH is a comprehensive multimodal benchmark that spans 7 input formats, 6 output formats, 10 different types of skills, and varying number of visual inputs, whether single-image, multi-image or from video, as shown Figure 2. Our benchmark covers distinct subject areas in hierarchical taxonomy evaluate VLMs' ability tackle various tasks.
3.1 BENCHMARK CONSTRUCTION PROCESS
Preparation. Figure illustrates our annotation process. the conceptualization stage, we propose a "draft" task taxonomy tree with the top two levels of Figure by getting inspirations from existing multi-task or multi-discipline LLM/VLM benchmarks (Srivastava et al., 2022; Liu et al., 2023b; Yue et al., 2024a). The first level consists of general applications like "perception", "planning", "reason- ing". etc., while the second level has more concrete meta-tase tasks like "document understanding". "app function understanding". "logic reasoning", etc. We host brainstorming session add exemplars under each second-lews level node and write descriptions about the number and quality of the tasks we expect. Based our empirical observations of how general users use VL.Ms in real-world scenar- ios, we assign more task budgets to perception, knowledge, and information extraction than other first-level nodes while strictly monitoring the application-Ieve distribution balance in the annota- tion process. We then distribute the second-level nodes in the "draft" tree to the annotators. This top-down framework minimizes overlaps between annotators and facilitates overall organization.
Texonomy
Tosk covellintruutionn Your Yourszak is onclyge the query image <boxcel, y1.x2, y2-booo
Perception
Plaming
Rassoning
Add Hoco Task
Logie
Mathh
group
Understancing
Action
Add tasks
Questorr What the Incatin
Quastien Wharle the locedin...
Add tanko
Irategrum
Shoppeng
Elamplle
Foomple
Figure 3: The annotation process of MEGA-BENCH, We propose a "draft" taxonomy tree and then distribute the second-level nodes annotators. We allow the annotators gradually refine the tree structure as they add new tasks. Fach task has many examples and shared task-level instruction. Each example has a question and ground truth answer.
To ensure reliable commitment and annotation quality, we call up 16 designated expert annotators with rich LLM experience and computer science backgrounds. All annotators are graduate students or above with majors in computer science, elecroniex/ommmnication engineering, bio-statistics, or finance, and 12 of them served as annotators or authors of LLM/VLMM benchmarks published at top conferences. The annotators can 1) refine the "draft" taxonomy by adding/deleting nodes, 2) add "task group" nodes and then add series of tasks under that, and 3) directly add tasks under an existing bigh-level node. We develop tools to facilitate the annotation process, including 1) an in- teractive annotation tool that defines the annotation format and automatically unifies all annotations into JSON files, 2) a GitHub repository to coordinate the task submission, reviewing, and discussion process, which was inspired by BIG-bench (Srivastava et al., 2022), and 3) visualization tool that

---

Published as a conference paper at ICLR 2025
allows annotators to browse the existing tasks and the evaluation results of representative vision lan- guage models (VLMs). We coordinate all the annotators to ensure they understand our expectations and continuously improve our tools. Please see $B for complete details of annotation protocols.
Task annotation. The annotation process contains two rounds. The annotators submit tasks to the benchmark by creating pull requests (PR) to the main branch of our GitHub repository. In the first round of the annotation process, we ask the annotators to contribute 20 tasks following the principles below ensure the quality of the task:
Data source and output format: Creative tasks with diverse data sources and output formats are encouraged. If the data was collected from existing datasets, we ask annotators to adapt the original annotation into more specific questions and design more diverse answer formats.
Number of examples: Each task should have at least 15 examples. Exceptions are allowed for some complicated tasks where the data are scarce.
Documentation: Each task should be accompanied by documentation that indicates the source of the data, the capabilities the task tries to evaluate, and the evaluation metric to be used.
Our core contributors review each PRs carefully to provide feedback, and the accepted PRs are merged into the main branch. We periodically run the evaluation with commercial VLMs (e.g., GPT- 40) and update the results of existing tasks on our visualization page, which allows the annotators to better understand the difficulty of their tasks and catch potential glitches in the annotation. We found that this helps significantly improve the annotation quality.
Before the second round of annotations, the core contributors review all tasks in the taxonomy tree and investigate the biases in the task distribution. We then host another annotator session to propose new meta-tasks to balance the distribution and maximize the coverage. We then distribute the updated tree nodes to annotators and employ the same guidelines to finish the second-round annotations. After this round, each annotator contributes at least another 30 tasks.
Quality control and refinement. We leverage commercial VLMs to examine the task quality. Con- cretely, we gather the results of GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro and compute an average score on each task. Tasks with almost 1.0 scores often have trivial questions (based on manual inspection) and can hardly distinguish the ability of different models. We ask the corre- sponding annotators to investigate and augment those tasks. For tasks with almost zero scores, the task reviewers audit them carefully and remove them if the zero score comes from incorrect an- notations or insufficient instruction contexts. Finally, the benchmark contains a total of 505 tasks with roughly 8,200 examples, which is large enough to minimize the sample variance within each high-level taxonomy node. Please refer to $4.3 for analysis of the number of examples per task.
3.2 METRICS FOR ANSWERS IN DIVERSE OUTPUT FORMATS
To properly evaluate the tasks with different output formats, we develop set of highly-customized evaluation metrics in parallel with the benchmark construction process ($3.1). Figure 4 shows sev- eral examples of the model outputs along with the task's associated metrics. When new tasks submitted to our GitHub repository, we implement any new metrics specified by the task authors. We use two types of metrics: rulc-based metries and LLM-assisted metrics. All metrics are normalized into [0. 1J. with 1.0 being the full mark.
Rused metries. When there is a unique answer under the question context or the correctness of the answer can be verified by rules (e.g., if the generated story/poetry meets the desired formats or if the generated code can pass test cases), we implement rule-based metrics for evaluation. To satisfy the needs of all tasks submitted by annotators, we end up with suite of over 40 rule-based metries. Robust string parsing is also implemented to extract the answer from the model's s response. We conduct sanity check to ensure the correct implementation of rule-based metrics. Specifically, we create an "oracle" model that always returns the ground truth, then compute its score over all tasks evaluated by rule-based metrics. The sanity check is passed when the "oracle" model gets full 1.0 score. See $D.4 for details.
LLM L-M-iiiiteed metries. For open-ended tasks that do not have unique answer, instead employ an LLM-assistte metric (Zheng et al., 2023; Li et al., 2023a). We design a ter- evaluation prompt template and fill in the tailored evaluation criteria for each task. The LIM is instructed to compare
5

---

Published as a conference paper at ICLR 2025
Output Type: Stractured
Output Type: Structured
Output Type: Structured
Ostgut Type Contretual Frematted
"Jssice
Anesser:
Ansver:
[Aggieisigg X3 mi..
#26185785
rnturn al runn uu gni apterdaau pudtcadd e2, et, af, a.l. straumled)
sale deck bpsdl Audeci DL
(1 Phili ZJ-- [ind,
357846192
the dut si us sgii Aemmor
and hepific Adh
(ay (earpfiL
(yme lolon Phg
196257246
Mull astull qaiii Daundeei dulect qus
(XU), verpoi endiallign"
(XLU verphi
785414629
241629738
Metric Program Judge
Metric Symbolic Planning
Metric LaTeX Expe Equaality
Metric: Simple String Match
090 Symbelic I' Planning (Barman)
0.0 LaTeX Complex Formula Convection
O
Code Programming Text Jeavy)
Sudoku
Output Type: Numerical
Output Type: Numerical
Outpet Type: Structured
Output Type: Open-ended
Andew: 20 was ot (N2AAR))
Answeer
Poot leal the 25 spenit ttl. Seath mise spados avd daasoed))
E SRL (oeee is 431
(9,02, 0.32, 0.12, 0,42) (R.00, 0.32, 0.70, (.42). (0.00, 0.32, 0.10, 0.42).
Art dud 'ou Att -lid mis
13 737056 431, (0.70uu (575555)
(ertourr 55.7
South vms br.... in cnly thet vecsiretion svr disuree swittesing canttol derini
chib pyllped
Ait old
Metric: Sequence Coordinates Similarity
Metric Sequence
Melzic Dict Equality
Metric: GPT-4o as Juddge
600
800
e @
Animal Pose Estimation
Multi-camera Car Tracking
Mensa IQ Test
Bridge Strategies (Werldeloss)
Figure 4: Representative examples of MEGA-BE -BENCI''s diverse output formats and customized metrics (input queries are omitted). The outputs are extracted from real responses of GPT-4o (0513). We implement robust parsing to extract the final answer from raw model responses.
the model response with the reference answer and assign a score from 1 to 10. The score is then normalized into [0. 1] to be consistent with the other metrics. See $D.3 for details.
We divide the tasks into two subsets based on the different evaluation processes. The Core Set is evaluated with rule-based metrics to make the evaluation fast and cost-free. The Open-Ended Set evaluated with metrics that use an LLM--asa-juudee, where the evaluation pipeline calls proprietary LLM over an API. Specifically, we use GPT-40-0806 (OpenAI, 2024a) as the judge LLM while maintaining an extensible implementation for using other judge models. The Core and Open-Ended sets contain 440 and 65 tasks, respectively.
3.3 MULTI-DIMENSIONAL KEYWORDS FOR FINE-GRAINED ANALYSIS
Existing multi-task multimodal benchmarks analyze models according to dimensions like the image type and academic discipline (Yue et al., 2024a), ability (Liu et al., 2023b), or meta-tast ( (Ying et al., 2024). MEGA-BENCE offers broad and diverse range of coverage across all these dimensions, and extends even further beyond them. As explained in 13.1, the taxonomy tree divides the tasks into general application scenarios, the most manageable dimension for distributing the annotation efforts to different annotators. After we collected all tasks and finished the quality control process, we grouped all tasks based on four extra dimensions: input visual type, input visual number, output format, and required skills (Figure 2). Each dimension has 6 to 10 keywords, enabling fine- -g-iindddedd analysis and comparison. Interactive visualization tools can then be developed based on our eval- uation results, which allows model developers to delve deep into different aspects of a model and compare different models comprehensively.
3.4 DATASET STATISTICS AND COMPARISON WITH OTHER RENCHMARKS
MEGA-BENCH contains 505 real-world tasks with 186 manually annotated repurposed sam- ples. Even for repurposed data, considerable effort is needed to convert the original annotations into specific task descriptions, diverse output formats, and additional instructions to include auxiliary information about formatting. Figure 2 shows the task distribution of all five dimensions, and the detailed task taxonomy tree and statistics of each dimension in Appendix C.
Table compares MEGA-BENCHE to existing multimodal benchmarks. The key feature of our benchmark is the diversity across all aspects, driven by our high-level designs of diverse task appli-

---

Published as a conference paper at ICLR 2025
Table A comparison between BENGH and existing works. BEGG-BBENC has a greater
diversity in data sources, input/output format, the number of metrics, and the number of tasks.
Dataset
Annotation
Source
Input
Output
#Metrics #Tasks
VQA-v2 (Antol et al., 2015)
New
Photo
Image
Phrase/Bool/Num
VizwizVQA (Gurari et ul., 2018)
New
Photo
Image
Phrase/Bool/Num
1
ChartQA (Masry ct al., 2022)
New
Chart
Image
Bool/Num
1
A12D (Kembhavi ct al., 2016)
New
Diagram
Image
Multi-choice (MC)
1
GeoQA (Chen et al., 2021)
New
Geometry
Image
Multi-choice (MC)
1
NLVR (Suhr & Artzi, 2019)
New
Photo
Images
Bool
1
IQA (Mathew et ul., 2022)
New
Infographics
Image
Phrase/Bool/Num
1
VoA (Mathew et al., 2021)
New
Document
Image
Phrase/Bool/Num
I
OCR-VQA (Mishra al., 2019)
New
Book covers
Image
Phrase
1
ScienceQA (Lu et al., 2022)
New
K12 Books
Image
Multi-choice (MC)
26
Vath (Lu et al., 2023)
Repurposed
Diverse
Image
MC Num
MMBench (Liu et al., 2023b)
Hybrid
Diverse
Image
Multi-choice (MC)
20
MME (Yin et al., 2023)
Repurposed
Existing
Image
Multi-choice (MC)
14
Seed- Bench (Li et al., 2024c)
New
Existing
Image/Video
Multi-choice (MC)
12
VisIT-Bench (Bitton et al., 2023)
Hybrid
Diverse
1/2 Images
Free-form (FF)
70
MMStar (Chen et al., 2024a)
Repurposed
Existing
1 Image
Multi-choice (MC)
18
MM-Vet (Yu et al., 20244b
Repurposed
Existing
1 Image
Free-form (FF)
16
MMMU (Yue et al., 2024a)
New
Diverse
21 Image
MC/ / FF
30
MUIRBench (Wang et al., 2024a)
Hybrid
Existing
> Image
Multi-choice (MC)
12
MileBench (Song et al., 2024)
Repurposed
Existing
> Image
MC/ / FF
12
VideoMMB (Fu al., 20244)
New
Youtube
Video
Multi-choice (MC)
30
MVBench (Li al., 2024e)
Repurposed
Existing
Video
Multi-choice (MC)
20
MMT-Bench (Ying et al., 2024)
Repurposed
Existing
21 Image/Video
Multi-choice (MC)
162
MEGA-BENCH
New
Diverse
21 Image/Video
Unrestricted
45
505
cations and output formats. MMMU (Yue et al., 2024a:b) focuses on college-level exam questions with various discipline and image formats. All questions are single-image and answered in multiple- choice format, MMT-Bench (Ying et al., 2024) covers 162 concrete sub-tasks, enabling in-depth analysis based on their "laskonomy" and diverse input forms. However, all of the tasks MMT-Bench are from existing datasets, mostly under the "Perception" sub-tree in our taxonomy, and all outputs in multiple-choics form like MMMU. To maximize task coverage and the diversity in model outputs with cost-effective inference, MEGA-BENCE includes a much broader range of task types and output formats, while having fewer total samples compared existing benchmarks.
EXPERIMENTS
We evaluate 19 VLMs with multi-image support on MEGA-BENCCE $4.1 describes the evaluated models and the evaluation pipeline. presents the evaluation results with fine- tne-grained analytical breakdown. $4.3 provides analyses on the number of examples per task and error types.
4.1 EVALUATION SETTINGS
Evaluated models. We evaluate diverse range of large multimodal models. The proprietary models assessed include GPT-4o (0513) and GPT-4o mini (OpenAI, 2024a), Seaude-3.5-Somme (0620 and 1022) (Anthropic, 2024a;b), Gemini-1.5-Pro (002) and Gemini-L.5-Flash (002) (Google, 2024a). For open-source models, we mainly focus on large flagship (>70B parameters) and small-to-medium efficiency models. The large models include Qwen2-WL-T3 VL-72B (Alibaba, 2024), ViemVVLL2-Llmm3-77666 (Chen et al., 2024d), LLaVA-One Visionn-77 (Li et al., 2024a), and NVLM (Dai et al., 2024). The medium-scale models comprise Qwen2-VI--7B (Alibaba, 2024), Pixtral 12B (Mistral, 2024), Aria (Li al., 2024d), InterVL2--8B (Chen al., 2024d), Phi-3.5- Vision (Abdin et al., 2024), MiniCPM-V24, (Yao al., 2024), LLaVA-One Vision-7BB (Li et al., 2024a), Llama-3.2-1 1B Meta (2024), and Srefies3-BB-Laanee (Laurençon et al., 2024).
has diverse and flexible formats. To ensure the models have clear instructions the output format, provide all evaluated VLMs with one-shot in-context example. For each query, we fill in a pre-defined prompt template with the task instructions written by the task annotators, the 1-shot example, and the concrete query question. Since this one-shot
Evaluation pipeline. MEGA-BENT

---

Published as a conference paper at ICLR 2025
The main results of different models on the Core and Open-ended subset of MEGA- BENCH, with 440 and 65 tasks, respectively. We report the macro mean scores across all tasks in each set. The overall score is the weighted average of the Core and Open-ended scores. When computing the overall score, we use the higher Core score from 'w/o CoT and "w/ CoT"
Table 2:
Core (rule eval)
Open-ended Overall (GPT eval)
Model
Eval Tier Open Source
w/o CoT w/ CoT
-laude-3.5-Somee (1022) (Anthropic, 2024b)
Flagship
49.20
52.59
65.63
54.27
GPT-4o (0513) (OpenAI, 2024a)
Flagship
52.03
52.65
64.78
54.21
Claude-3.5 (0620) (Anthropic, 2024a)
Flagship
48.80
50.41
63.74
52.13
1.5imi-1.5-PPr--012 (Google, 2024b)
Flagship
46.99
48.22
58.58
49.55
(Google, 2024b)
Efficiency
No
41.90
41.89
56.91
43.82
GPT-4o mini (OpenAI, 2024b)
Efficiency
No
39.85
40.77
58.65
43.07
Qwen2-VL-72B (Alibaba, 2024)
Flagship
Yes
46.41
45.42
56,40
47.70
I 76B (Chen et al., 2024d)
Flagship
Yes
35.02
35.63
51.93
37.73
LLaVA-OneViiioon-77 (Li et al., 2024a)
Flagship
Yes
31.99
29.74
45.99
33.79
NVLM-72B (Dai et al., 2024)
Flagship
Yes
24.21
21.59
34.78
25.57
Owen2-VL-788 (Alibaba, 2024)
Efficiency
Yes
34.80
32.93
43.96
35,98
Pixtral-12B (Mistral, 2024)
Efficiency
Yes
31.91
31.36
45.66
33.68
Aria-MoE-25B (Li ct al., 2024d)
Efficiency
Yes
30.49
28.90
51.03
33.13
I (Chen et al., 2024d)
Efficiency
Yes
25.96
24.09
39.79
27.74
Phi-3.5-Vision-4B (Ahdin et al., 2024)
Efficiency
Yes
23.27
23.00
39.48
25.36
MiniCPM-V2.6-8B (Yao et al., 2024)
Efficiency
Yes
22.88
22.96
41.73
25.38
LLaVA-One Vision-7B (Li et al., 2024u)
Efficiency
Yes
22.41
21.36
33.98
23.90
Llama-3.2 2- IB (Meta, 2024)
Efficiency
Yes
10.04
16.00
31.73
18.02
Idefics3-8B Baass33-B-llamaa Llama3 (Laurengon al., 2024)
Efficiency
Yes
11.12
8.96
32.11
13.82
example's primary purpose is to illustrate the output format, we allocate it a tiny portion of the total image budget. For each model, conduct experiments with and without Chain-of-Thought (CoT) prompting (Wei et al., 2022) for the Core tasks. Full evaluation details are in $D. Our default eval- uation pipeline focuses on models with multi-image support. To properly evaluate models trained mainly for single-image use cases, we create a single-image setting using the single-image tasks of MEGA-Bench. See SA for the detailed results and analyses of the angle-image setting.
4.2
MAIN RESULTS WITH BREAKDOWN ANALYSIS
Table 2 presents the main evaluation results, while Figure 5 and Figure 6 are the fine- grained break- downs enabled by MEGA MEGA-BENCH'S BENCH's 'ss dilli-iimensoonal diversity. This section discusses some im- portant findings, and the full breakdown results are in $E. We organize the evaluated models into two tiers: (1) The Flagship Model Tier compares the strongest performing models from each model's organization, (believed) with #params 70B. (2) The Efficiency Model Tier compares efficiency models from each model's organization, (believed) with #params < 20B.
Flagship models. Unlike the results recent benchmarks like MMMU-Pro (Yue et al., 2024b) where GPT-4o (0513) and Claude-3.5-Sonnc (0620) get close scores, GPT-4o (0513) outperforms Claude-3.5-Sone (0620) with a clear margin on MEGA-BENCH (> 2%). Investigating the break- down results, we observe that GPT-4o (0513) wins in most applications/skills except for coding, math, and planning-relate tasks, where the answers are typically "structured" output format ( Figure 5). The recent update of Claude-3.5-Sonme
(1022) makes improvements across almost all inputs, and
dimensions, especially planning tasks and those with infographic/UU//hhooggapp slightly surpasses GPT-4o in the overall score (< 0.1 1)). The "planning" application keyword con- tains tasks like symbolic planning (Zhu et al., 2024), navigation (Ku et al., 2020), chess games (Fu et al., 2024b), puzzle games (e.g., maze, Sudoku), etc., and even the best models get low scores.
One typical observation of Claude-3.5 3.5-onnet models is that they tend to be meticulous and refuse to answer routine knowledge commonsense questions, such the name and nationality of famous actors. The bottom radar maps show that they fall behind in knowledge, information extraction, and commonsense reasoning compared GPT-4o, partially because of this refusal behavior.
The evaluation results suggest that Qwen2-VL VL performs particularly well amongst open-source models of similar parameter sizes. In Figure 5, Qwen2-VL--2B gets similar closed-source
8

---

Published as a conference paper at ICLR 2025
Contexual
Structurae"
Fomgatted
Videgd
ArtsiCrettive
0.08
Open-ended
0.08
infographics
GPT-4o (0513)
Claude-3.5-Sennes (1022)
Claude-3.5-Senee
Documents
Photographs
Geminn-11.-PPr-000
Numental
Muitiple- choice
IntemVL2-768
Common Reason ing
Qwen2 VL-728
Science
OCR
Domain Knowlodge
Llava OneVision-728
Coding
Planning
Spatial/leerpprr
al Reasening
0.76
0.76
0.59
0.59
RtiicalSsafety
nno extaation
0.08
0.08
Scenee Understapding
Perception
Parsing/Gien
PlanninolDecis Makingg
edge
MaUl-Logical Reasoning
Metrics
OBffect Recognition
Math
Figure 5: Fine-grained breakdown analysis of flagship models on four dimensions. From top-left to bottom-right: input format, output format, skills, and application.
related
Science
Coding
Videos
0.65
0.65
0.50
0.50
Info Extraction
0:24
Arts/Creative Planning
TT
Ul related
0.02
0.02
Knowleddee
Perception
Infographics
Documents
Metrics
Math
Photographs
4o mini
Pixtral 12B
InternV.2.6 888
Phi- Vision
Gemini-115-llesh-000
Qwen2-VL-78
Llava-OneVision-7e
MiniCPM-V2.6
Figure 6: Fine-grained analysis of efficiency models on input format (left) and application (right).
models in the general perception category and outperforms Gemini 1-5-inn-----Pr-0022 on information ex- traction tasks. Llava-Onee One Vision-7225 scores very low when the visual inputs are in "UI related" and "Document" formats while performing well on video inputs. This suggests a lack of OCR and language parsing abilities, which can confirmed with skills radar plot.
Efficiency models. Figure 6 analyzes the results efficiency models. In general, Gemini-1.5 flash-002 has the best performance with exceptional scores Science and Metrics applications. The Metrics keyword contains tasks such rating the quality of GenAl results (He et al., 2024; Jiang et al., 2024b) and requires deep multimodal reasoning and commonsense. However, its performance on Ul-related inputs and information extraction tasks falls behind GPT-4o mini.
Chain-of-Thoughhe An interesting finding that the CoT prompt (See $D) effectively guides all proprietary models generate detailed reasoning process, and flagship-tier proprietary models all obtain better performance on the Core set. However, has almost no effect on most open- source models. For example, the Qwen2-VI. I1.ernVI.2, and LLaVA-One Vision models rarely

---

Published as a conference paper at ICLR 2025
GPT-4o (0513)
Claude-3.5-5onnet
0.56
0.56
0.54
0.54
Mega-Bend Macro-Mean Score
0.52
0.52
Lack of Knowledge
0.50
0.50
11.0%
0.48
0,48
Lack of Reasoning Capability 48.0%
3
5
9 111315
5 7 9 111315
GeSPPo-1.S-Pro
Owen2-VL-728
Perception Error
0.52
30.3%
0.48
Fail to Follow Instructions
0.46
0.48
6.3%
0.44
0.46
0.42
0.44
Refuse Answer
Annotation Glitch
1315
7 9 111315
2.0%
2.4%
Examples per Task
Figure 7: (Left) The bootstrap distribution of benchmark scores (w/ CoT prompting) with a gradually increased bootstrap sample size of the number of per-task examples. (Right) The task-wise error distribution of GPT-4o (0513) over a subset of 255 Core tasks.
produce reasoning with a CoT instruction, and sometimes get confused about the required format after generating the reasoning process, leading lower score on the Core set.
Some open-source models get comparatively low scores for their parameter count (e.g., Llama-3.2 11B, Idefics3). One typical difficulty is leveraging the one-shot example to understand the output format, which is alleviated with CoT because the prompt provides extra instructions on the output format beyond the one-shot example. The lack of multi-image ability another bottleneck of these models, and $A presents single-image setting of MEGA- BECCI with further analyses.
4.3 MORE ANALYSIS
Number of samples per task. As discussed in $1, one of MEGA- goals is optimizing the inference cost while producing comprehensive breakdown analyses, making us prioritize the task diversity over examples per task in the benchmark construction process. To understand the robust- ness of the benchmark score with 15 examples per task, we obtained bootstrap distributions (Efron & Tibshirani, 1994; Hesterberg. 2011) of the model scores for our Core set with COT. We did this by taking a random subset of the model's responses of size n (n 5,.... 15) with replacement for each task and calculating the lasel macro-mean scores. To ensure the bootstrap distribution was numerically stable, we ran 10,000 Monte Carlo simulations. Figure (left) shows that the variance in model scores rapidly narrows as the number of examples per task increases. As the number of examples per task increases beyond 7, the marginal return in variance reduction diminishes.
Error analysis. To understand the limitations of tate-of-the-art VLMs, we analyze the GPT-4o (0513) results by manually identifying the error types over a subset of 255 tasks from the Core set. We use the CoT setting since the reasoning process helps determine the eπror type. Figure (right) presents the error distribution. For GPT-4o, the lack of various reasoning capabilities (e.g.. symbolic reasoning for planning/coding tasks, spatial or temporal reasoning for complex perception tasks, etc.) is the dominating failure mode on MEGA-BENCH Please refer to $F for the full definition of error types and detailed example-wise inspection results with different models.
CONCLUSION
This paper presents MEGA-BENCHI a comprehensive benchmark that scales multimodal evalua- tion to over 500 real-world tasks but at a manageable inference cost. By systematically organizing tasks across dimensions like skill, output format, and input type, we enable fine- ine-grained analysis of multimodal models. Our evaluation of state-of-the-ar VLMs revealed significant performance vari- ations between models that previously seemed similar. MEGAA-EENH provides new standard for multimodal evaluation, offering robust analysis tool for model development.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We would like to thank Vector Institute for their support throughout this research. We also express our gratitude Tung Vu from GreenNode for providing access GPUs, which were instrumental in running some of the evaluation experiments. Additionally, we appreciate the contributions of all open-source language model providers, whose efforts have significantly propelled the advancement of research in this field.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly language model locally on your phone. ArXiv preprint, abs/2404.14219, 2024. URL http:://araivvorg//////440 /244.
Alibaba. Qwen2-vl: To the world more clearly. https///weeeinnggiihhhhhhhgevvve 2024.
Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan AlRashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq. M Saiful Bari, and Haidar Khan. When benchmarks are targets: Revealing the sensitivity large lan- guage model leaderboards. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13787-13805. Association for Compu- tational Linguisties, August 2024.
Anthropic. Claude 3.5 sonnet. https///wwwnnntrgp....ooo////e//
http://www.aattnpiiccccooooooooe
2024a. URL
sonnet..
Anthropic. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, 2024b. URL https: https://www.anttrrpic..coo///e//
m modelo ls and wwwputer- use.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zit- nick, and Devi Parikh. VQA: visual question answering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 2425 2433. IEEE Computer Society, 2015, doi: 10. N0.110//CCV.2215.2277 URL htwps:tts////e / ICCV.2015.277
Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin- Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffusion transformers for 3d camera control. arXiv preprint
11X1 2024.
Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Ziqiang Liu, Junting Zhou, Tianyu Zheng, Xincheng Zhang. Nuo Ma, Zekun Wang, et al. Coig-cqia: Quality is all you need for chinese instruction fine-tuning, 2024.
Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gard- ner, Rohan Taori, and Ludwig Schimdt. Visit-bench. A benchmark for rision-language instruction following inspired by real-world use. arXiv preprint arXiv:2308.06599 2023.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621-11631 2020.
Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler-Lussier, and Ningchuan Xiao. Mapqa: A dataset for question answering choropleth maps. arXiv preprint arXiv:2211.00m 1.08555. 2022.
Ting- Yun Chang and Robin Jia. Data curation alone can stabilize in-context learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 8123-88 21448844. As- sociation for Computational Linguistics, July 2023.

---

Published as a conference paper at ICLR 2025
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint MtXiv2205..45517 2021.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403 2024a.
Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li. Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, et al. :mmbnchh A comprehensive multimodal evaluation benchmark towards general medical ai. arXiv preprint Xiix:2448.0036611 2024b.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint 1909:1909.02111 2019.
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Guicourse: From general vision language models to versatile gui agents, 2024c.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, al. How far to gpt-4v? closing the gap to commer- cial multimodal models with open-source suites. ArXiv preprint, abs/2404. 16821, 2024d. URL
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena: An open platform for evaluating lims by human preference, 2024.
Tai-Yi Yin Chiu, Yinan Zhao, and Danna Gurari. Assessing image quality issues for real-world prob- lems. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
François Chollet. On the measure intelligence, 2019. URL https:///rxxivorr//hh 1911.01547.
Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang. Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nylm: Open frontier-class multimodal 1lms. arXiv preprint, 2024.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference computer vision (ECCV), pp. 720-736, 2018
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 326-335, 2017.
Rocktim Jyoti Das, Simeon Emilov Hristov, Haonan Li, Dimitar Iliaanvv Dimitrov, Ivan Koychev, and Preslav Nakov. Exams-v: A multi-discipline multilingual multimodal exam benchmark for evaluating vision language models. arXiv preprint arXiv:2403. 10378, 2024.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint 11X.v.2499.17146. 2024.
Li Deng. The mnist database of handwritten digit images for machine learning research, 2012.
Bradley Efron and Robert Tibshirani. An introduction to the bootstrap. Chapman and Hall/CRC, 1994.
12

---

Published as a conference paper at ICLR 2025
Yue Fan, Jing Gu, Kaiwen Zhou, Qianqi Yan, Shan Jiang, Ching-Chen Kuo, Xinze Guan, and Xin Eric Wang. Muffin or chihuabua? challenging large vision- lsionnlanguage models with multipanel vqa. arXiv preprint arXiv:2401.15847, 2024.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075, 2024a.
Deqing Fu, Ruohao Guo, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. IsoBench: Benchmarking multimodal foundation models on isomor- phic representations. In First Conference on Language Modeling (COLM), 2024b. First four authors contributed equally.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng. Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishma, Blink: Multimodal large language models can see but not perceive. ArXiv preprint, hts/2404.12390, 2024c. URL 2209.12390.
Daniel Geng and Andrew Owens. Motion guidance: Diffusion-bel based image editing with differen- tiable motion estimators. arXiv preprint 18Xiv:2401.18085 2024.
Zahra Gharaee, ZeMing Gong, Nicholas Pellegrino, Iuliaa Zarubiieva, Joakim Bruslund Haurum, Scott Lowe, Jaclyn McKeown, Chris Ho, Joschka McLeod, Yi-Yun Wei, et al. A step to- wards worldwide biodiversity assessment: The bioscan- Im insect dataset. Advances in Neural Information Processing Systems, 36, 2024.
Google. Gemini: family of highly capable multimodal models. ArXiv preprint, abs/2312.11805, 2023. URL httpp::/aaxxi..ogg/////// hbtprg//abb/211.118/81
Google. Updated production-ready gemini models, reduced 1.5 pro pricing, increased rate http://lww...coom//////aaaatttta wwt.g/goollooooooo//////ooe///ee
limits, and more.
2024a. URL https:
//devel Lerevelopperssee ooteeveeloorrrggoogllllooooo.o hwt.://owo..cc/o//o/////eoe////e wpdatedd peppdaated-produucce production- ready-
models - reduced models-reducedd 15 1e55-pro-pricinn increased rate- limits more.
Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
ArXiv preprint, abs/2403.05530, 2024b. URL https:://arxiv.t httpr///abb/20..5555/5/
Yash Goyal, Tejas Khot, Douglas Summers-Stua Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 6325 6334. IEEE Computer Society, 2017. doi: 10. 1 1..T1///VPP.2117 10.11011111 (0009//CVR.22100 201.6.201.660.
URL www.hhtt::://// / /ttp::::///oooo///////// 10.
Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Infinity-mmi Scaling multimodal performance with lcrge-scale and high-quality instruction data. arXiv preprint MxXi:22410.185555 2024.
Haisu Guan, Huanxin Yang, Xinyu Wang, Shengwei Han, Yongge Liu, Lianwen Jin, Xiang Bai, and Yuliang Liu. Deciphering oracle bone language with diffusion models, 2024. URL https: //a //arxiv.org//a/ / 2tt///44660000 006684.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
Xuan He, Dongfu Jiang, Ge Zhang, Max W.F. Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, Kai Wang, Quy Due Do Yuansheng Ni, Bohan Lyu, Yaswanth Narsupalli, Rongqi Fan, Zhiheng Lyu, Yuchen Lin, and Wenhu Chen. Videoscore: Building automatic metries to simulate fine- fine-grained human feedback for video generation. ArXiv, abs/2406. 15252, 2024. URL 270688037.
13

---

Published as a conference paper at ICLR 2025
Lukas Helff, Felix Friedrich, Manuel Brack, Kristian Kersting, and Patrick Schramowski. Llava- guard: Vlm-based safeguards for vision dataset curation and safety assessment. arXiv preprint arXiv:2406.05113, 2024.
Jack Hessel, Ana Marasović, Jena Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. Do androids laugh at electric sheep? humor" understanding" benchmarks from the new yorker caption contest. arXiv preprint S1li:222090062990 2022.
Tim Hesterberg. Bootstrap. Wiley Interdisciplinary Reviews: Computational Statistics, 3(6):497- 526, 2011.
Xin Hong, Yanyan Lan, Liang Pang, Jiafeng Guo, and Xueqi Cheng. Visual reasoning: From state to transformation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):11352- 11364, 2023.
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal. Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming- Wei Chang. Open-doman visual entity recognition: Towards recogniz- ing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12065-12077 2023.
Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6700-6709. Computer Vision Foundation / IEEE, 2019. doi: 10. 10.1109/CVPR.2019.000
HuggingFaceM4. Doematix dataset, 2024. URL httpp::/hugginggaac...o/tttsssss HuggingFacceM4DDccatti
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024a. URL 2403.67974.
abs/
Neel Jain, Ping ych Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. NEFTune: Noisy embeddings improve instruction finetuning. In The Twelfth International Conference on Learning Representations, 2024b.
Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippp Thiran. Funsd: dataset for form under- standing in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pp. 1-6. IEEE, 2019.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Man- tis: Interleaved multi-image instruction tuning. ArXiv preprint, abs/2405.01483, 2024a. URL https: / rxiv.oviioooooo www.o/org//ooooo abs//b///// 222450004822
Dongfu Jiang, Max Ku, Tianle Li, Yuansheng Ni, Shizhuo Sun, Rongqi Fan, and Wenhu Chen. Genai arena: An open evaluation platform for generative models. arXiv preprint arXiv:2406.04485, 2024b.
Yiqiao Jin, Minje Choi, Gaurav Verma, Jindong Wang, and Srijan Kumar. Mm-soc: Benchmarking multimodal large language models in social media platforms. ArXiv preprint, abs/2402.14154, 2024. URL https::tpppp..pppm. http://///axxxx///a//aa/o/
/20..2242..442 14155.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2901-2910, 2017.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visual- izations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018.
14

---

Published as a conference paper at ICLR 2025
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kädár, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Computer Vision-ECCV 2016: 14th European Conference. Amsterdam. The Netherlands, October 11-14. 2016, Proceedings. Part IV 14, pp. 235-251. Springer, 2016.
Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang. Md Rizwan Parvez, and Shafiq Joty. xcodeeval: large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval, 2023. URL http:://aaxvv.or//aaa /bbb//// 2303.03004
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neu- big, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating multi- modal agents realistic visual web tasks. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 881 381-905, Bangkok, Thailand, 2024. Association for Computational Linguistics. URL https:tppp... htt////laantthtoo
Lantho www.hhology.rrrg222222
long.50.
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lec. Beyond the nav- graph: Aision-and-langure navigation in continuous environments, 2020. URL https://
htt.orrg/ab//0044/ 2004.02857. 02857.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-Across-Room Multilingual iision-add-languae navigation with dense spatiotemporal grounding. Conference on Empirical Methods for Natural Language Processing (EMNLP), 2020.
Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. Im- agenhub: Standardizing the evaluation of conditional image generation models. arXiv preprint arXiv:2310.015%, 2023.
Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. VIEScore:
Towards ex- plainable metrics for conditional image synthesis evaluation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12268 12290.0 Bangkok, Thailand, Au- gust 2024. Association for Computational Linguistics. doi: 10.116653/1/2222.alll 663. URL https: htap:::///allae httpa/aaaatoooooy.ooooooooooolat orraaatthoooooyyog///aaaeeee 20t. long.663. 663.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on rescarch development in information retrieval, pp. 104, 2018.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better under- standing lision-language models: insights and future directions. ArXiv preprint, abs/2408. 2024. URL http:ttt////iiiorrrrg abs
/ 2408. 12637.
Benjamin Charles Germain Lee, Jaime Mears, Eileen Jakeway, Meghan Ferriter, Chris Adams, Nathan Yarasavage, Deborah Thomas, Kate Zwaard, and Daniel S Weld. The newspaper nav- igator dataset: Extracting headlines and visual content from 16 million historic newspaper pages in chronicling america. In Proceedings of the 29th ACM international conference on information & knowledge management, pp. 1065--002,, 2020.
Jiyoung Lee, Seungryong Kim, Sunok Kim, Jungin Park, and Kwanghoon Sohn. Context-aware emotion recognition networks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10143-10152, 2019.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. ArXiv preprint, 0as/2408.033224 2024a. URL Ut..https://arxiioorr//bbs
03326.
15

---

Published as a conference paper at ICLR 2025
Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Benchmarking multimodal large language models with text-rich visual comprehension, 2024b. URL
abs/2404.16790.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang. Ruimao Zhang, and Ying Shan. Sced-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13299-13308, 2024c.
Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993, 2024d.
Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2024e.
Mukai Li, Lei Li, Yuwei Yin, Masood Ahmed, Zhenguang Liu, and Qi Liu. Red teaming visual language models. arXiv preprint 1tt::2401.129155 2024f.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of cnntructton-followwee models. https: // //gi
lab/ a lpaca eval, 5 2023a.
Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille. Super-clevr: A virtual benchmark to diagnose domain robust- ness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14963-14973, 2023b.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCH 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal Ilms evolved in web page understanding and grounding? Conference on Language Modeling, 2024a.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. ACM Comput. Surv., 55(9), Jamuary 2023a. ISSN 0360-0300. doi: 105/145/3560815. URL
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et Mmbench: Is your multi-modal model an all-around player? ArXiv preprint, abs/2307.0622 2023b. URL https: hwtpt::t//atttttet /ttp://///avvv./////////// abs/ 230.
Yuan Liu, Zhongyin Zhao, Ziyuan Zhuang, Le Tian, Xiao Zhou, and Jie Zhou, Points: Improving your vision-language model with affordable strategies. arXiv preprint arXiv:2409, 140.:2409.04888,, 2024b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pp. 3730-3738, 2015.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song- Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:21 1111111.1111 13214, 2021.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
16

---

Published as a conference paper at ICLR 2025
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. ArXiv preprint, abs/2310.02255, 2023. URL https: / //arxiv.org/ant
Yujie Lu. Dongfu Jiang, Wenhu Chen, William Wang, Yejin Choi, and Bill Yuchen Lin. Wild- vision arena: Benchmarking multimodal Iims in the wild, February 2024. URL https: //huggingface..coom
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Vidco-chatgpt: Towards detailed video understanding via large vision and language models. arXiv:2306.05224 2023.
Xiaofeng Mao, Yuefeng Chen, Yao Zhu, Da Chen, Hang Su, Rong Zhang, and Hui Xue. Coco-o; A benchmark for object detectors under natural distribution shifts. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6339-63550 2023.
U-V Marti and Horst Bunke. A full english sentence database for off-line handwriting recognition. In Proceedings of the Fifth International Conference Document Analysis and Recognition. ICDAR' (Cat, No. PR00318), pp. 705-708. IEFF, 1999.
Ahmed Masry, Do Xuan Long. Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa document images. In Proccedings the IEEE/CVF winter conference on applications computer vision, 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawa- har. Infographicvqa, Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022.
Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https:/ai.metta.co/bbee come -obiee- 2024. URL https: /ttp://://a.imaaa.... wa.. cww/blog////ccccoooooo Llam 3 2 connect 2024 visi-vvisonn edde-- mobile-devices
Anand Mishra, Shashank Shekhar, Ajeet Kumnr Singh, and Anirban Chakraborty. Oer-vqa: Visual question answering by reading text images. 2019 international conference on document analysis and recognition (ICDAR), pp. 947-952. IEEE, 2019.
Mistral. Announcing pixtral 12b. https:/miittalla//eesssppaaaa lasttralaiineessppii
2024. URL https://
12b/.
Linyong Nan, Chiachun Hsich, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojeiech Kryšcifiski, Hailey Schoelkopf, Riley Kong, Xiangru Tang, al. Fetaqa: Free-form table question answering. Transactions of the Association for Computational Linguistics, 2022.
Teng Niu, Shiai Zhu, Lei Pang, and Abdulmotaleb El-Saddik. Sentiment analysis on multi-view
social data. In MultiMedia Modeling, pp. 15-27, 2016.
Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, al. Docci: Descriptions of connected and contrasting images. arXiv preprint 2XXx:2244411955. 2024.
OpenAl. Gpt-4v(ision) system card, 2023. URL httpss//cchh.opena..ccm///pppeee
GPTV.System.Carddpff
OpenAl. Hello gpt4-o.
2024a.
URL https://
Openai..omm/nneex/eellloo- -pttelllo--ppppp000
17

---

Published as a conference paper at ICLR 2025
OpenAl. Gpt-4o mini: advancing cost-efficient intelligence. 40 ww--min-advvaciing
2024b. URL https://opppnii..ccoommmbobooooo www.eeeftttiiieeeeeaeeeeee----
g
Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, et al. Vibe-eval: A hard evaluation suite for mcasuring progress of multimodal language models. arXiv preprint arXiv:2405.0228, 2024.
Panupong Pasupat and Percy Liang- Compositional semantic parsing on semi-structured tables. arXiv preprint 1Xxv:1508.000003 2015.
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Ba- narse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36, 2024.
Obioma Pelka, Sven Koitka, Johannes Rückert, Felix Nensa, and Christoph M Friedrich. Ra- diology objects in context (roco): a multimodal image dataset. Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis: 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 3, pp. 180-189. Springer, 2018.
Sai Raj Kishore Perla, Yizhi Wang, Ali Mahdavi-Amiri, and Hao Zhang. Easi-tex: Edge-aware mesh texturing from single image. ACM Trans, Graph., 2024.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu, Tim Z. Xiao, Katherine M. Collins, Joshua B. Tenenbaum, Adrian Weller, Michael J. Black, and Bernhard Schölkopf. Can large language models understand symbolic graphics programs?, 2024. URL https::/arxxv.org// 2408.08313
Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. arXiv preprint arXiv:2407.0588. 2024.
Jonathan Roberts, Kai Han, and Samuel Albanie. Satin: A multi-task metadataset for classifying satellite imagery using lasioaelagguage models. arXiv preprint arXiv:2304.11619 2023.
Joshua Robinson, Christopher Michael Rytting, and David Wingate. Leveraging large language models for multiple choice question answering. In The Eleventh International Conference on Learning Representations, 2023.
Stefan Romberg, Lluis Garcia Pueyo, Rainer Lienhart, and Roelof Van Zwol. Scalable logo recogni- tion in real-world images. In Proceedings of the 1st ACM international conference multimedia retrieval, pp. 1-8, 2011.
David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, et al. Cvqa: Culturally-diverse multilingual visual question answering benchmark. arXiv preprint arXiv:2406.05967, 2024.
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sen- sitivity to spurious features in prompt design or: How i learned start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations. 2024.
Silvia Sellán,
Chen, Ziyi Wu, Animesh Garg, and Alec Jacobson. Breaking bad: A dataset for geometric fracture and reassembly. Advances in Neural Information Processing Systems, 35; 38885-38898, 2022.
Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. Annotating objects and relations in user-generan rer-geeerated videos. In Proceedings of the 2019 on International Conference on Multimedia Retrieval, pp. 279-287. ACM, 2019.
18

---

Published as a conference paper at ICLR 2025
Zehong Shen, Huaijin Pi, Yan Xia, Zhi Cen, Sida Peng. Zechen Hu, Hujun Bao, Ruizhen Hu, and Xiaowei Zhou. World-grounded human motion recovery via gravity-view coordinates. arXiv preprint arXiv:2409.0660 266626669.0666 2024.
Haojun Shi, Suyu Ye, Xinyu Fang, Chuanyang Jin, Layla Isik, Yen-Ling Kuo, and Tianmin Shu. Muma-tom: Multi-modal multi-agent theory of mind, 2024. URL https://arxiv.
abs/2408.12574
Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Benchmarking milms in long context. arXiv preprint arXiv:2404.1855 18532, 2024.
Khurram Soomro, Amir Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. ArXiv, abs/ 2012.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alone et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
Alane Suhr and Yoav Artzi. NIvr2 visual bias analysis. arXiv preprint arXiv: 10411, 2019.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv:181 1100099 2018.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal Ilms. ArXiv preprint, abs/2406.16860, 2024. URL
http::///aaxvoor///////44 16860.
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408. 14837, 2024.
Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wemxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi- image understanding. ArXiv preprint, abs/2406.09411, 2024a. URL https: //txxiv.//rxx..rrgg/ abs/2406.09411.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions Machine Learning Research, 2023a.
Jing Wang, Weiqing Min, Sujuan Hou, Shengnan Ma, Yuanjie Zheng, Haishuai Wang. and Shuqiang Jiang. Logo-2k+: A large-scale logo dataset for scalable logo classification. In Proceedings of the AAAI Conference Artificial Intelligence, 2020.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring mul- timodal mathematical reasoning with math-vision dataset, 2024b.
Shengkang Wang, Hongzhan Lin, Ziyang Luo, Zhen Ye, Guang Chen, and Jing Ma. Mfc-bench: Benchmarking multimodal fact-checking with large vision-language models. arXiv preprint arXiv:2406.1 1288, 2024c.
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scien- tific problem-solving abilities of large language models. arXiv preprint arXiv:2307. 230...55.0.1 2023b.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for sidco-and-language research. In Proceedings of the IEEE/CVF international conference computer vision, pp. 4581-4591, 2019.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. ArXiv preprint, abs/2406.01574, 2024d.
19

---

Published as a conference paper at ICLR 2025
Zhengqing Wang. Jiacheng Chen, and Yasutaka Furukawa. Puzzlefusion++: Auto-agglomerative 3d fracture assembly by denoise and verify. arXiv preprint aiXiv:2406.00259, 2024e.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et Chain-of-thought prompting elicits reasoning large language models. Advances in neural information processing systems, 35:24824-248871 2022.
T. Weyand, Araujo, B. Cao, and J. Sim. Google Landmarks Dataset v2 A Large-Scale Bench- mark for Instance- Level Recognition and Retrieval. In Proc. CVPR, 2020.
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. arXiv preprint arXiv:2405.09711 2024.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting, Advances in neural information processing systems, 34:22419-22453 2021.
Yang Wu, Shilong Wang, Hao Yang, Tian Zheng, Hongbo Zhang, Yanyan Zhao, and Bing Qin. An early evaluation of gpt-4v (ision). arXiv preprint arXiv:2310.16534 2023.
XAL. Grok-1.5 vision preview 2024. URL WKKLhttps:/xxai/bllg//// 1.5v
Haotian Xia, Zhengbang Yang. Yun Zhao, Yuqing Wang. Jingxi Li, Rhys Tracy, Zhuangdi Zhu, Yuan-fang Wang. Hanjie Chen, and Weining Shen. Language and multimodal models in sports: A survey of datasets and applications. arXiv preprint arXiv:240.112252 2024.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9777-9786, 2021.
Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal I1m logical reasoning 2497704477.....
benchmark in visual contexts, 2024. URL
Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang. and Ziwei Liu. Funqa: Towards surprising video comprehension. arXiv preprint 1rXiv:2306.14899 2023.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents. In Forty-first International Conference on Machine Learning. 2024.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-
grained image understanding. arXiv preprint 11Xiv:1901.06706. 2019.
Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang. Wider face: face detection bench- mark. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5525-5533. 2016.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, al. Miniepm-v: gpt-4v level milm on your phone. ArXiv preprint, abs/2440.01180 2024. URL
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey multimodal large language models. ArXiv preprint, Abs/2306.13544 2023. URL https:
/ //aaxiv.org/abl aaaaa//...... 2306.13499. 13549.
Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprchensive multimodal benchmark for evaluating large llnionnl models towards multitask agi. arXiv preprint arXiv:2404 1...:2444..0006 2024.
20

---

Published as a conference paper at ICLR 2025
Hang Yu, Yufei Xu, Jing Zhang, Wei Zhao, Ziyu Guan, and Dacheng Tao. Ap-10k: A benchmark for animal pose estimation in the wild. arXiv preprint arXiv:2108. 12617, 2021.
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu,, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405. 17220, 2024a.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. MM-vet: Evaluating large multimodal models for integrated capabilities. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scar- lett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 57730-57754, PMLR, 2024b. URL https: prw.eedinggs..ooo .m wwt.prrees/233.. 2ttSss//235//224...1tt yu24o.htm.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xiang Bai. Syntax-aware network for handwritten mathematical expression recognition. arXiv preprint arXiv:2203.01601. 2022.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi- dulti-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024a.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Ming Yin, Botao Yu, Ge Zhang, et al. Mmmu-pro: more robust multi diseiiddsecpline multimodal understanding benchmark. arXiv preprint arXiv:2409.022 02211, 2024b.
Xiaoxue Zang, Lijuan Liu, Maria Wang, Yang Song, Hao Zhang, and Jindong Chen. Photochat: A numan-human dialogue dataset with photo sharing behavior for joint image-tex modeling. arxiv preprint 21X.:21088.015... 2021.
Hanlei Zhang, Hua Xu, Xin Wang, Qianrui Zhou, Shaojie Zhao, and Jiayan Teng. Mintrec: A new dataset for multimodal intent recognition. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 1688-1697, 2022.
Yiming Zhang, ZeMing Gong, and Angel Chang. Multi3drefer: Grounding text description to multiple 3d objects. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15225-15236. 2023.
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph Gonzalez. and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL htt.ttt:////xxi..rrt. abs/2306.05685.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 40(6)): 2017.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, 2021.
Wang Zhu, Ishika Singh, Robin Jia, and Jesse Thomason. Language models can infer action seman- ties for classical planners from environment feedback. arXiv preprint arXiv:2406.02791, 2024.
21

---

Published as a conference paper at ICLR 2025
Table of Contents in Appendix
A Single-image Setting: Results and Analyses
23
Details of Annotation Protocols
25
B.1 The unified annotation format
25
B.2 General task collection and creation guidelines
................................
25
B.3 Tools for coordinating annotation and quality control
27
Taxonomy Tree and Multi-dimeenional Keywords
28
C.I Details of the Taxonomy Structure
28
C.2 Statistics of each keyword dimension
32
D Evaluation Details
34
D.1 Prompt template
................................
34
D.2 Model query details
34
................................
D.3 LLM-assisted metrics
................................
35
D.4 Rule-based metrics
36
D.5 Answer extraction from model response
Complete Multi-dimensional Breakdown Results
39
E.1
Breakdown results on the skill dimension
39
E.2
Breakdown results on the input format dimension
40
E,3
Breakdown results on the output format dimension
E,4
Breakdown results on the application dimension
42
E,5
Breakdown results on the visual input number dimension
................................
43
Detailed Inspection of Model Behaviours MEGA A- -BENHH
44
G Detailed Task Information
66
Author Contribution Statement
110
22

---

Published as a conference paper at ICLR 2025
A SINGLE-IMAGE SETTING: RESULTS AND ANALYSES
Table 2 in the main paper focuses on models with multi-image support. However, some open-source models are only trained with single images. To provide feasible evaluation setting for these models, we create a single-image (SI) setting using the single-image tasks in MEGA-BENCI,, containing 273 and 42 tasks from the Core and Open-ended sets, respectively.
Evaluation setup. The Chain-of- Chain-of-Thought (CoT) prompting is used for Core SI tasks. To make the entire query contain only one image as needed by some single-image models, we drop the image input of the 1-shot demonstration example ("X demo im" column in the table). In this case, the 1-shot example only demonstrates the output format, which necessary for inferring the correct answer. For those models already evaluated in Table 2, we calculate the task-level average scores on single-image tasks to obtain the "V demo im" results. Compared to Table 2, 3 single-image models are evaluated and added: Molmo-72B-099 (Deitke et al., 2024), Nolmo-7B-D-0924 (Deitke et al., (Liu et al., 2024b).
2024), and POINTS-QWen2..5
Evaluation results. Table presents the evaluation results of the SI setting. The Core and Open- ended scores of the standard setting (with CoT prompting) are also in the table for reference. Some observations from the table are listed below:
Single-image tasks are easier than multi-imaae tasks in general, and all models get higher scores in the SI setting than in the standard setting.
GPT-4o has the best overall SI score, slightly higher than Claude 3.5 Sonnet (1022). Interest- ingly, GPT-4o mini overtakes Seemini-1.5-Flashee 1.5-Flash has pretty stable performance across different numbers of image inputs.
under the SI setting, suggesting that Gemini-
NVLM-72B (Dai et al., 2024) has much better scores in the SI setting than in the standard setting, suggesting its training data might only contain single or few images.
Comparing the "V demo im" and "X demo im" results of open-source models, the image input in the 1-shot demonstration example is not well utilized by the models to better understand the task logic. Including the image input the demonstration example makes the results much worse for models like Llama-3.2-11B.
More detailed breakdown results available on our project page and the leaderboard (hosted with Hugging Face Spaces)
23

---

Published as a conference paper at ICLR 2025
Table 3: The single-image (SI) setting results of MEGA-BENCH. The Core set evaluation uses Chain-of-Thougee (CoT)) prompting. The "demo img" means the image input of the 1-shot demon- stration example. The "V demo im" directly takes the single-image subset average from the full results in Table The "X demo im" means the 1-shot demonstration example only demonstrates the output format, and the entire query has single image. We report "V demo im" alone for the proprietary models because they have good multi-image support. For open-source models, we do additional evaluations with the "X demo im" setting and use it to compute the overall score.
Core SI
Open SI
Overall SI
Model
Core
Open
/ demo im demo im
/ demo im X demo im
GPT-4o (0513) (OpenAI, 2024a)
$2.65
55.30
64.78
66.00
56.73
Claude-3.5-Sonme (1022) (Anthropie, 2024h)
52.59
54.63
65.63
67.64
56.36
Claude-3.5-Somee (0620) (Anthropic, 2024a)
50.41
52.03
63.74
64.80
53.73
Memini-1.5-Pro-eee (Google, 2024b)
48.22
49.14
58.58
$8.15
50.34
GPT-4o mini (OpenAI, 2024h)
40.77
44.31
58.65
59.56
46.32
Gemini- 1--haii-.--- (Google, 2024b)
41.89
43.48
56.91
57.87
45,40
Qwen2- 11-12-VV--200 VL-72B (Alibaba, 2024)
45.42
47.31
47.31
56.40
58,50
$5.10
48.34
(Chen ct al., 2024d)
35.63
39.32
39.99
51.93
55.33
55.47
42.05
Molmo-72B-0949 (Deitke et al., 2024)
36.48
44.66
37.58
NVLM-72R (Dai et al., 2024)
21.59
31.19
32.99
34.78
48.67
44.69
34.55
LLaVA-One V2B (Li et al., 2024a)
29.74
31.77
31.26
45.99
46.12
44.26
32.99
Qwen2-VL-7H) (Alibaba. 2024)
32.93
35.04
35,39
43,96
45.87
45.17
36,69
Pixtral-12B (Mistral, 2024)
31.36
34.87
34.37
45.66
44.03
44.17
35.68
Aria-MoE-25B (Li ct al., 2024d)
28.90
31.67
31.79
51.03
50,92
$1.37
34.40
InternVL.28B (Chen et al., 2024d)
24.09
27.19
27.65
39.79
40.94
39.39
29.21
Phi-3.5-Vision-44B (Ahdin et al., 2024)
23.00
25.72
25.61
39.48
44.61
42.72
27.89
POINIS-QWeen25-775 (Liu al., 2024b)
25.51
30.32
26.15
MiniCPM-V226-8B (Yao et al., 2024)
22.96
23.82
23.23
41.73
42.54
43.61
25.95
LLaVA-One VB
al., 2024a)
21.36
22.70
23.68
33,98
36.44
38.71
25.69
Qwen2-VL--BB (Alibaha, 2024)
20.88
24.16
22.78
31.54
30.59
35.09
24.43
Molmo-7B-D (Deitke al., 2024)
20.98
35.70
22.95
Llama-3.2-11BB (Meta, 2024)
16.00
17.34
20.79
31.73
34.29
38.61
23.17
Aquila-VL-2B-llvaaaaaa (Gu al., 2024)
16.00
16.98
20.77
24.57
24.58
31.47
22,20
InternVL2-2B (Chen et al., 2024d)
13.14
13.83
12.07
23.86
24.28
28.52
14.26
Idefies3-8B Jamaa (Laurençon et al., 2024)
8.96
9.13
8.94
32.11
33.25
32.31
12.06
24

---

Published as a conference paper at ICLR 2025
B DETAILS OF ANNOTATION PROTOCOLS
This section presents additional details of our task annotation pipeline and protocols, providing complete details for $3.1 of the main paper.
B.1 THE UNIFIED ANNOTATION FORMAT
Figure 8 presents the annotation format designed and used in our annotation process. All annotated tasks share this unified structure, including task instruction, optional global media to provide context to all the questions (typically used in retrieval-related tasks). Additionally, each specific example contains distinct media path(s), a concrete question, and an answer with a single or multiple answer fields. Multi-field answers are organized JSON structures.
annotation.json
Task instruction:
$tak_jnnstrutffe
Global media:
Iggobal_ media>, <global_ .....]
Example
Example
Media paths:
[<example_ment media>> sexample essapplemedia>> ...
Answers:
onswerr_ieldd> <answer>
<answer fieldd_ <answer>
Example question:
_qeetiion
Example
Example
Figure 8: The structure of our task annotation format, which helps coordinate all task annotators and standardize the annotation format.
Our evaluation pipeline follows this format to convert the task information into concrete queries and feed them to the evaluated model. Based on this format, we establish an interactive annotation tool to ensure the tasks submitted by all annotators have the correct and unified format. Figure demonstrates the GUI of the annotation tool.
B.2 GENERAL TASK COLLECTION AND CREATION GUIDELINES
This subsection provides more detailed annotation guidelines for our annotators, complementing the descriptions in $3.1.
Data source of each task. There no restriction on the data source as long as the annotator follows
the copyright and license requirements of the original data. Below are three typical task types and their data sources:
(1). The task designed entirely by the annotator, and the anmotator looks for the image or video resources from the Internet using code/simulator;
25

---

Published as a conference paper at ICLR 2025
MEGA-Banch Talk Arnctation
to VCGA-derch Annetation Toxl.
Giobal Desci plan::
1. First the fLaad Faider
5 L.rrcesa.pm png
afficice
conetetions a0aze
L pmg
are rombers. Select offesde player the given candidatios,
wute glast vestr art oor.
7. p..poross..
see chirk nalcoel to castiin
Puyncesso png
processs png
Biompie of
Selected Iragens
information
There snctial
Zoomed image
for the carrent cxmpple and
LNG
ViSIOAAAR
iiiEOAT
aree
ouarpie" create
Lyrocees png
Terts asing yeur and
Nest
Add New
Remove Current
3.4 fore bag the "AdI besr buitee
Example Guassore
ansradion reskits lete, 1nd Tolcer.
Deletn Fiels
soca-Ja riow if a toa
upcets image
Add Sual Contect Fiidd
Remooe Fail Content Fiedd
videoplay win dow, type la weit
Fald
Reaste:
Metiic:
Weigh:
Global
Daimpl
Barmash
Figure 9: screenshot of our GUT annotation tool.
(2). The task is inspired by existing benchmarks or datasets. The annotator collects the raw im- age/video data from existing datasets but does use the original annotation. The annotator designs/repurposes the data by writing concrete task descriptions and creating new questions and answers, or using scripts to re-process the data for the designed task.
(3). The task directly converted from existing benchmarks or datasets. The annotator randomly samples subset from the existing benchmark, directly using its image/video and the annotation without redesign.
In our annotation process, the first two task types encouraged. The task reviewers strictly control the number of the third type and reject the task an annotator submits many tasks of the third type. Table 18 shows the detailed data source of all tasks in MEGA-BENCH
Output format and answer uniqueness. We aim to cover diverse output formats MEGA- BENCH. Therefore, always require the task annotators consider adapting the original dataset's answer format, especially avoiding unnecessary multiple-choice questions (many MCQs are unnatu- ral and mainly for evaluation convenience). Notably, the annotator must provide sufficient context in the task description and per-example question so that the range of the correct answer is manageable and the task can be evaluated with a clearly defined metric.
Metric specification. When creating a task, the annotator must specify the corresponding evaluation metric. Since the metric implementation in parallel the task construction process, described in $3.2, our GUI annotation tool (Figure 9) allows annotators to choose from existing metrics for each answer field of the task and assigns different weights to each field. When the desired metric is unavailable, the annotator chooses an "unsupported" metric type and writes down detailed metric specifications in the pull request. Our core contributors periodically check the needs of new metrics and implement them.
Documentation. When submitting the pull request, the annotator must write README documen- tation for each task. If the desired metric has not been implemented, the documentation should contain the specification described in the last point. Furthermore, the doe should record the data source (c.g., the Web, existing dataset, etc.) and brief descriptions of the task. These descriptions are instrumental in helping the core contributors assign various keywords to the task and creating Table 18 show the details of all tasks.
26

---

Published as a conference paper at ICLR 2025
B.3 TOOLS FOR COORDINATING ANNOTATION AND QUALITY CONTROL
As described in $3.1, we have two additional tools for coordinating the annotation process and maintaining the data. We present the details in this subsection.
The GitHub repository for task organization. We created a private GitHub repository for con- structing MEGA-BENCE The repository's 's main branch is protected, and all task submissions must go through pull requests (PRs). The core contributors serve as the task reviewers and discuss with task annotators in the pull request forum ensure the task conforms our data collection guide- lines ($B.2). The code of our evaluation pipeline, including the model query and score computation, maintained in the same repository. The core contributors submit pull requests to support different VLMs and add new evaluation metrics, and these PRs are cross-reviewed by other core contributors.
We also actively use the repository's Issues forum to report bugs annotation or metric implemen- tation so the corresponding contributors can get notified and work on the fix. At the end of the annotation process, our repository has 685 pull requests and 40 issues. 277 out the 685 PRs are
for task submission, indicating that many annotators submit task groups with more than one task in each PR. Other PRs are mainly for the evaluation pipeline and bug fixing.
postry Jalka
(a). List of existing tasks
(b). Task overvlew
Ollen gnd pooe
Suy
Wiven
Ml 24 Ligh
Tiarre 10), muter near, sumh, Ne*
(c). Per-example model response and evaluation results
Figure 10: Illustrations of our task visualization page.
Task visualization web page. We developed simple visualization web page and periodically syn- chronized the evaluation results of existing tasks on the page. The page provides several benefits: 1) allows the core contributors keep track of the overall annotation process, 2) it helps the anno- tators understand the capability of state-of-the-an VLMs, so that they can adjust the task difficulty accordingly, and 3) it facilitates the checking of the potential annotation glitches or metric bugs, Figure 10 shows screenshots of the significantly improving the overall quality of construction process. Note that the task names in visualization page taken during the benchmark the figure might not align with the final names in the paper. In our project page, we will provide similar visualization page for users to interactively inspect the behaviors of different VLMs.
27

---

Published as a conference paper at ICLR 2025
C TAXONOMY TREE AND MULTI-DIMENSIONAL KEYWORDS
This section presents the full details of our application-based taxonomy tree and the multi- dimensional keywords.
C.I DETAILS OF THE TAXONOMY STRUCTURE
Table 4 shows the detailed structure of our application-driven task taxonomy. The first level defines the broad scope of use cases. At the second level, tasks are categorized into more specific domains. These first two levels guide the annotation process of our benchmark and are gradually updated/re- fined in the annotation process. The third level lists the concrete names of tasks or task groups. the level node is task group, the number of concrete tasks under this group is shown in the parenthesis.
Table 4: Details of taxonomy of MEGA-BENCH
Level-2 Tasks
Leaf Tasks (at Level-3 or deeper)
Tasks
Coding
Code Debugging
Stackoverflow Debug Qa, Code Error Line Identifica-
2
tion
Code Generation
Document Conversion (8 tasks), Programming Prob- lems (4 tasks), Visualization With Code
13
Code Translation
Code Translation Easy, Code Translation Python, Code Translation Hard, Code Translation Advanced
4
Code Understand-
Symbolic Graphics Programming (2 tasks), Webpage Code Understanding, Code Add Tag, Code Match (5
12
ing
tasks), Code Output (3 tasks)
Information Extraction
App Function Un- derstanding
App Layout Understanding Leetcode, App Layout Understanding Youtube, App Layout Understanding Amazon, App Layout Understanding Word, App Lay- out Understanding Notes, App Layout Understanding Ppt, App Layout Understanding Alipay, App Layout Understanding Instagram, App Layout Understanding Zoom, App Layout Understanding Excel, App Layout Understanding Iphone Settings, App Layout Under- standing Tiktok, App Layout Understanding Twitter
13
Compound Search
Cheapest Flight Identification, Weather Info Re- trieval, Stock Info Retrieval, Game Platform Sup- port Identification, Top Rated Hotel Identification, Movic Info Retrieval, Top Video Creator Identifi- cation, Highest Discount Game Price Identification, Newspaper Page Parse And Count, Remaining Play- back Time Calculation
10
and Calculate
Detailed Manual
Multi Lingual Manual Explanation Scooter Spanish, Multi Lingual Manual Explanation Scooter Arabic, Multi Lingual Manual Explanation Scooter French, Multi Lingual Manual Explanation Scooter Chinese,
5
Understanding
Multi Lingual Manual Explanation Scooter Russian
Multimodal QA
Multilingual News Qa, Product Ocr Qa, Large Image (3 tasks), Gui Chat (2 tasks), Realworld Qa En2en, Star Object Interaction Video, Video Qa (7 tasks)
16
28

---

Published as a conference paper at ICLR 2025
Table 4- continued from previous page
Level-2 Tasks
Leaf Tasks (at Level-3 or deeper)
# Tasks
Search by Attribute wo Calculate
Coco Ood Global Image Retrieval By Query Prop- erty, Places365 Similar Scene Retrieval, Booking Web Recommendation, Game Info Retrieval, Media Homepage Profile, Movie Retrieval By Actor, Music Info Retrieval, Tv Show Retrieval By Character
8
Structured Parsing
Multilingual Movie Info Parsing, Movie Info Pars- ing, Stock Info Parsing, Music Info Parsing, Mul- tilingual Game Info Parsing, Ocr Article Authors, Youtube Video Info Parsing, Tv Show Info Parsing, Ocr Resume School Plain, Image Translation En2en, Booking Web Rating, Weather Info Parsing, Game Info Parsing, Weather Map Climate Type Temperature Parsing, Hotel Booking Confirmation Parsing, Enter- tainment Web Game Style
16
Summarization
Video Summary, Video Short Title, Video2notes, 4 Video Content Reasoning
Knowledge
Arts
Poetry Generation (7 tasks), Ascii Art 30
8
Fact Checking
Background Change, Out Of Context, Text Entity Re- place, Text Style, Face Autribute Edit, Face Swap. terpret Force Perspective Illusion, Clip Stable Diffu- sion Generate, Unusual Images, Forensic Detection Of Different Images, Veracity, Distinguish Ai Gen- crated Image
12
Human and Culture
Cultural Vqa, Human Relationship Reasoning, Sign Language, Ishihara Test, Safety And Norm (13 tasks), Video Content Follow Up. Emotion And Intent Un- derstanding tasks), Theory Of Minds (2 tasks), Hashtag Recommendation
30
World Knowledge
Dish Ingredient Match, Music (6 tasks), Insect Order Classification, Signage Navigation, Song Title Identi- fication From Lyrics, Logo And Sign (3 tasks), Chi- nese Idiom Recognition, Ruozhiba (6 tasks), Font Recognition, Traffic Accident Analysis, Multiple State Identification (4 tasks), Worldle, Location Vqa, Daily (2 tasks), Ancient Map Understanding, Rocks Samples Compare, Painting (2 tasks), Memorization (4 tasks), Soccer Offside, Deciphering Oracle Bone, Actor Character And Famous People (3 tasks), Land- mark And Buliding (3 tasks), Defeasible Reasoning
47
Mathematics
Algebra
Algebra
Calculus
Scibench Calculus Wo Solution
Functions
Math Parity, Math Breakpoint, Math Convexity Value Estimation
3
General
Math Exams V, Theoremqa, Math
3
29

---

Published as a conference paper at ICLR 2025
Table 4- continued from previous page
Level-2 Tasks
Leaf Tasks (at Level-3 or deeper)
# Tasks
Geometry
Geometry Reasoning Count Line Intersections, Ge- ometry Length, Geometry Reasoning Nested Squares, Geometry Transformation, Geometry Reasoning Overlapped Circle, Geometry Area, Geometry Rea- soning Grid, Polygon Interior Angles, Geometry Solid, Geometry Analytic, Geometry Descriptive
11
Graph Theory
Graph Shortest Path Kamada Kawai, Graph Short- est Path Planar, Graph Connectivity, Graph The- ory, Graph Isomorphism, Graph Hamiltonian Cy- cle, Graph Hamiltonian Path, Graph Chordless Cycle, Topological Sort, Graph Maxflow
10
Number Theory
Counterfactual Arithmetic
Numeric Reasoning
Clevr Arithmetic, Iconqa Count And Reasoning, Number Comparison
3
Metrics
Generated Eval
Image
Autorater Artifact, Autorater Control, Autorater Arti- fact Reason, Autorater Aesthetics, Autorater Unmask, Autorater Subject, Autorater 3d Model Texturing, Au- torater Semantics, Autorater Motion Guided Editing. Autorater Mask
10
Generated
Video
Video Eval Visual Pref, Generated Video Artifacts, Video Eval Factual Pref, Video Eval Dynamic Pref
4
Eval
Paper Review
Paper Review Writing, Paper Review Rating, Paper Review Acceptance
3
Quality Assessment
Vizwiz Quality Accessment For Blind
1
Reward Models
Reward Models T2i Reward, Reward Models 12t Re- ward
2
Perception
3D understanding
Adapted Cvbench Depth, Relative Depth Of Differ- ent Points, Visual Prediction Rater Depth Estima- tion, Visual Prediction Rater Novel View Synthe- sis, Pokemon Recognition, Av View Identifica- tion, Multiview Reasoning Camera Moving, 3d In- door Scene Text Bbox Prediction, Google Streetview Circle Reasoning, Google Streetview Direction Un- derstanding, Video Motion Matching Real 3d, Video Motion Matching 3d Real, Visual Prediction Rater 3d Assembled Quality Understanding, Visual Predic- tion Rater Surface Normal Estimation, Visual Predic- tion Rater Plane Segmentation, 3d Indoor Scene Text Bbox Selection, Google Streetview Circle Sorting
17
Counting
Ad Count Detection, Adapted Cvbench Count, Av Vehicle Multiview Counting, Counting Multi Image, Av Human Multiview Counting, Shape Composition Shapes, Counting Single Image, Clevrer Video Mov- ing Object Count, Shape Composition Colours
9
Diagram and Doc- ument Understand-
Diagram (23 tasks), Document (9 tasks), Table Qa (6 38 tasks)
ing
30

---

Published as a conference paper at ICLR 2025
Table continued from previous page
Level-2 Tasks
Leaf Tasks (at Level-3 or deeper)
# Tasks
Image
Segmenta-
Visual Prediction Rater Openable Part Segmentation, Visual Prediction Rater Panoptic Segmentation, Vi- sual Prediction Rater Semantic Segmentation
3
tion
Multimodal tioning
Cap-
Video Detail Description, Guess Image Generation 5 Prompt, Docci Image Description Long, Tweets Cap- tioning, Image Captioning With Additional Require- ments
Multimodal Con- strained Captioning
Contain Contain Images, Contain Repeat Length, Multi Contain Repeat Position Only Length, Contain Length, Contain Position Images, Contain Position Length, Xor Images, Multi Contain Repeat, Contain Contain Length, Multi Contain Position Only
10
Object and Scene Understanding
Autonomous Driving Scene Analysis, Super Clevr Scene Understanding, Functionality Matching In Dif- ferent Objects, Visual Dialog Image Guessing, Nlvr2 Two Image Compare Qa. Egocentric Analysis Sin- gle Image, Clevrer Object Existence Video, Snli Ve Visual Entailment, Oer Open Ended Qa, Semantic Matching Two Images
10
Physical
Under-
Physical Reasoning (8 tasks), Lighting And Shading (2 tasks)
10
standing
Spatial Understand-
Adapted Cvbench Relation, Visual Correspondance In Two Images, 2d Image Jigsaw Puzzle Easy, Ge- ometry Plot Position Relationship, Adapted Cvbench Distance, Video Grounding Spatial, Egocentric Spa- tial Reasoning
7
ing
Temporal
Under-
Video To Camera Trajectory Retrieval, Sceneqa Scene Transition Video, Video Segments Reordering, Video Action Recognition, Action Sequence Understanding, Google Streetview Line Sorting, Next Action Predie- tion, Perception Test Video Action Count, Google Streetview Line Reasoning. Video Camera Motion Description, Video Grounding Temporal, Web Action Prediction, Cam Traj To Video Selection, Sta Action Localization Video
14
standing
Visual Recognition
Face Identity Matching, Rocks Samples Identify, Animal Pose Estimation, License Plate Recogni- tion, Image Style Recognition, Long String Letter Recognition, Coco Object Detection By Query Prop- erty, Widerface Face Count And Event Classifica- tion, Handwritten Math Expression Extraction, Ge- ometry Reasoning Circled Letter, Av Multicamera Tracking Predict Bbox, Ascii Art Understanding, Face Keypoint Detection, Extract Webpage Head- line, Waldo, Geographic Remote Sensing Land Cover, Signboard Identification, Long String Number Recog- nition, Waybill Number Sequence Extraction, Sin- gle Person Pose Estimation, Coco Person Detection, Places365 Scene Type Classification
22
Planning
31

---

Published as a conference paper at ICLR 2025
Table continued from previous page
Level-2 Tasks
Leaf Tasks (at Level-3 or deeper)
# Tasks
Agents and Plan-
Wikihow Complex Task Completion, Navigation (6 tasks). Gui Operation (18 tasks), Calendar Schedule Suggestion, Symbolic Planning (13 tasks)
39
ning
Puzzles and Games
Logical Reasoning Find Odd One Out, Logical Rea- soning Fit Pattern, Perception Test Object Shuffle Video, Board Games (12 tasks), Bongard Problem, Number Puzzle Kakuro 5x5, Mensa Iq Test, Are Agi, Mnist Pattern, Number Puzzle Sudoku, Move Pos To Pos Hanoi 4 Pole, Pictionary (5 tasks), Annoy- ing Word Search, Logical Reasoning Views Of 3d Shapes, Maze 2d 8x8, Crossword Mini 5x5, Rebus, Icon Arithmetic Puzzle, lq Test Open Ended, Ball Cup Swap Logical Reasoning 2d Folding
36
Reordering
Perception Test Video Character Order, Comic Page Ordering, Recipe Image Ordering
3
Science
Chemistry
Chemistry Exams V, Science Molecule Chemistry
2
Life Sciences
Biology Exams Medical (15 tasks)
16
Physics
Circuit Diagram Understanding, Mmmu Physics Chemistry Selected, Science Basic Physics, Physics
4
Exams V
STEM
Mmmu Pro Exam Screenshot, Scibench W Solution
7
Open Ended, Arxiv Vqa, Tqa Textbook Qa. Question Solution Solving, Quizlet Question Solving, Scibench Fundamental Wo Solution
C.2 STATISTICS OF EACH KEYWORD DIMENSION
Figure of the main paper presented the overall keyword distribution. As a complement, Table provides more detailed statistics. Each of the five dimensions contains multiple keywords, and for each keyword, we explicitly show the number of related tasks and the total number of samples.
32

---

Published as a conference paper at ICLR 2025
Table Number of tasks and samples across the five dimensions, with detailed breakdown into each keyword.
Dimension
Keywords (number of tasks, num of samples)
Skills
Object Recognition (303, 4755), OCR (137, 2239). Language Parsing & Gen. (154, 2509). Scene & Event Understanding (154, 2467), Math & Logical Reasoning (109. 1910), Common- sense & Social Reasoning (51, 855), Ethical & Safety Reasoning (15, 245), Noomain-Specife Knowledge/Skills (77, 1387), Spatial & Temporal Reasoning (152, 2437), Planning & Deci- sion Making (37, 577)
Input Format
User Interface (93, 1517), Text-rich Image & Doc (82, 1294), Diagrams Visualizations (101, 1718), Vidcos (43, 698), Artistic & Creative (32, 542), Photographs (143, 2248), 3D Related (11, 169)
Output Format Contextual Formatted (98, 1514), Structured (110, 1714), Exact (83, 1279), Numerical (49, 862). Open- (80, 1454), Multiple Choice (85, 1363)
Input Number
6-8 images (21, 314), 9-image+ (41, 623), 1-image (315, 5228), Video (43, 698), 4-5 images (34, 520), 2-3 images (51, 802)
Application
Information Extraction (72, 1124), Planning (78, 1239), Coding (31, 474), Perception (145, 2313), Metrics (20, 309), Science (29, S74), Knowledge (97, 1605), Mathematics (33, 547)
33

---

Published as a conference paper at ICLR 2025
D EVALUATION DETAILS
This section details our evaluation settings, including the prompt template design, model query details, and evaluation metries.
Prompt Template
<lusk jiss_jnntttcconn>> sglobal_media>, media>,<glabad sallaall_aaaaiiiiee modia?,
Demonstration example(s) task:
Example <example_mcdiaz, eexample_media,,
Example Question: <example <example_questioo>>>>
Example Response:
Sanswer field>: <answer>> funswer_feelaze <answer> /fif multi-field
Answer: <answer> i/ single-field
Example n:
Answer the new question below, following the same task logic and output furmat of the demonstration example(s). Do not output additional contents that violate the specified format.
investion_mediae ciuestion_media>>
Question: <question>
Figure 11: The prompt template structure without ohaon-of-Thought
Thought (CoT).
D.1 PROMPT TEMPLATE
We provide the concrete prompt template in Figure 11 and Figure 12. All the information orga- nized by the prompt template serialized by our evaluation pipeline before sending queries to the evaluated model.
The non-CoT prompt instructs the VLM to strictly follow the one-shot example, directly producing the answer without additional text. In contrast, the CoT prompt instructs the VL.M to output step-by- step reasoning before providing the final answer, and the model must strictly separate the reasoning process from the final answer.
Note that our prompt sets different formats for single-field and multi-field outputs. Single-field must explicitly indicated by the "Answer: ... format that our output parser can bustly locate and extract the model's answer. Multi-field answers are in JSON format, and our JSON parser can robustly extract the JSON-style answer from the entire response without the "Answer. format.
D.2 MODEL QUERY DETAILS
Since the evaluated VLMs have different context windows, we must tailor the number of query images video frames for each model. We implement an image/video pre- pre-processing pipeline that follows the settings listed in Table 6 to sub-sample the input images and videos. We allocate different budgets for in-context examples and the query. Since the in-context examples (we
34

---

Published as a conference paper at ICLR 2025
Prompt Template
jnnkk_instructin>>
<global_medu>> medua>,<lobal gglobal_mediae
Demonstration example(s) of the task:
Example 1: <example media>, <example media?,
Example Question: <example_question
Example Response:
[PLEASE OUTPUT YOUR REASONING]
Sanswer field>: <answer> <answer Canswer_field>; <answer>
Answer: <answer> If single-field
1 i/ multi-field
Example
// nishutt
Answer the new question below. The last part of your response should be of the following format: "Answer: <YOUR ANSWER>" (without angle brackets) where YOUR ANSWER is your answer, following the same task logic and output format the demonstration example(s). For your answer, do not output additional contents that violate the specified format. Think step by step before answering.
<qjestion_medii>> media>, Squestion ppuestionmmdlu>.. mediu> Question: <question>
Figure 12: The prompt template structure for the Chain-of-Thougee (CoT) setting
one-shot example) mainly help models understand the task logic and the output format, we reserve most of the image budget for the query. Images or video frames surpassing the budget discarded. To make sure the open-source models can run smoothly, we implement fallback strategy, which reduces the image budget to decrease the number of input tokens if the model's maximum context length is exceeded.
For images or video frames with longer side larger than 1000 pixels, we resize the longer side to 1000 without changing the aspect ratio before sending them to the evaluated model. Each
D.3 LLM-ASSISTED METRICS
The LLM-assisted metric instructs multimodal LLM to evaluate VLM's response by providing detailed evaluation prompt. When submitting task with open-ended answers that cannot be evaluated by rule-based metrics, the annotator is asked to write down a detailed evaluation prompt for the L.LM judge following the prompt format in Figure 13.
Concretely, the task annotator decides if the LLM judge should consider the question's 's visual input when evaluating the model's response. If yes. then the query media (images or videos) will be passed to the LLM as well (we use GPT-40-0806 as a multimodal judge model). For most tasks, the LLM judge can do a proper evaluation by comparing the model's response with the reference answer, and the visual media is not needed. The task annotator also writes thorough evaluation criteria, explaining to the judge model the meaning of each score range, which important to get reliable evaluation results.
35

---

Published as a conference paper at ICLR 2025
Table 6: The maximum number of images and the budget for the in-context example per model.
Model
Max # of images
In-context example budget
GPT-4o (0513) (OpenAl, 2024a)
64
Claude-3.5 -onnet (1022) (Anthropic, 2024a)
64
Claude-3.5-Someee (0620) (Anthropic, 2024a)
64
Gemini- 1.5-ii--1.-PrOO000 P-o---000 (Google, 2024b)
128
16
1e5mini-.55-Flah---i (Google, 2024b)
128
16
GPT-4o Mini (OpenAI, 2024b)
64
Qwen2-VL-72B (Alibaba, 2024)
24
Intern WieeLWLL2-Llama32- (Chen al., 2024d)
24
4
NVLM-72B Dai et al. (2024)
32
Molmo-72B-0922 (Deitke al., 2024)
1
LLaVA-One Vision 72B (Li al., 2024a)
28
Qwen2-VL-7B (Alibaba, 2024)
18
Pixtral-128 (Mistral, 2024)
48
Aria-MoL-25B (Li al., 2024d)
32
POINTS-Qwen2.5-1 (Liu al., 2024b)
Intem (Chen et al., 2024d)
18
Phi-3.5-Vision Visio (Abdin et al., 2024)
16
MiniCPM-V2 (Yao et al., 2024)
64
Molmo-7B-D (Deitke et al., 2024)
1
LLLVAA-One Visionn- (Li et al., 2024a)
20
Llama-3.2-11D (Mela, 2024)
32
Idefics3-8B-LLanee (Laurençon et al., 2024)
20
Qwen2-VL-2B (Alibaba, 2024)
16
InterVVL2-2B (Chen et al., 2024d)
18
Aquila-V1-2B--lava-geee (Gu et al., 2024)
8
LLM-Assisted Metrics Prompt Template
<media>,<media>
<media>, <media>,
// if judge with image // defined by task annotator
inaluaationcciterrin
Reference: <reference answer>
Model Response:
<model_respee response>
(Optional):
<per_example Lamnple_abbel>>>> label>
// some tasks require per-example crileria
Please output your score in the following format:
""Score*: <single number?,
""Score explanation **; <detailed <detailed_expplaattionss explamations>
Figure 13: The prompt template structure for LLM-Assisted Metries
At the end of the prompt, a pre-defined scoring format instruction is attached, ensuring the judge model outputs score between and 10 and explanation for the score.
D.4 RULE-BASED METRICS
have over 40 highly customized rule-based metrics to evaluate the Core set of MEGA-BENCIL Basic metrics like "extract string match" and "simple string match" (which ignores punctuation and special characters) are first added to the supported metric set. New metrics are implemented when our task annotators submit tasks, as shown in Table 7. The usage distribution is long-tail because many metric implementations triggered by single novel task.
tasks requiring uncovered metries. In the end, get 45 customized
36

---

Published as a conference paper at ICLR 2025
Table 7: All metrics used
Metric Name
Usage Count (# tasks)
Exact String Match
198
GPT-4o as Judge
64
Simple String Match
61
Multi Reference Phrase Evaluation
25
Constrained Generation
Set Equality
15
Sequence Equality
15
General Single Numerical Match
14
Exact String Match Case Insensitive
14
Sequence Accuracy Case Insensitive
Symbolic Planning Test
13
String Set Equality Comma
9
Normalized RMSE
8
Program Judge
Set Precision
Dictionary Equality
String Set Equality Line Break
Sequence Coordinates Similarity
LaTeX Expression Equality
Jaccard Index Case Insensitive Jaccard Index Normalized Bounding Box IOU Tuple
2
Number Relative Difference Ratio
2
XML Bounding Box IOU
2
Dictionary Exact String Match Aggregate Recall
Boxed Single Numerical Match
2
Positive Integer Match
2
Chess Move List Jaccard Index
2
Code Result Exact String Match
Normalized Bounding Box IOU Single
Normalized Bounding Box IOU Sequence
Normalized Similarity Damerau-L
Near String Match
XML Normalized Point Distance
Dictionary Precision
Text with LaTexX Expression Equality
Angle Sequence Float RMSE
Point in Bounding Box
Longest Common List Prefix Ratio
Sequence Equality Case Insensitive
Set Equality Case Insensitive
GLEU (Chinese)
ASCII Art GPT-4O Judge
Dictionary Jaceard Aggregate Jaccard
Dictionary Normalized Bounding Box IOU Tuple Aggregate Jaccard
D.5 ANSWER EXTRACTION FROM MODEL RESPONSE
For Core tasks, our rule-based evaluation metrics compare the model's answer with ground-truth answer or some ground-truth constraints. Therefore, an answer extraction step is necessary to sep- arate the final answer from the reasoning process and other irrelevant texts. We implement robust extraction logic for different types of outputs based on the format specified in the prompt template:
Single-field answer. We first reduce the answer by the "Answer: ..." pattern. If this pattern does not exist, we take the entire response. Since many VLMs do not strictly follow the format instructions. we have specific and extra processing for different output formats to improve robustness. Some typical examples are: 1) For multiple-choice outputs, using sophisticated regular expressions, which excludes any potential parenthesis or accompanying texts; 2) For code outputs, extract the code from the potential code blocks; For structured
locate the exact letter or index choice
37

---

Published as a conference paper at ICLR 2025
outputs, parse the structural data into the proper Python data structures (list, set, dictionary, etc.), with tolerance on minor syntax emrors (e.g., we automatically fix wrong quotes).
Multi-field answer. Since the prompt requires the model output the final answer in JSON format, we implement robust JSON parser to locate the JSON structure in the raw response and convert the JSON structure into the corresponding Python data structure.
our comprehensive answer extraction fails to obtain any meaningful final answer from the model response, we consider the model "fail to follow instructions".
38

---

Published as a conference paper at ICLR 2025
E COMPLETE MULTI-DIMENSIONAL BREAKDOWN RESULTS
This section provides the full breakdown results over the five dimensions of MEGA-BENCH, com- plementing section the main paper.
E. BREAKDOWN RESULTS THE SKILL DIMENSION
Table 8: Average scores for each model on the skill dimension. The best-performing model in each category in-bold, and the second best underlined.
LUAG
Claude-3.5- S.onmettt
(00ne (1022) (Anthropic, 2024b)
59.1
54,9
65.7
60.8
48,9
56.9
29.1
55.1
43.2
62.2
GPT-4o (0513) (OpenAl, 2024a)
55.1
44.2
58.2
Clande-3. Claude-3.-Soomett S-Sonnet (0620) (Anthropic, 2024n)
57.6
5288
69.7
57.5
47.7
54.1
23.8
54.5
40.8
60.8
Gemini-1,5PPro0022 (Google, 2024b)
57.5
51.4
69.8
55.3
#2.6
52.0
23.9
54.7
38.5
50.2
Gemini-1.5- 1.5rmii-..- (Google, 2024bj
44.8
49.9
46.3
19.0
$1.0
34,5
GPT-4o mini (OpenA1, 2024b)
55.7
41.9
69.0
51.7
34.1
44.9
19.4
46.7
29.4
49.0
Qwen2-VL-722 (Alibaba, 2024)
50.9
V1.2
(Chen et 2024d)
$2.6
33.3
57.8
43.7
29.8
38.2
17.0
42.7
29.5
41.3
LLaVA OneVisionn
al., 2024a)
31,7
36.7
29.5
36.2
13.9
42.1
29,6
28.3
NVLM-728 (Dai al., 2024)
26.4
24.0
22.8
Qwen2- VL 70 (Alihabu, 2024)
49.4
33.3
52.2
40.3
28.2
37.1
14.7
41.1
27.6
40.2
Pixtral 128 (Mistral, 2024)
3288
37.8
31.8
Mria-MoF-2SBB (1. al., 2024d)
494
32.8
58.1
40.0
27.6
32.6
11.9
37.8
24.8
35.7
VL2- (Chen al., 2024d)
39.7
27.1
47.0
32.0
24.1
28.2
8.3
32.6
23.2
5hisa.5.Visinn (Abdin 2024)
21.7
30.5
MiniCPM (Yao al., 2024)
40.7
23.7
48.8
30.0
18.3
26.0
8.7
31.8
19.7
25.0
Qwen2 2L (Alihabu, 2024)
31.3
20.8
41.4
25.7
17.6
22.2
6.2
26.5
17.3
23.7
Llama-3,2-11B (Meta, 2024)
32.3
17.7
42.6
19.6
13.3
19.1
6.6
22.4
15.4
14,3
Aquila-V1.222 11l-2B-llaava-ueee
al., 2024)
Intem (Chen et al., 2024d)
24.0
14.8
34.2
16.9
13.9
14.5
1.7
18.5
13.0
12.1
(Laurençon al., 2024)
The abbreviations used in the table above are explained in the following table:
Table 9: Abbreviation list of the keywords in the skill dimension.
Abbreviation
Skill
CASR
Commonsense and Social Reasoning
DKAS
Domain-Specinn Speccffc Knowledge and Skills
EASR
Ethical and Safety Reasoning
LUAG
Language Understanding and Generation Mathematical and Logical Reasoning
MALR
ORAC
Object Recognition and Classification
PADM
Planning and Decision Making
SAEU
Scene and Event Understanding Spatial and Temporal Reasoning
SATR
TR
Text Recognition (OCR)
39

---

Published as a conference paper at ICLR 2025
E.2 BREAKDOWN RESULTS ON THE INPUT FORMAT DIMENSION
Table 10: Average scores for each model on the format dimension. The best-performing model in each category in-bold, and the second best is underlined.
Model
3MAAI
AACC
DADV
P
TIAD
UIS
Claude-3.5-Someee (1022) (Anthropic, 2024b)
44.2
55.6
55.6
54.3
48.9
60.5
49.5
GPT-4o (0513) (OpenAI, 2024a)
47.8
56.4
50.0
56.1
60.8
53.2
Claude-3.5-Sonnet (0620) (Anthropic, 2024a)
44.3
57.0
52.6
51.0
48.0
56.9
50.9
Gemini-1.5-PP P.5--r0002 (Google, 2024b)
42.9
55.8
48.7
55.0
42.9
46.3
50.3
Semmii-1,5-Plash---e (Google, 2024b)
38.5
50.5
40.1
51.7
36.0
38.7
49.0
GPT-4o mini (OpenAl, 2024b)
29.4
47.6
38.9
46.5
36.2
47.2
45.5
Qwen2-VL-72B (Alibaba, 2024)
36.2
50.8
42.1
49.8
42,9
54.0
49.9
Intern BllemVLL2LLlama37760 (Chen et al., 2024d)
28.7
45.0
34.7
42.9
31.4
36.3
39.6
LLaVA VINNA--OnVVssio-72200 (Li al., 2024a)
23,9
44.0
34.6
42.5
23.4
44.5
NVLM-72B (Dai et al., 2024)
34.7
30.3
32.6
21.7
23.9
Qwen2-VL- -onn--VL-7B-ao (Alibaba, 2024)
26.2
34.8
32.2
40.7
29.0
38.2
41.1
Pixtral 12B (Mistral, 2024)
24.0
37.5
32.2
37.1
28.8
30.7
41.0
Aria-MoE-25BB (Li et al., 2024d)
19.6
36.1
32.4
37.3
27.8
28.3
42.9
IntemVI12-8B (Chen et al., 2024d)
10.9
29.4
28.0
33.9
20.1
22.8
34.8
Phi-3.5-Vision (Abdin al., 2024)
15.4
27.9
26.1
34.1
17.5
18.7
24.7
MiniCPM-V2.6 (Yao al., 2024)
7.6
31.0
21.6
31.8
18.6
21.2
35.3
LLaVA -isionnn- et al., 2024a)
13.0
32.0
24.2
32.6
14.7
Qwen2-VL--BB (Alibaba, 2024)
13.4
24.9
19.6
28.8
16.3
19.1
Llama-3.2-11 2BB (Meta, 2024)
6,4
25.2
16.9
24.9
11.5
11.9
21.2
Aquila-VL.-2B-Ilavaal
(Gu et al., 2024)
10.1
19.7
19.4
24.6
11.4
7.5
21.4
IntemVL2-22B (Chen et al., 2024d)
11.9
14.9
16.3
20.1
10.5
5.7
19.0
Idefics3-8B-LJamm33 (Laurençon al., 2024)
4.0
18.4
16.2
14.9
11.4
10.1
16.2
The abbreviations used in the table above explained in the following table:
Table 11: Abbreviation list of the keywords in the input formats dimension.
Abbreviation
Input Format
3MAAΣ
Models and Acrial Imagery
AACC
Artistic and Creative Content
DADV
Diagrams and Data Visualizations
P
Photographs
TIAD
Text-Based Images and Documents
UIS
User Interface Screenshots
V
Videos
40

---

Published as a conference paper at ICLR 2025
E.3 BREAKDOWN RESULTS ON THE OUTPUT FORMAT DIMENSION
Average scores for each model on the ouppu format dimension. The best-performing model in each category is in-hold, and the second best is underlined.
Table 12:
Model
E
M
N
0
S
Srande-3.5-Sonne (1022) (Anthropic, 2024b)
51.9
53,9
57.8
48.2
62.4
50.7
GPT-4o (0513) (OpenAI, 2024a)
53.9
59.9
54.5
44.6
62.7
48.0
Claude-3.5-Somne. (0620) (Anthropic, 2024a)
50.7
52.8
54.6
44.9
58.4
49.7
Demmnni-1.5-Prov (Google, 2024b)
44.9
51.5
55.4
46.9
55.8
44.4
Gemini- 1.5-aii--.-- (Google, 2024b)
38.7
44.8
47.8
37.0
54.5
39.9
GPT-4o mini (OpenA1, 2024h)
41.2
44.2
39.9
36.3
57.1
39.1
Qwen2-VL-72 VL-72B (Alibaba. 2024)
44.7
51.0
52.0
40.3
51.6
45.0
InlemVV.2-Llamaa- (Chen al,, 2024d)
36.3
39.4
38.8
29.2
45.8
34.8
LLaVA-One LLavA-Omeeiiioon-7 Vision-720 (Li et al., 2024a)
28.7
37.1
39.9
30.7
42.9
25.9
NVI.M-72B (Dai et al., 2024)
22.9
27.9
18.5
23.3
32.2
27.9
Owen2-VL-77 (Alibaba, 2024)
34.3
35.2
39.9
32.7
39.1
34.3
Pixtral I2B (Mistral, 2024)
30.8
36.4
30.1
32.1
41.7
31.9
Aria-MoB-25B (Li et al., 2024d)
30.9
29.3
32.8
30.9
45.2
30.4
IntermV1.2-8B (Chen et al., 2022d)
25.1
27.4
30.3
22.4
35.4
25.2
Phi-3.5-Vision (Abdin et al., 2024)
21.8
25.7
26.0
21.4
36.5
21.4
MiniCPM-V2.6 (Yao al., 2024)
29.3
20.8
36.5
17.8
LLaVA-OmeViiionn-22 (Li et al., 2024a)
20.3
25.4
28.0
22.0
31.3
18.3
Qwen2-VL-2B (Alibaba, 2024)
16.2
20.0
25.7
22.0
30.2
21.0
Llama-3.2-11B (Meta, 2024)
12.4
15.8
19.3
15.0
30.0
16.4
Aquila-VL-2B-II -llav----vv (Gu et al., 2024)
11.9
18.5
22.1
19.9
23.3
12.3
Intern V1.2-BB (Chen al., 2024d)
11.3
15.5
21.3
16.0
21.4
5.7
Brefis3-8B-Llanee (Laurençon et al., 2024)
14.0
7.1
11.6
9.8
29.9
10.6
The abbreviations used in the table above are explained in the following table:
Table 13: Abbreviation list of keywords in the output formats dimension.
Abbreviation
Output Format
Contextual Formatted Text
E
Exact Text
M
Multiple Choice
Numerical Data
O
Open-ended Output
Structured Output
41

---

Published as a conference paper at ICLR 2025
E.4 BREAKDOWN RESULTS ON THE APPLICATION DIMENSION
Table 14: Average scores for each model on the application dimension. The best-performing model in each category is in-bold, and the second best is underlined.
Model
M2
Claude-3.5- 3.5-oonett (1022) (Anthropic, 2024b)
51.7
65.9
56.6
47.6
61.2
55.6
39.9
55.1
GPT-4o (0513) (OpenAl, 2024a)
50.3
70.6
61.4
44.0
61.0
55.1
33.2
52.8
Claude-3.5 3.5-oonett (0620) (Anthropic, 2024a)
51.9
66.6
55.1
47.5
58.1
53.2
33.8
51.3
Gemini-1 Sommini-1.5-Prooce 1.5ooo-000 (Google, 2024b)
43.5
57.2
41.2
58.2
52.5
33.4
51.2
Gemini-1.5-Plass 0.5-Plash-002 (Google, 2024b)
40.4
33.7
60.1
48.0
45.7
GPT-4o mini (OpenAT, 2024b)
34.6
54.0
32.9
51.8
43.6
35.5
Qwen2-VL-72B (Alibaba, 2024)
43.7
58.1
51.7
31.2
49.7
53.6
31.2
44.9
JntemVVL2-Llamm3-7766 (Chen al., 2024d)
29.5
43.1
46.3
28.7
47.4
42.2
21.3
30.0
LLaVA-OneVisionn222 (I.i al., 2024a)
23.2
30.8
43.6
31.6
48.1
38.4
18.2
31.7
NVLM-72B (Dai al., 2024)
23.9
22.8
37.2
24.5
18.9
30.2
8.0
24.9
Qwen2-VL-7B (Alibaba, 2024)
32.7
42.7
42.8
25.6
42.5
40.0
20.0
29.9
Pixtral 12B (Mistral, 2024)
25.7
43.0
38.1
24.2
50.2
38.9
13.6
31.3
Aria-MoF-25B (1.i et al., 2024d)
28.5
38.3
41.0
26.2
39.7
37.8
14.3
29.7
IntemVL2-8B (Chen et al., 2024d)
24.7
29.1
33.9
22.1
40.0
32.1
12.2
24.6
Phi-3.5-Vision (Abdin et al., 2024)
21.9
22.4
33.3
17.6
39.5
31.6
8.9
21.9
MiniCPM-V2.6 (Yao al., 2024)
15.3
26.7
33.2
16.5
37.8
29.2
11.7
25.7
LLaVA-One Vision-7B (Li et al., 2024a)
15.2
19.3
32.7
22.1
36.0
28.5
23.7
Qwen2-VL. -LB (Alibaba, 2024)
17.0
16.4
31.0
Llama-3.2-11B (Meta, 2024)
5.8
17.3
28.1
13.9
25.4
19.9
16.3
Aquila-VL-2BB 2ooua--VL-2Blllaa--eeenn (Gu al., 2024)
13.3
9.5
24.1
20.7
29.3
20.7
5,9
21.1
VntemV1.2-2B (Chen al., 2024d)
11.3
8.7
21.2
11.0
33.3
17.0
4.1
16.9
Idefics3-8B-Linne L--lmmm3 (Laurençon al., 2024)
9.1
14.7
17.6
13.2
14.6
14.6
5,4
22.7
The abbreviations used in the table above are explained in the following table:
Table 15: Abbreviation list of keywords in the applications dimension.
Abbreviation
Application
C
Coding
cnommation-Extrattinnn
K
Knowledge
M
Mathematics
M2
Metrics
Perception
P2
Planning
Science
42

---

Published as a conference paper at ICLR 2025
E.5 BREAKDOWN RESULTS ON THE VISUAL INPUT NUMBER DIMENSION
Table 16: Average scores for each model on the visual input number dimension. performing model in each category in-hold, and the second best is underlined.
The best-
Model
21
41
61
90M
V
Claude-3.5-Sonne (1022) (Anthropic, 2024b)
56.4
48.8
48.3
46.3
59.1
49.5
GPT-4o (0513) (OpenAI, 2024a)
56.7
49.1
45.0
47.5
53.4
53.2
Claude-3.5- (0620) (Anthropic, 2024a)
53.7
49.3
44.2
46.3
54.1
50.9
Semini-1.5-Pro-000 (Google, 2024b)
50.3
45.5
48.9
39.1
53.7
50.3
Gemini-1.5-Flashee (Google, 2024h)
44.3
42.0
42.3
33.7
43.7
49.0
GPT-4o mini (OpenAl, 2024b)
46.3
37.0
24.7
33.6
43.1
45.5
Qwen2-VL-72B (Alibaba, 2024)
49.2
45.2
36.7
31.0
54.7
49.9
Intern MitemVL22LLamm337755 (Chen et al., 2024d)
41.5
31.5
24.4
20.3
34,8
39.6
LLaVA-OneVisiion- (Li et al., 2024a)
34.8
34.2
25.0
20.7
28.1
44.5
NVLM-72B (Dai et al., 2024)
36.8
23.3
3.8
0.0
0.0
0.0
Qwen2-VL- (Alibaba, 2024)
37.7
33.0
26.4
19.4
41.1
Pixtral 12B (Mistral, 2024)
37.1
31.0
25.8
19.7
41.0
Aria-MoE-25BD et al., 2024d)
35.8
27.3
19.8
21.1
27.1
42,9
IntemVL2--BB (Chen et al., 2024d)
30.1
25.3
17.7
15.4
19.9
34.8
Phi-3.5-Vision (Abdin al., 2024)
27.8
28.5
20.2
12.5
24.7
MiniCPM-V2.6 (Yao al., 2024)
26.3
22.3
17.9
14.0
35.3
LLa (Li al., 2024a)
25.5
24.1
17.8
14.8
13.8
31.0
Qwen2-VL--2B (Alibaba, 2024)
25.0
21.3
17.4
7.7
25.2
Llama-3.2-10 IBB (Meta, 2024)
19.6
18.6
13.5
14.6
7.3
21.2
A (Gu et al., 2024)
18.2
23.3
19.0
11.1
1.2
21.4
IntemVL2-2B (Chen et al., 2024d)
15.2
15.8
17.7
3.7
5.8
19.0
Idefics3-8B-Llama3 (Laurençon et al., 2024)
14.8
12.3
12.2
10.1
9.3
16.2
The abbreviations used in the table above are explained in the following table:
Table 17: Abbreviation list of keywords in the visual input number dimension.
Abbreviation
Input Number
1-image
21
2-3 images
41
4-5 images
61
6-8 images
90M
9-image more
V
video
43

---

Published as a conference paper at ICLR 2025
F DETAILED INSPECTION OF MODEL BEHAVIOURS ON MEGA-BENCH
To complement $4.3 of the main paper, this section presents case study analysis of the error types of different models on different tasks in MEGA-BENCIL. We use similar error categories in MMMU (Yue et al., 2024a) and MMT-Bench (Ying et al., 2024):
Perception Error: VLMS fail recognize or perceive the content of interest in the query image(s). Perception errors indicate the
Lack of Knowledge: VLMs lack the domain-speciinn knowledge answer specialized questions, such as identifying the taxonomic order of an insect.
Lack of (Reasoning) Capability: VLMs lack the necessary capabilities to solve the task, mainly related to various reasoning abilities, such as logical reasoning, counting, spatial or temporal rea- soning, symbolic reasoning for code or various programs, and on. This broad type that covers many errors. One typical case for this error type is that the models can accurately follow instructions and perceive the visual inputs but struggle with the required reasoning process, leading to incorrect answers.
Refuse Answer: VLMs refuse to answer questions that they believe involve sensitive content.
Fail to Follow Instructions: VLMs fail to correctly understand instructions and provide wrong answers. The tasks in MEGA-BENCH usually have more instructions on the answer format com- pared to previous benchmarks. A typical error pattern is not comprehending the required format, thus providing answers with incorrect formats or generating irrelevant responses. This error type is much more common in open-source models.
Figure 14 to Figure 33 shows the case study for samples from different tasks. We use distinct colors highlight the tags in each task sample. We borrow the error case analysis template from MMMU (Yue et al., 2024a) while adding the keywords information of MEGA-BEIE -ENCH. We mainly focus on the flagship proprietary models. The Claude-3.5 in these figures refers to the Claude-3.5- Sonnet (0620) model.
44

---

List of Figures Depicting Model Behaviors
14
Coding Code Debugging: Error Case
46
15
Coding Code Understanding: Error Case
47
16
Coding Code Understanding: Error Case 2
48
17
Information Extraction Multimodel QA: Error Case
18
Information Extraction Detailed Manual Understanding: Error Case
50
19
Information Extraction Search by Attribute without Calculate: Error Case
20
Knowledge World Knowledge: Error Case 1
52
21
Knowledge World Knowledge: Error Case 2
53
22
Knowledge- Art: Error Case
54
23
Mathematics Graph Theory: Error Case
55
24
Metrics Generated Image Eval: Error Case
56
25
Metrics Generated Image Eval: Error Case 2
57
26
Planning Puzzles and Games: Error Case
58
27
Planning Puzzles and Games: Error Case 2
59
28
Planning Reordering: Error Case
60
29
Perception Multimodal Constrained Captioning: Error Case
61
30
Perception Visual Recognition: Error Case
62
31
Perception Visual Recognition: Error Case 2
63
32
Perception Visual Recognition: Error Case 3
33
Science STEM: Error Case
65
45

---

Published as a conference paper at ICLR 2025
Coding: Code Debugging
Task Name: Code Error Line Identification
Given piece of program code, identify the most obvious error. Output the line number the left where the error occurs. Attention: Do not invoke any external compilers or related programs. Question:
Where the occurs?
Keywords:
Domain-Specific Knowledge and Skills,
Images and
Documents, Mathematical and Logical Reasoning, Multiple Choice
Topsort (NUN), Topllun NUM)
int SIMMI, Indegree (NUM);
3
int Counter 9, top,
top- -1;
neMUM;
11=0; ssn; i++)
B
Indegren m31
for (i) Indegree [j]reel]]]-
21
Indegree|1] II S(etbop)
white (tog>=0))
S[top=];
Toplinn Counters
26
for (j=B; jen; j++]
(1
1f (Indegree [- [-j1
s(+stoll-j;
19
printfl
"Graph cycle"
21
correct. 2. The first properly structured. The while loop (lines 13-19) also seems structurally correct. 4. However, within this while loop, there's an issue on line 18: 5. This error the most obvious and significant in the code, will likely cause runtime or incorrect results. Therefore, the most obvious error occurs on line 18.
Claude-3.5: 1. The function declaration and variable initializations
nested loop (lines 7-12) appears
Answer: 18
(score: 1.0)
Error Category: Lack of Capability
GPT-4o: The most obvious error the code is use "printf' instead of prrinff line The correct function name should be "printf..
Answer: 20 (score: 0.0)
Correct Answer:
Figure 14: sample error case of Coding (subfield: Code Debugging). Source:Weh Back to List of Figures
46

---

Published as a conference paper at ICLR 2025
Coding: Code Understanding
Task Name: Symbolic Graphics Programs Computer Aided Design
Given symbolic graphics program, the task is to answer a related question by understanding the program. Question: How many visible screws or screw holes are there on CAD object?
Keywords:
Mathematical and Logical Reasoning,
Contextual Formatted Text, Domain-Specific Knowledge and Skills, Fext-Based Images and Documents
SOL:
(1-11300126664..udeeee2222 (223
L1ne:222..7722/vr:221,, 178
Arcc(128.776644.))1eee222333 1788644
SOL:
(13,11.33
SOL:
(13,1,633
C1rcle(176,161.10g
Circlr(112,13.333
SOL,;
Circee(11..11133)
1,,,,44,199,,,,,,,,,,,,,,,000,0, Newbboy One sidedd;
SOL;
SOL. Circle(176.1733.1g
Ext((192,64,922 2,,,,,4,,99,,,,,,,,,,,,, 28,107,224,112 htw.://onn...eeieeeeea.....
Claude-3.5: 1. In CAD programs, screws screw holes are often represented by circles. Looking at the program, Answer: (score: 1.0)
several Circle commands:...
Error Category:
Lack of Capability
GPT-4o: To determine the number of visible screws or screw holes in the CAD object, we need to identify the instances of circles in the program, as circles typically represent screws holes in CAD drawings
Answer: 6 (score: 0.0)
Error Category: Lack of Capability
Gemini-1.55 1--poo The code shows six circles that could be screw holes and two "Ext" commands that specify extrusions. Extrusions be used to create screw holes.
Answer: (score: 0.0)
Correct Answer:
Figure 15: sample error case of Coding (subfield: Code Understanding).
Source:Wel
Back to List Figures
47

---

Published as a conference paper at ICLR 2025
Coding: Code_ Understaatt
Task Name: Code Output Result
Given several images of program code, output the program's execution result. Attention: Do not invoke any external compilers or related programs. Place the code output in a Markdown result code block. Question: What's the output of the program?
Keywords:
OCR, Mathematical and Logical Reasoning Domain-Specific Knowledge, Text-Based Images and Documents
Le Dor Notadd
cata corattutuu met 43.1
1st hevile coredtt now II
Error Category: Lack of Capability
Claude 3.5 :Let's analyze this Rust code step by step:
1. We start with vector [1, 2, 3] wrapped in an Are=Mutexx...>>
2. We create empty vector 'handles' store thread handles.
3. We spawn threads (for in 0.33):
Each thread gets clone of the Arc.
Each thread locks the mutex and increments the first element of the vector.
4. After spawning, we join all threads, ensuring they complete.
5. Finally, we print the contents of the vector.
Let's trace changes:
Initial state: [1, 2, 3]
Thread 1: [2, 2, 3]
Thread [3, 2, 3]
Thread 3: [4, 2, 3]
The final state of the vector will be [4, 2, The printin! macro will output this vector. Answer: Final data: (4, 3| 3] (score: 0.0)
Correct Answer: Final data: [1,
Figure 16: A sample enror case of Coding (subfield: Code Understanding).
Source: Web
Back to List of Figures
48

---

Published as a conference paper at ICLR 2025
Information Extraction: Multimodel QA
Task Name: Object Interaction Video
The task involves identifying objects person interacts with in various videos based on the provided questions. You need to output the object name. Question: Which object was taken by the person?
Keywords:
Contextual Formatted Text, Scene Understanding. Scene and Event Understanding, Videos
GPT-4o: Answer:
pillow (score: 1.0)
Error Category: Perception Error
Claude-3.5: After carefully analyzing the sequence of images, can that the person in the video is interacting with hat. In the earlier frames, the person holding what appears be white or light-colored object, which is likely laundry or bedding. However, as the sequence progresses, see the person in front of mirror, and they appear be adjusting handling a hat. In the final frames, there's motion that suggests the person is tossing or throwing the hat upwards.
Answer: hat (score: 0.0)
Correct Answer: pillow
Figure 17: A sample error case of Information Extraction (subfield: Multimodel QA). Source: MVBench (Li et al., 2024e) and STAR (Wu et al., 2024) Back to List of Figures
49

---

Published as a conference paper at ICLR 2025
Information Extraction Detailed Manual Understanding
Task Name: Multi-lingual Manual Explanation Scooter (Chinese)
~ R # ( i Question: / 7 +
Keywords:
Document, Open-ended
Uheree
codeeryee light
GPT-40 Answer: - SAAAA
दr. A Mi, 7 #R (score: 0.9)
Error Category: Perception Error, Lack of Knowledge
Idefics3:
Answer: eb.
# B
(score: 0.0)
Reference Answer: + wtw.:H
Aee 0 #
X
H 1 #Tx
$
45
Figure 18: A sample error case of Information Extraction (subfield: Detailed Manual Understanding).Souiee Source: Web Back to List of Figures
50

---

Published as a conference paper at ICLR 2025
Information Extraction: Search by Attribute without Calculate
Task Name: Game Info Retrieval
Given set screenshots from game information page and a specific query, retrieve the names of the games that match the query. If there are multiple matching games, provide a list of game names separated by commas. Question: Find the game available on 'MaCOS'
Keywords:
Object Recognition and Classification, OCR, Language Understanding and Generation, User Interface Screenshots
IAW NIT
Ore on Insronaoo
Claude-3.5: Let's go through this step-by-step:
We need to look for game that specifically mentions Mac MacOS platform... Answer: World of Goo 2, Snow Runner (score: 1.0)
Error Category: Perception Error GPT-4o: To find the game available MacOS', I will examine each screenshot for the platform information.
6. World of Goo 2: No mention of MacOS.
Answer: Snow Snoweunner(scor:: 0.0)
Correct Answer: World of Goo 2, SnowRunner
Figure 19: A sample error case of Information Extraction (subfield: Search by Attribute without Calculate). Source: Web Back to List of Figures
51

---

Published as a conference paper at ICLR 2025
Knowledge: World Knowledge
Task Name: Insect Order Classification
This task involves the classification of insects into their respective taxonomic orders based on visual input. The goal to accurately identify and categorize insect from image into of the pre-defined orders. Each order represented by a specific label corresponding to the insect's taxonomic classification, including orders such as Diptera, Hymenoptera, Coleoptera, Hemiptera, Lepidoptera, Psocodea, Thysanoptera, Trichoptera, Orthoptera, Blattodea, Neuroptera, Ephemeroptera, Dermaptera, Archaeognatha, Plecoptera, or Embioptera.
Keywords:
Object Recognition and Classification, Photographs, Domain- Specific Knowledge and Skills, Contextual Formatted
Error Category: Lack of Knowledge
GPT-4o: 1. The insect has pair of wings. 2. The antennae are feathery, which is a
characteristic feature of some insects. The body structure relatively small and delicate
Answer: Trichoptera (score: 0.0)
Error Category: Lack of Knowledge
Claude-3.5: 1. The insect is small and appears to be in circular container, likely for observation purposes. The insect has an elongated body shape
Answer: Lepidoptera (score: 0.0)
Error Category: Lack of Knowledge
Gemini-1.5-pro. The insect has long, slender antennae and its wings are held roof-like over its body. These are characteristic the order Trichoptera, or caddisflies. Answer: Trichoptera (score: 0.0)
Correct Answer: Hymenoptera
Figure 20: A sample error case of Knowledge (subfield: World Knowledge). Source: BIOSCAN-IM (Gharaee et al., 2024)
Back to List of Figures
52

---

Published as a conference paper at ICLR 2025
Knowledge: World Knowledge
Task Name: Memorization Chinese Celebrity
Please answer the name and nationality of the celebrity. For the name, please put last name in the front.
Keywords:
Object Recognition and Classification, Photographs, Language Understanding and Generation, Structured
Error Category: Lack of Knowledge GPT-4o: don't know who they are.
Answer: (scors: 0.0)
Error Category:
Refuse Answer
Claude-3.5: apologize, but I'm not able to identify or name specific individuals in images. However, can describe what I see in this photo:...
Answer: null (score: 0.0)
Error Category: Lack of Knowledge
Gemini-1.5-pro: The person in the image is Eric Chou. He is Taiwanese singer. Answer: I"name": "Chou Chih-chich", "nationality": "Taiwanese") (score: 0.0)
Correct Answer: I"name": "Lin Junjie"; "nationality":: "Singaporean"i
Figure 21: A sample error case of Knowledge (subfield: World Knowledge). Source: Web Back to List of Figures
53

---

Published as a conference paper at ICLR 2025
Knowledge: Arts
Task Name: Poetry Generation (Acrostic)
You are an award-winning poet who has been writing poems for over 20 years. Please write acrostic poem about the attached image, using the letters in the specified word words start off the first letter of each line the poem. space between the words indicates spaces, there should not be any blank lines. No title is needed.
newline. If there Question: Word: hamster
Keywords:
Object Recognition and Classification, Language Understanding and Generation, Photographs
GPT-4o: Answer:
Huddled in cozy corner,
tiny heart beats with gentle grace. Mischief sparkles in those bright eyes,
Soft fur, warm embrace.
Tiny paws clutching a hidden treasure, Eagerly nibbling, lost in pleasure.
Radiating charm, a small wonder, (score: 1.0)
Error Category: Fail to Follow Instruction Dw-22-VL-72B: Answer: Humble hamster, with fur fine,
creature s small, yet full of might. Majestic in its tiny form, Standing tall, without Eyes so bright, curious gleam, Ruling its world, in tiny dream. (score: 0.0) (acrostic: 0, contain:
Eval Context: acrostic: hamster, contain: [hamster, paw, tail, whisker, seed, rodent, scurry, fur]"
Figure 22: A sample error case of Knowledge (subfield: Art), Source: Web Back to List of Figures
54

---

Published as a conference paper at ICLR 2025
Mathematics: Graph Theory
Task Name: Topological Sort
For given Directed Acyclic Graph (DAG), your task is output all possible topologica orders. The output a set of topological order.
Keywords:
Spatial and Temporal Reasoning, Structured, Mathematical and Logical Reasoning, Diagrams and Data Visualizations
DMG Example
Error Category: Lack of Capability
GPT-4o: To determine which 3D object can be formed from the given 2D foldable sheet, we need visualize how the sheet folds into 3D shape. Let's analyze the 2D sheet and the 3D options by step
Answer: [0>>1-22->3->4, 0->1->3->4->2 0->... (score: 0.0)
Error Category:
Lack of Capability
Claude-3.5: 1. Identify the nodes with no incoming edges (sources):
-oode 0 is the only source this graph ....
Answer: [0=11=.2-2332440.-.1222-----
(score: 0.0)
Error Category: Lack of Capability
Gemini-1.55pro. We can perform a depth-first search (DFS) from each node with no incoming edges. During the DFS, we maintain a stack to keep track of the visited nodes...
Answer: [0>>2-11-24-33
0->2->1->3->4
(score: 0.0)
Correct Answer: [->1->2-~3--]
Figure 23: A sample error case of Mathematics (subfield: Graph Theory).
Source:Web
Back to List of Figures
55

---

Published as a conference paper at ICLR 2025
Metrics: Generated Image Eval
Task Name: Autorater Motion Guided Editing
Motion-guiee guided image editing is important for generating realistic and context-aware animation. Your task rank the motion-guided image generation results. The desired motion is described by text input, several generative models try to edit the input image based the described motion. You should rate the generation results based 1) how well the generated image follows the motion described by the text query and 2) whether generated image is still consistent with the source image in terms of semantics and object attributes. The first image is the source image; the rest the edited results. Rank the results using their indices, starting with Exclude the source image from the ranking. Question: Move the lion to the left
Keywords:
Object Recognition, Photographs, Spatial and Temporal Reasoning, Structured
Input image
Claude-3.5: 1. Image 2: Best follows the instruction move left while maintaining the lion's features and overall composition. 2. Image 3: Maintains consistency with the source image but shows minimal leftward movement. 3. Image 1: Follows the leftward motion but obscures too much
Answer: 2, 3, (score: 1.0)
Error Category: Fail to Follow Instruction Idefics3:
Answer: (score: 0.0)
Error Category: Fail Follow Instruction
The first image shows the lion facing forward, with head straight. The second image shows the lion tilting its head to the left ... Based on these observations, we can conclude that the second image the one that shows the lion moving the left.
internyl-76B:
Answer: (score: 0.0)
Correct Answer: 2,3,1
Figure 24: A sample error case of Metrics (subfield: Generated Image Eval). Source:Motion Guidance (Geng & Owens, 2024) Back to List of Figures
56

---

Published as a conference paper at ICLR 2025
Metrics: Generated Image Eval
Task Name: Autorater 3d Model Texturing
You will asked evaluate quality of textured models by comparing two texture transfer results. The first image will show an untextured 3D model and source image whose texture needs to be transferred the mesh. The second image will show two textured 3D models for comparison, and you should pick one with better texturing quality (left or right).
Keywords:
Object Recognition, 3D Models and Aerial Imagery, Scene and Event Understanding. Contextual Formatted Text
3D Model
Texture Imoge
Claude-3.5: Model doesn't match the source image well. While it's realistic vehicle, it's different type (cargo truck instead of passenger van).
Answer: right (score: 1.0)
Error Category: Perception Error
GPT-4o: determine which textured model has better texturing quality, we need to consider the realisticness of the textured object and the consistency between the textured mesh and the source image....
Answer: left (score: 0.0)
Perception Error Gemini-1.5-pro: The left image (A) is white truck, while the right image (B) is combination of truck and van. The source image is van. Therefore, left image is more realistic and consistent with the source image....
Error Category:
Answer: left (score: 0.0)
Correct Answer: right
Figure 25: A sample error case of Metrics (subfield: Generated Image Eval). Source: EASI-Tex (Perla al., 2024) Back to List of Figures
57

---

Published as a conference paper at ICLR 2025
Planning: Puzzles and Games
Task Name: Pictionary GenAl Output Chinese
Look at the provided image and question, guess which Chinese word or phrase the image is trying convey. Answer the question with format " Answer: SOUTPUT" where SOUTPUT your answer in Chinese character (simplified).
Question:[]] #a क
> #
# ?
Keywords:
Object Recognition and Classification, Artistic and Creative Content, Language Understanding and Generation, Contextual Formatted
Error Category: Lack of Capability
GPT-4o: The image shows tiger making tea, which is visual representation of the Chinese idiom " - Á" (a tiger that doesn't eat people). However, the specific six-character idiom being depicted here is " A (a tiger that doesn't eat ginseng)-
Answer: * 7
(score: 0.0)
Error Category: Lack Capability
Claude-3.5: 1. We see tiger in the image, which is the central figure
tiger seems be focused on or studying burning incense.
Answer: E g
(score: 0.0)
Correct Answer: MA
Figure 26: A sample of Planning: (subfield: Puzzles and Games).
Source: Web
Back to List of Figures
58

---

Published as a conference paper at ICLR 2025
Planning: Puzzles and Games
Task Name: Logical Reasoning 2d Folding
Given the query image including a 2D foldable sheet (the first image) and several options for the folded 3D objects (the remaining images), please find out which 3D object comes from the 2D sheet. The answer should the image index starting from that represents the second image.
Keywords:
Spatial and Temporal Reasoning, Multiple Choice, Mathematical and Logical Reasoning, Diagrams and Data Visualizations
Error Category: Lack of Capability
GPT-4o: To determine which 3D object can be formed from the given 2D foldable sheet, we need visualize how the sheet folds into 3D shape. Let's analyze the 2D sheet and the 3D options step by step...
Answer: (score: 0.0)
Error Category:
Lack of Capability
Claude-3.5: First, we need to analyze the foldable sheet (the first image). has a unique shape with several rectangular sections and one triangular section. 2. Now, let's look at each the 3D objects (images 1-5) and try match them with the 2D sheet
Answer: (score: 0.0)
Error Category:
Lack of Capability
Gemini-1.5-pror The 2D sheet has a long base with a small square attached to the right. There are two protrusions from the base, one with triangular cutout and one rectangular. Only option satisfies these conditions.
Answer: (score: 0.0)
Correct Answer:
Figure 27: A sample enror case of Planning (subfield: Puzzles and Games).
Source: Web
Back to List of Figures
59

---

Published as a conference paper at ICLR 2025
Planning: Reordering
Task Name: Perception Test Video Character Order
The task involves identifying the order of letters person writes, types, or arranges them, predicting subsequent letters, and reversing letter sequences in various scenarios. What was the order of the letters at the beginning?
Keywords:
Object Recognition, Video, Spatial and Temporal Reasoning, Contextual Formatted Text
GPT-4o: The order of the letters at the beginning was M X.
Answer: mx (score: 1.0)
Error Category: Fail Follow Instruction
Idefics3: The order the letters at the beginning was not mentioned in the facts provided.
Answer: null (score: 0.0)
Error Category: Perspection Error Gemini-1.5-pra The letters at the beginning of the video spell out MIX. The person first places the M, then the I, and finally the X.
Answer: mix (score: 0.0)
Correct Answer: MX
Figure 28: A sample error case of Planning (subfield: Reordering). Source:Perception Test (Patraucean et al., 2024) Back to List of Figures
60

---

Published as a conference paper at ICLR 2025
Perception: Multimodal Constrained Captioning
Task Name: Constrained Generation Contain Length
Can you write story about the animal in the figure with more than 10 but less than 20 words?
Keywords:
Object Recognition and Classification, Open-Ended, Language Understanding and Generation, Photographs
Two curious cals, one tabby and one ginger, explored their cozy home, discovering hidden treasures together. (score: 1.0)
GPT-4o:
Error Category: Fail Follow Instruction
Claude-3.5: Two feline friends, tabby and ginger, lounged side by side, their curious eyes gleaming with mischief and companionship. (score: 0.0) (contain: 0, length1: 1, length2: 0)
Eval Context: contain: [cat, kitten], length1: >10, length2: <20
Figure 29: sample error case of Perception (subfield: Multimodal Constrained Captioning).
Source: Web
Back List of Figures
61

---

Published as a conference paper at ICLR 2025
Perception: Visual Recognition
Task Name: License Plate Recognition
Please extract the license plate name from the image, ignore the punctuations.
Keywords:
Text Recognition (OCR), Contextual Formatted Text, Object Recognition and Classification, Photographs
HINIO
GPT-4:
N -INIO
Answer: ÆNHINIO (score: 1.0)
Error Category: Lack of Knowledge
Claude-3.5: Let's approach this step-by-step: 1. The image shows blue license plate, similar the example provided ... The task asks ignore punctuation, so we should remove the dot (+). 6. Combining all elements and ignoring punctuation, we get: NHINN0..
Answer: NHINIO (score: 0.0)
Error Category: Lack of Knowledge
Gemini-1.5-pro, The license plate in the image contains the characters " followed by "HINIO""
Answer: NHINNO (score: 0.0)
Correct Answer: NHINIO
Figure 30: A sample error case of Perception (subfield: Visual Recognition).
Source:Wet
Back to List of Figures
62

---

Published as a conference paper at ICLR 2025
Perception: Visual Recognition
Task Name: Coco Person Detection
You asked to detect persons in query image. The format of bounding box is (x1.y1xx2y22, where (x1.y1) is the top-left corner and (x2,y2) the bottom-right corner. The coordinates are normalized by the image's height and width.
Keywords:
Object Recognition and Classification, Photographs, Numerical Data
Error Category: Perception Error
GPT-4o: 1. Identify the persons in the image. 2. Determine the bounding boxes for each person.
3. Normalize the coordinates by the image's height and width
Answer: (nnumber person" "2";
"bounding boxes": (,.000000003.0000000000000000000
(00000000000000000000000001",,,, (score: 0.54)
Error Category: Perception Error
image, both partially submerged water. The first person is on the left side of the image, sitting in shallow water near the shore...
Claude-3.5: 1. can see two people
Answer: I" l"number of person": " "2";
"bounding boxes": (0.20,0.4000.55.0009) (0.70,0015,0900003555))
(score: 0.64)
Error Category: Perception Error
Gemini-1.5-pro: There
people in the image. The man is the left and the woman is
on the right...
Answer: ("number of person": "2";
"bounding boxes": (0.4220.5,,074,.88)),....) (0.0,,220000000000030000000011,,
(score: 0.62)
Correct Answer: (" n"number of person": "2"; "bounding boxes": (0.,304446...50777,008844040,..,
Figure 31: A sample error case of Perception (subfield: Visual Recognition). Source: COCO (Lin et al., 2014)
Back to List Figures
63

---

Published as a conference paper at ICLR 2025
Perception: Visual Recognition
Task Name: Face Identity Matching
This task involves analyzing set of 5 images to identify which images show the same person. Among the 5 images, there individual, and one image shows different person. The goal is to determine the matching pairs by providing the indices of the images that depict the same person. The answer should list the smaller indices first within each pair.
pairs of images where each pair shows the same
Keywords:
Scene and Event Understanding, Object Recognition, Photographs, Contextual Formatted
Claude-3.5 The first image shows an older woman with short, brown hair. 2. The second image shows man with short, dark hair. 3. The third image shows man with short, light brown hair. 4. The fourth image shows an older woman with short, light brown hair ...
Answer: 14,25
(score: 1.0)
Error Category: Fail to Follow Instruction Idefics3: Example Response: [PLEASE OUTPUT YOUR REASONING]
Answer: 1, (score: 0.0)
Correct Answer: 14,25
Figure 32: A sample error case of Perception (subfield: Visual Recognition).
Source: CelebA (Liu al., 2015)
Back to List of Figures
64

---

Published as a conference paper at ICLR 2025
Science: STEM
Task Name: Scibench Fundamental wo Solution
Solve the problem. The answer is a number. A proton is a distance d/2 directly above the center of square of side d. What is the magnitude of the electric flux through the square? (Hint: Think of the square as one face of cube with edge d).
Keywords:
Mathematical and Logical Reasoning, Numerical Data, Domain- Specific Knowledge and Skills, Diagrams and Data Visualizations
d 2
d
Error Category: Lack of Capability
GPT-4o: solve the problem, we need determine the electric flux through the square due to the proton positioned at a distance \( d/2 \) directly above the center of the square
Answer: 3.01
\times 104(-21]
(score: 0.0)
Error Category: Fail to Follow Instruction
Claude-3.5: First, recall Gauss's for electric fields. states that the total electric flux through a closed surface is equal to the enclosed charge divided by (the permittivity of free space)
Answer: / (score: 0.0)
Error Category: Fail Follow Instruction
Gemini-1.5-pro: The electric flux through closed surface is given by Gauss's law: P\PhiEE [enc]]| (nccll/\eppslon O)$...
Answer: S 0)S (score: 0.0)
Correct Answer: 3.01
Figure 33: A sample error case of Science (subfield: STEM).
Source: SciBench (Wang et al., 2023b)
Back to List of Figures
65

---

Published as a conference paper at ICLR 2025
G DETAILED TASK INFORMATION
In Table 18, we list data source details for every task in our benchmark. We also list the output format and metries help better understand each task's form.
Table 18: Detailed task information description of MEGA-BENCE
Task Name
Source Description
Output Format
Metrics
Information Extraction
Cheapest Flight Identification
Screenshots were taken by the human annotator on Google Flights. Questions and answers were created by the annotator.
Contextual
Simple
String
Match
Weather Info Re- trieval
Screenshots were taken by the human annotator on Microsoft Weather. Questions and answers were created by the annotator.
Contextual
String Set Equal- ity Comma
Stock Info Re-
Screenshots were taken by the human annotator on Yahoo Fi- nance. Questions and answers were created by the annotator.
Contextual
Set Equality
trieval
Game Support cation
Platform Identifi-
Screenshots were taken by the human annotator on the Steam store. Questions and answers were created by the annotator.
Structured
Exact Match, Set Equal- ity
String
Top Rated Hotel Identification
Screenshots were taken by the Book-
Contextual
String Set Equal- ity Comma
human annotator ing.com. Questions and answers were created by the annotator.
Movie Info Re- trieval
Screenshots were taken by the human annotator on the Amazon Prime Video webpage. Ques- tions and answers by the annotator.
Contextual
String Set Equal- ity Comma
created
Top Video Creator Identification
Screenshots were taken by the human annotator on YouTube. Questions and answers were cre- ated by the annotator.
Exact
Exact Match
String
Highest Discount Game Price Iden- tification
Screenshots were taken by the human annotator on the Steam store. Questions and answers were created by the annotator.
Numerical
Exact Match
String
Newspaper Page Parse And Count
Data collected from the News- paper Navigation Dataset (Lee et al., 2020). Questions and an- swers were created by the anno- tator.
Exact
Exact Match
String
Remaining Play- back Time Caleu- lation
Screenshots were taken by the human annotator on YouTube. Questions and answers were cre- ated by the annotator.
Exact
Exact Match
String
66

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Multi
Lingual Expla-
Screenshots
taken from
Open
GPT-4o Judge
Manual
user
manual located at
nation
Scooter
http://fcc..epporfFCC.
Spanish
ID/2A33E5I.CHI 1UHH888999/888
Questions and answers created
by human annnotator.
Multi Manual
Lingual
Screenshots
taken from
Open
GPT-4o as Judge
Expla- Scooter
user manual located at https//ffcc.rpporf/CCC
nation
Arabic
ID/2A3355LCHGI 11/5S//89.91
Questions and answers created by human annnotator.
Multi Manual nation
Lingual Expla- Scooter
Screenshots taken from Open
GPT-4o as Judge
user manual located at https//ficc.repor/FCCC
French
Questions and answers created
by human annnotator.
Multi Manual
Lingual Expla-
Screenshots
taken from
Open
GPT-4o as Judge
manual located at https:/fcce.repor/FCC..
nation
Scooter
Chinese
D/DA3BELCHGLUU6288999
Questions and answers created
by human annnotator.
Multi Manual nation
Lingual Expla- Scooter
Screenshots user manual located at https:/ffcc.reporFFCC.
taken from
Open
GPT-4o as Judge
Russian
ID2A3BES.CHG
Questions and answers created
by human annnotator.
Video Summary
Videos taken from WikiHow or YouTTube. Questions and an- swers created by human annno- tator.
Open
GPT-4o Judge
Video Short Title
Videos taken from YouTube. Questions and answers created by human annnotator.
Open
GPT-4o Judge
Video2notes
WikiHow or YouTube. Ques- tions and answers created by hu- man annnotator.
Open
GPT-4o Judge
Video Content
Videos and annotations were taken from the HME100k (Yuan et al., 2022) dataset. Questions and answers human annotator.
Contextual
Simple Match
String
Reasoning
adapted by
COCO
OOD Global Image Re-
Images were from COCO- O (Mao et al., 2023). Questions and answers were re-designed by the annotator manually
Structured
Jaccard Index
trieval By Query Property
67

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Places365 Similar
Images and labels were taken from the Places365 dataset (Zhou al., 2017) and adapted into questions and answers by human annotator.
MC
Exact Match
String
Scene Retrieval
Web Recommendation
Booking
Images and labels come from the SEED-Bench (Li et al., 2024b) dataset. Some images are from Yelp. Questions and annotations were adapted by a human anno- tator.
Contextual
Jaccard Case Insensitive
Index
Game Info Re- trieval
taken by the human annotator on the Epic Games Store. Questions and an- swers were created by the anno- tator.
Screenshots
Contextual
String Set Equal- ity Comma
Media Homepage Profile
Most images and labels come from the SEED-Bench (Li et al., 2024b) dataset, while one came from screenshot taken by hu- man annotator. Questions and annotations were adapted by human annotator.
Structured
Jaccard Case Insensitive
Index
Movie Retrieval Screenshots were taken by the human annotator on the Amazon Prime Video webpage. Ques- tions and answers were created by the annotator.
Contextual
String Set Equal- ity Comma
By Actor
Music Info Re-
Screenshots were taken by the human annotator on the Spotify Web Player. Questions and an- swers were created by the anno- tator.
Contextual
String Set Equal- ity Comma
trieval
Tv
Show Re-
Screenshots were taken by the human annotator the Amazon Prime Video webpage. Ques- tions and answers by the annotator.
Contextual
String Set Equal- ity Comma
trieval By Charac- ter
created
App Layout Un- derstanding Leet- code
Screenshots were taken by the human annotator on Leeteode. Questions and answers were ated by the annotator.
Exact
Fxact Match
String
App Layout Understanding Youtube
Screenshots were taken by the human annotator on YouTube. Questions and answers were cre- ated by the annotator.
Exact
Exact Match
String
App Layout Un- derstanding Ama- zon
Screenshots were taken by the human annotator on Amazon. Questions and answers were cre- ated by the annotator.
Exact
Exact Match
String
68

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
App Layout Un-
Screenshots were taken by the human annotator on Microsoft Word. Questions and answers were created by the annotator.
Exact
Exact Match
String
derstanding Word
App Layout Un- derstanding Notes
Screenshots were taken by the Exact human annotator on the Google Notes app. Questions and an- swers were created by the anno- tator.
Exact Str Match
Case Insensitive
App Layout Un- derstanding Ppt
taken by the human annotator on Microsoft PowerPoint. Questions and an- swers were created by the anno- tator.
Screenshots
Exact
Exact Match
String
App Understanding Alipay
Layout
Screenshots were taken by the human annotator the Alipay app. Questions and answers were created by the annotator.
Exact
Exact Match
String
App Layout Un- derstanding Insta- gram
Screenshots were taken by the human annotator on the Insta- gram app. Questions and an- swers were created by the anno- tator.
Exact
Exact Match
String
App Layout Un- derstanding Zoom
Screenshots were taken by the human annotator Zoom. Questions and answers were created by the annotator.
Exact
Exact Match
String
App Layout Un- derstanding Excel
Screenshots were taken by the human annotator on Microsoft Excel. Questions and answers were created by the annotator.
Exact
Exact Match
String
App Layout Understanding
Screenshots
taken by the Exact human annotator on the iPhone. Questions and answers were cre- ated by the annotator.
Exact Match
String
Iphone Settings
App Layout Understanding Tiktok
Screenshots were taken by the human annotator on TikTok app. Questions and answers were created by the annotator.
Exact
Exact Match
String
App Layout Un- derstanding Twit- ter
Screenshots were taken by the human annotator the X (for- merly Twitter) app. Questions and answers were created by the annotator.
Exact
Exact Match
String
Multilingual News Qa
Screenshots were taken by the human annotator on (formerly Twitter). Questions and answers were created by the annotator.
Contextual
Multi Ref Phrase
69

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Product Ocr Qa
Images were taken from various
Exact
Exact Match
String
websites. Questions and answers created by the annotator.
Research Website Parsing Blogpost
Screenshots were taken of vari- ous ML research websites, Ques- tions and answers were created by the annotator.
Contextual
Multi Ref Phrase
Research Website
taken of vari- ous ML research websites. Ques- tions and answers were created by the annotator.
Screenshots
Contextual
Multi Ref Phrase
Parsing Home-
page
Research Website
Screenshots were taken of vari- ous ML research websites. Ques- tions and answers were created by the annotator.
Contextual
Multi Ref Phrase
Parsing Publica-
tion
Gui Chat Easy
Images and annotations were adapted from the GUI Chat dataset (Chen et al., 2024c) by the human annotator into open-ended question.
Open
GPT-4o as Judge
Gui Chat Hard
Images and annotations were adapted from the GUI Chat dataset (Chen al., 2024c) by the human annotator into an open-ended question.
Open
GPT-4o as Judge
Realworld
Qa
Images and annotations were adapted from the RealWorldQA benchmark (xAI, 2024) by the human annotator into an open- ended question. The translation requirement was added by the human annotator.
Contextual
Multi Ref Phrase
En2en
Star Object Inter- action Video
Videos and annotations were adapted from the STAR bench- mark (Wu et al., 2024) by the hu- annotator into questions and answers.
Contextual
Multi Ref Phrase
Funqa pected Magic Video
Unex- Action
Videos and annotations were adapted from the FunQA bench- mark (Xie et al., 2023) by the human annotator into being an open-ended question.
Open
GPT-4o Judge
Activitynetqa
Images and annotations were adapted from the ActivityNetQA benchmark (Yu al., 2019) by the human annotator into being an open-ended question.
Open
GPT-4o Judge
70

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Funqa
Unex- Action
Videos and annotations were adapted from the FunQA bench- mark (Xie et al., 2023) by the human annotator into being an open-ended question.
Open
GPT-4o Judge
pected
Creative Video
Nextqa Me
Images and annotations were adapted from the NExTQA benchmark (Xiao et al., 2021) by the human annotator into questions and answers.
MC
Exact Match
String
Video Qa
Videos taken from YouTube. Questions and answers created by human annnotator.
Open
GPT-4o as Judge
Nextqa Oe
Images and annotations were adapted from the NEXTQA benchmark (Xiao et al., 2021) by the human annotator into being open-ended question.
Open
GPT-4o as Judge
Funqa
Unex- Action
Videos and annotations were adapted from the FunQA bench- mark (Xie et al., 2023) by the human annotator into being open-ended question.
Open
GPT-4o Judge
pected
Humor Video
Multilingual
Screenshots were taken by the human annotator the Amazon Prime Video webpage. Ques- tions and answers were created by the annotator.
Structured
Exact Match, String Match
String Simple
Movie Info
Parsing
Movie Info Pars-
Screenshots were taken by the human annotator on the Amazon Prime Video webpage. Ques- tions and answers were created by the annotator.
Structured
Exact Match
String
ing
Stock Info Parsing
Screenshots were taken by the human annotator on Yahoo Fi- nance. Questions and answers were created by the annotator.
Structured
Exact Match
String
Music Info Pars-
Screenshots were taken by the human annotator on the Spotify Web Player. Questions and an- swers were created by the anno- tator.
Structured
Exact Match
String
ing
Multilingual Game Info Pars-
Screenshots were taken by the human annotator on the Epic Games Store. Questions and an- swers were created by the anno- tator.
Structured
Exact Match
String
ing
Ocr Article Au- thors
Screenshots taken of various aca- demic papers. Questions and swers created by human annota-
Structured
Simple Match
String
71

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Youtube
Video Videos taken from YouTube. Questions and answers created by human annnotator.
Structured
Exact Match
String
Info Parsing
Tv Show Info
Screenshots were taken by the human annotator on the Amazon Prime Video webpage. Ques- tions and answers were created by the annotator.
Structured
Simple Match
String
Parsing
Ocr
Resume
Resumes taken from various per- sonal websites. Questions and answers were created by the an- notator.
Contextual
String Set Equal- ity Line Break
School Plain
Image Translation En2en
Images were collected from vari- ous sources, including academic papers, news articles, shopping receipts, etc. The annotations are obtained by GPT-4o translation followed by human check.
Contextual
Gleu Cn
Booking
Web
Images and labels come from the SEED-Bench (Li et al., 2024b) dataset. Some images are from Yelp. Questions and annotations were adapted by a human anno- tator.
Structured
Exact Match
String
Rating
Weather
Info Images collected from the Microsoft Weather by taking screenshots. Questions and an- swers were designed by the an- notator.
Structured
Exact Match
String
Parsing
Game Info Pars-
Screenshots were taken by the human annotator on the Epic Games Store. Questions and an- swers were created by the anno- tator.
Structured
Exact Match
String
ing
Weather Map Cli- mate Type Tem- perature Parsing
One of the examples comes from the SEED-Bench Plus bench- mark (Li et al., 2024b). The rest of the images were col- lected from various online web- sites. Questions and annotations were adapted by human anno- tator.
Structured
Exact Match
String
Hotel Booking Confirmation
Screenshots were taken by the human annotator on Book- ing.com. Questions and answers were created by the annotator.
Structured
Exact Match
String
Parsing
72

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Entertainment Web Game Style
Some of the examples come from the SEED-Bence mark (Li et al., 2024b). The rest of the screenshots were taken the Steam store. Questions and annotations were adapted by human annotator.
Structured
Exact
Str Match
Plus bench-
Case Insensitive,
Exact String
Match
Planning
Wikihow
Com-
Data collected from website, and the questions and answers are de- signed by human annotator
Open
GPT-4o as Judge
plex Task Com-
pletion
VIn
Identify
Data collected from RxR dataset (Ku al., 2020), the question and answer are adapted to select the robot that should
Exact
Exact Match
String
Robot
execute the instruction
VIn English Next Step
Data collected from RxR dataset (Ku et al., 2020), the question and answer are adapted by human annotator
Contextual
Simple Match
String
VInqa Egocentric Navigation Video
Data collected from VLN- CE (Krantz et al., 2020) and the task is adapted from MVBench (Li et al., 2024c), the question and answer adapted by human annotator
Contextual
Simple Match
String
VIn Identify Lo-
Data collected from RxR dataset (Ku al., 2020), the question and answer are adapted by human annotator
Structured
Exact Match
String
cation
VIn Tegulu Next
Data collected from RxR dataset (Ku al., 2020), the question and answer are adapted by human annotator
Structured
Simple Match
String
Step
VIn Hindi Next
Data collected from RxR dataset (Ku et al., 2020), the question and answer are adapted by human annotator
Contextual
Simple Match
String
Step
App
Interac- Operations
Data collected from application screenshots by human annotator, and the questions and answers are designed by human annotator
MC
Exact Match
String
tive
Instagram
App Interac- Operations
Data collected from application screenshots by human annotator, and the questions and answers are designed by human annotator
MC
Exact Match
String
tive Leetcode
73

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Gui Act
Web
Data collected from webpage screenshots by human amnotator, and the questions and answers bounding boxes are annotated by human annotator
Structured
Exact Match,
String Xml
Multi
Nbbox lou Single
App Interactive Operations Ppt
Data collected from application screenshots by human annotator, and the questions and answers are designed by human annotator
MC
Exact Match
String
Gui Act Mobile Swipe
Data collected from application screenshots by human annotator, and the questions and answers bounding boxes are annotated by human annotator
Structured
Xml Norm Point Distance
App Interactive Operations Excel
Data collected from application screenshots by human annotator, and the questions and answers designed by human annotator
MC
Exact Match
String
Gui Act Mobile
Data collected from application screenshots by human annotator, and the questions and answers bounding boxes are annotated by human annotator
Numerical
Xml Norm Point In Bbox
Tap
App Interactive Operations Alipay
Data collected from application screenshots by human annotator, and the questions and answers designed by human annotator
MC
Exact Match
String
Gui Act Web Sin-
Data collected from webpage screenshots by human annotator, and the questions and answers bounding boxes are annotated by human annotator
Numerical
Xml Nbbox Iou Single
gle
App tive Twitter
Interac- Operations
Data collected from application screenshots by human annotator, and the questions and answers are designed by human annotator
MC
Exact Match
String
App Interactive
Data collected from application screenshots by human annotator, and the questions and answers designed by human annotator
MC
Exact Match
String
Operations Word
App Interac- Operations
Data collected from application screenshots by human annotator, and the questions and answers are designed by human annotator
MC
Exact Match
String
tive
Iphone Settings
App Interactive Operations Tiktok
Data collected from application screenshots by human annotator, and the questions and answers are designed by human annotator
MC
Exact Match
String
74

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
App
Interactive
Data collected from application screenshots by human annotator, and the questions and answers are designed by human annotator
MC
Exact Match
String
Operations Notes
Interactive Operations Zoom
App
Data collected from application screenshots by human annotator, and the questions and answers are designed by human annotator
MC
Exact Match
String
App Interac- Operations
Data collected from application screenshots by human annotator, and the questions and answers designed by human annotator
MC
Exact Match
String
tive Amazon
Web Action
Data collected from Visual- WebBench (Liu et al., 2024a), and the questions and answers adapted by human annotator
MC
Exact Match
String
Grounding
App tive Youtube
Interac- Operations
Data collected from application screenshots by human annotator, and the questions and answers are designed by human annotator
MC
Exact Match
String
Calendar ule Suggestion
Sched-
Data collected from Google Cal- endar by human annotator, and the questions and answers are designed by human annotator to identify all possible starting times for a meeting within a specified time range and duration
Contextual
Set Equality
Planning Barman
Visual
Data collected from Planning Domain Definition Language (PDDL) of Barman, and the questions and answers are adapted to match the transitions from init state to goal state
Structured
Symbolic ning Test
Plan-
Planning Floortile
Visual
Data collected from website, and the questions and answers are adapted to match the transitions from init state to goal state
Structured
Symbolic ning Test
Plan-
Planning Storage
Visual
Data collected from website, and the questions and answers are adapted to match the transitions from init state to goal state
Structured
Symbolic ning Test
Plan-
Planning Screen- shot Grippers
Data collected from website, and the questions and answers are adapted to match the transitions from init state goal state
Structured
Symbolic Plan-
ning Test
Planning Visual Blocksworld
Data collected from website, and the questions and answers are adapted to match the transitions from init state goal state
Structured
Symbolic ning Test
Plan-
75

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Planning Screen-
Data collected from Planning Domain Definition Language (PDDL. of Barman, and the questions and answers are adapted to match the transitions from init state to goal state
Structured
Symbolic
Plan-
shot Barman
ning Test
Planning Screen- shot Termes
Data collected from website, and the questions and answers are adapted to match the transitions from init state to goal state
Structured
Symbolic ning Test
Plan-
Planning Screen- shot Floortile
Data collected from website, and the questions and answers are adapted to match the transitions from init state goal state
Structured
Symbolic ning Test
Plan-
Planning Screen- shot Blocksworld
Data collected from website, and the questions and answers are adapted to match the transitions from init state to goal state
Structured
Symbolic Plan-
ning Test
Planning Screen- shot Storage
Data collected from website, and the questions and answers are adapted to match the transitions from init state to goal state
Structured
Symbolic Plan-
ning Test
Planning Termes
Visual
Data collected from website, and the questions and answers are adapted to match the transitions from init state to goal state
Structured
Symbolic Plan-
ning Test
Planning shot Tyreworld
Screen-
Data collected from website, and the questions and answers are adapted to match the transitions from init state to goal state
Structured
Symbolic Plan-
ning Test
Planning Grippers
Visual
Data collected from website, and the questions and answers adapted to match the transitions from init state goal state
Structured
Symbolic Plan-
ning Test
Logical Reason- ing Find Odd One Out
Data collected from website, and the questions and answers are adapted to match strings
Structured
Dict Equality, Ex- act String Match
Logical Reason- ing Fit Pattern
Data collected from Log- icVista (Xiao al., 2024), and the questions and answers are adapted by human annotator
MC
Exact Match
String
Perception-Tesd Test Object Shuffle Video
Data collected from VLN- CE (Krantz et al., 2020)
MC
Simple Match
String
and the task adapted from MVBench (Li et al., 2024e), the question and answer are adapted into single choice by human annotator
76

---

and Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Chess Puzzles
Data collected from Lichess, and the questions and answers are adapted match strings
Structured
Set Equality
Checkmate
Chess
Puzzles
Data collected from Lichess, and the questions and answers are adapted match strings
Structured
Set Equality
Equality
Bridge Strategies
Data and answer are collected from Bridge Master 2000
Open
GPT-4o as Judge
Expert
Chess Crushing
Puzzles
Data collected from Lichess, and the questions and answers are adapted match strings
Exact
Exact Match
String
Chess Puzzle Sin- gle Step
Data collected from Lichess, and the questions and answers are adapted to match strings
Exact
Exact Match
String
Chess Find Legal Moves
Data collected from game po- sitions of games in the 2024 FIDE Candidates tournament, and the questions and answers are adapted to match strings
Exact
Chess Move List
Jaccard
Index. String
Exact
Match
Bridge Strategies Advanced
Data and answer are collected from Bridge Master 2000
Open
GPT-4o Judge
Chess Winner Identification
Data collected IsoBench (Fu et al., 2024b), and the questions and answers are adapted by human annotator
from
Exact
Exact Match
String
Bridge Strategies Worldelass
Data and answer are collected from Bridge Master 2000
Open
GPT-4o as Judge
Mahjong
Data collected from website and screenshot of MajSoul, and the answer are annotated by human annotator
Exact
Exact Match
String
Chess Endgames
Sygyzy
Endgames created by human annotator and data collected from ttns://szzygy-table..im and the questions and answers adapted to match Jaccard in- dex
Exact
Chess Move List
Jaccard Exact Match
Index, String
Go Capture Stone
Data collected https://online-go0 wtw.o/iiiiingo..o/laaaamam go..omm/eam- to-play-go/capture https:/ffrrums.oniine vo.com//tcappureego problems/31531/ and the questions and answers are
from Exact
Exact Match
String
adapted to match strings
Bongard Problem
Data collected http://www..cebboo//weleo...ppp
from Contextual
String Set Equal- ity Comma
and
http::/wwww..aanaaalcccccc/////p
77

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Number Puzzle
Data
collected
from
Exact
Exact Match
String
Kakuro 5x5
http:///wzzyddd..oomaaukaak and the questions and answers are adapted to match strings
Mensa lq Test
Data collected from website, and the questions and answers are adapted to match dict equality
Structured
Dict Equality
Arc Agi
Data collected
from
Exact
Exact Match
String
https//arcprize.rr//hhe the task is adapted from In- telligence (Chollet, 2019), the question and answer are adapted into a grid of digits by human annotator
and
Mnist Pattern
from MNIST (Deng. 2012), and the questions and answers are adapted to match strings
Data collected
Numerical
Exact Match
String
Number Sudoku
Puzzle
Data collected from puzzles.ca, and the questions and answers are adapted match strings
Contextual
Simple Match
String
Move Pos To Pos Hanoi Polc
Shortest paths derived from a di- agram found on website and the questions and answers are cre- ated to match strings and the longest common move prefix
Structured
Exact Match, Common
String Longest List
Prefix Ratio
Pictionary toon Guess
Car- Drawing
Data collected from An early evaluation of gpt-4v (ision) (Wu et al., 2023), the question and an- swer are adapted to match strings by human annotator
Exact
Exact Str Match Case Insensitive
Pictionary nese Food Img2en
Chi-
Data collected from website, and the questions and answers are adapted match strings
Exact
Exact Str Match Case Insensitive
Pictionary Doodle Guess
Data collected from website, and the questions and answers are adapted to match strings
Exact
String Match
Exact
Pictionary Skribbl lo
Data collected from screenshots collected by human annotator skribbl.io and the questions and answers adapted to match strings
Exact
Exact Str Match Case Insensitive
Pictionary Genai Output Chinese
Data collected from screenshot of website, and the questions and answers are adapted to match strings
Exact
Exact Match
String
Annoying Word
Data collected from various web- sites, and the answers tated by human annotator
Contextual
Dict Jaccard Agg Jaccard
Search
78

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Logical Reason- ing 2d Views Of 3d Shapes
Data collected from website, and the questions and answers are adapted match strings
Structured
Dict Equality
Maze 2d 8x8
Data generated from Exact
Exact Str Match
Case Insensitive
and the questions and answers are adapted to match strings
Crossword 5x5
Mini
Data collected from website, and the questions and answers are adapted to match strings
Structured
Dict Exact Str
Match Agg Recall
Rebus
Data collected from website, and the questions and answers are adapted to match strings
Contextual
Simple Match
String
Icon Arithmetic Puzzle
Data collected from An carly evaluation of gpt-4v (ision) (Wu et al., 2023), the question and an- adapted to match strings by human annotator
Structured
Exact Match, Equality
String Sequence
Iq
Test
Open
Data and answers are collected from website
Open
GPT-4o as Judge
Ended
Ball Cup Swap 3
Screenshots taken from video and edited together using images found online, and the questions
MC
Exact Match
String
and answers are adapted match strings
Logical Reason- ing 2d Folding
Data collected from website, and the questions and answers are adapted to match strings
MC
Exact Match
String
Perception Test Video Character Order
Data collected from Perception Test (Patraucean et al., 2024) and the task is adapted from MVBench (Li et al., 2024c), the question and answer are adapted into single answer string by hu- man annotator
Contextual
Simple Match
String
Comic Page Or-
Data collected from website
Contextual
Sequence Equal-
dering
ity
Recipe Image Or- dering
Data collected from website
MC
Sequence Equal- ity
Coding
Code Translation Easy
Data and test cases collected from Pintia
Structured
Program Judge
Code Translation
Data collected from xCodeEval split (Khan et al., 2023), and test cases are annotated by human
Structured
Program Judge
Python
Code
Translation
Data and test cases collected from Pintia
Structured
Program Judge
Hard
79

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Code Translation
Data and test cases collected from Pintia
Structured
Program Judge
Advanced
Symbolic Graph- Programs Computer Aided
Data and answer are collected Contextual from SGP-Bench (Qiu et al., 2024)
Multi Ref Phrase
ics
Design
Symbolic
Graph- Programs Vector
Data and answer are collected Contextual from SGP-Bench (Qiu et al., 2024)
Multi Ref Phrase
ics
Scalable
Graphics
Webpage
Code Understanding
Data are collected from website, and the question and answer are adapted for string match
MC
Exact Match
String
Code Add Tag
Data collected from xCodeEval (Khan et al., 2023), the question and answer are adapted to match code tag
Contextual
Set Equality
Media mend Stackoverflow
Recom- Solutions
Data are collected from Stack Overflow Website, and the ques- tion and answer are adapted to match string
MC
Exact Match
String
Flowchart Generation
Code
Data collected from website, and the question and answer are designed by human annotator
MC
Exact Match
String
Code Compare
Solution
Data collected from SGP-
Exact
Exact Match
String
Bench (Qiu et al., 2024), and the question and answer are adapted for string match
Code Match Prob-
Data collected from SGP- Bench (Qiu et al., 2024), and the question and answer are adapted to match code
Exact
Exact Match
String
lem
Code
Visual- Output
Data are collected from website, and the question and answer are designed by human annotator
MC
String Set Equal- ity Comma
ization
Understanding
Code Output Re-
Data collected from San- Foundry MCQs, and the question and answer designed by hu- man annotator
Exact
Code Result Exact Str Match
sult
Code Execution
Data collected from execution- v2 (Jain al., 2024a), the ques- tion and answer adapted to match string
Contextual
Simple Match
String
Code Retrieval
Data collected from SGP- Bench (Qiu al., 2024), and the question and answer are adapted match string
Exact
Exact Match
String
80

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Table2latex Com-
Data collected from SGP- Bench (Qiu et al., 2024), and the question and answer are adapted for LLM Judge
Structured
GPT-4o Judge
plex
Ocr
Table
To
Data are collected from website, and the question and answer are designed by human annotator
Structured
Simple Match
String
Html
Ocr
Table
To
Data collected from website, and the question and answer are designed by human annotator
Structured
Simple Match
String
Markdown
Ocr Math Text La- tex
Data are collected from website, and the question and answer are designed by human annotator to match text with KT
Contextual
Text With Latex Expr Equality
Ocr Math Equa- tion
Data collected from website, and the question and answer are designed by human annotator to match ВТК
Contextual
Latex Expr Equal- ity
Latex
Com- Formula
Data are collected from latex- formulas and TexTeller, and the question and answer are de- signed by human annotator
Structured
Latex Expr Equal- ity
plex Convertion
Oer Table To La- tex
Data are collected from website, and the question and answer designed by human annotator
Structured
Simple Match
String
Ocr Table To Csv
Data are collected from website, and the question and answer are designed by human annotator
Structured
Simple Match
String
Code
Program- Test Easy
Data and test cases collected from Pintia
Structured
Program Judge
ming
Code
Program-
Data and test cases are collected from Pintia
Structured
Program Judge
ming
Test Hard
Code
Program- Test
Data and test cases are collected from Pintia
Structured
Program Judge
ming
Ad-
vanced
Code ming
Program- Extremely
Data and test cases are collected from Pintia
Structured
Program Judge
Hard
Visualization With Code
Data are collected from website, and the question and answer are designed by human annotator
Structured
GPT-4o Judge
Stackoverflow Debug Qa
Data collected from Stack Overflow Website, and the ques- tion and answer are adapted match string
Structured
Exact
Str Match
Case Insensitive.
Exact String
Match
Code Error Line Identification
Data collected from Pintia, and the question and answer are adapted to match string
MC
Exact
String
Match
81

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Perception
Visual Correspon- dence In Two Im- ages
Images are from BLINK (Fu et al., 2024c). Annotator man- ually added one more reference point per sample and designed structured answers
Structured
Dict Equality
2D Image Jigsaw Puzzle Easy
Images created by playing the online Jigsaw simulator and tak- ing screenshots
Structured
Dict Exact Str
Match Agg Recall
Adapted Cvbench Distance
Data collected from CV-Bench's distance split (Tong et al., 2024), and adapted into exact text an- swer
Exact
Exact Match
String
Geometry Plot Position Relation- ship
Data collected from Internet. Question and answers were de- signed by the annotator
Exact
Exact Match
String
Video Grounding Spatial
Videos collected from Vi- dOR (Shang al., 2019). Re-designed questions and answers for this specific task
Contextual
Simple Match
String
Adapted Cvbench Relation
Data collected from CV-Bench's relation split (Tong et al., 2024), and adapted into exact text an-
Exact
Exact Match
String
Egocentric Spatial Reasoning
Data are collected from Epic- Kitchen (Damen et al., 2018) and the Internet. Questions and an- swers are adapted for contextual formatted output
Contextual
Multi Ref Phrase
Trance Physics Reasoning Basic
Data are collected from Exact Trance (Hong al., 2023) by specifically picking up sam- ples with the casiest settings. Questions and answers are re-designed for this specific task
Exact Match
String
CLEVER Moving Direction Video
Video data are collected from MVBench (Li et al., 2024e). Questions and answers are adapted for the contextual formatted output format
Contextual
Multi Ref Phrase
Trance Physics Reasoning Event
Data are collected from al., 2023)
MC
Set Equality
Trance (Hong by selecting settings where objects are moved. Questions and answers are re-designed for indicating changed objects
3D Fragments Understanding
We write rendering scripts to Numerical produce the data from the assets of the Break Bad dataset (Sellán al., 2022)
Simple Match
String
82

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Physical Property Reasoning
Images collected from the In- ternet, questions and answers are designed by annotator
Contextual
Simple Match
String
CIEVRER
Images are collected from CLEVRER (Yi et al., 2019), questions and answers are re-designed for testing the
Numerical
Exact Match
String
Physics
understanding of physical status
CIEVRER
Video Ob-
The videos are collected from MVBench (Li et al., 2024e), the questions and answers are adapted test the understanding of physical property and dynam- ics
Contextual
Multi Ref Phrase
Moving
ject
Property
Recognition
Trance Physics Reasoning View
Data are collected from Trance (Hong et al., 2023) by selecting the most challeng- ing settings (objects moved, and two states are captured by different cameras). Questions and answers are re-designed for indicating changed objects
MC
Set Equality
Photoshop Opera- tion
Images are collected from the Web, questions and answers de- signed by annotator
Structured
Jaccard Index
Relative
Re- Of
Images come from BLINK (Fu al., 2024c), the annotator added one more point per image and converted the task into re- flectance sorting task
Structured
Sequence Equal- ity
flectance
Different Regions
Autonomous
Images are collected from the In- ternet, questions and answers are designed by annotator
Exact
Exact Str Match Case Insensitive
Driving Scene Analysis
Functionality Matching Different Objects
The images come from BLINK (Fu et al., 2024c).
Structured
Dict Equality
In
The annotator manually added one ref point per image to augment the task
NLVR2 Two Im- age Compare QA
Images are collected from NLVR2 (Suhr & Artzi, 2019).
MC
Multi Ref Phrase
Questions and answers re- designed by the annotator
Egocentric Analy- sis Single Image
The images are collected from Epic-Kitchens (Damen et al., 2018). Questions and answers are re-designed by the annotator
Exact
Exact Match Insensitive
String Case
CIEVR Object Existence Video
Videos collected from MVBench (Li et al., 2024c).
MC
Simple Match
String
Questions and answers are slightly adapted
83

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
SNLI-VE Visual Entailment
Data collected and converted Exact from SNLI-VE dataset (Xie
Exact Match
String
et 2019)
OCR Open-ended QA
Images collected from the Inter- net. Questions and answers made up by the annotator for the open-
Open
GPT-4o Judge
ended output format
Super Scene
CIEVR Under-
Images collected from Su- perCLEVR (Li et al., 2023b). Questions and answers re-
Contextual
Multi Ref Phrase
standing
designed by the annotator
Visual Dialog Im- age Guessing
Images are collected from Visual Dialog dataset (Das et al., 2017). Questions and answers are de- signed by the annotator
MC
Exact Match
String
Semantic Match- ing Of Two Im- ages
Images come from BLINK dataset (Fu et al., 2024c). The annotator augmented the data by adding one more ref point and re-designed the answer
Structured
Dict Equality
Recover Masked Word In Figure
The annotator took screenshots from few public papers on arXiv and designed the question- answer pairs
Contextual
Simple Match
String
Graph Interpreta- tion
The images of line/dot graphs are collected from the Internet, and the annotator created the ques- tion and open-ended reference answer
Open
GPT-4o as Judge
Science Explanation
Figure
The images of science figures are collected from the Internet, and the annotator created the ques- tion and open-ended reference answer
Open
GPT-4o Judge
Bar Chart Inter- pretation
The images of bar graphs are collected from the Internet, and the annotator created the ques- tion and open-ended reference answer
Open
GPT-4o Judge
Electricity Load Estimate Plot
The temporal data were collected from Informer (Zhou al., 2021) and AutoFormer (Wu et al., 2021). The annotator re-processed the data to design more specific task
Numerical
Normalized RMSE
Average Humidity Estimate Plot
The temporal data were col- lected from AutoFormer (Wu et al., 2021). The annotator re- processed the data design more specific task
Numerical
Normalized RMSE
84

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Exchange Rate
The temporal data were collected from Lai et al. (2018) and Aut- oFormer (Wu al., 2021). The annotator re-processed the data to design
Numerical
Normalized Rmse
Estimate Plot
specific task
Road Map Find Be-
The road map images were Exact collected from Beed-Bencn (Li
Exact Match
String
Highway tween Two Place
et al., 2024c) and the Internet.
Questions and answers are de- signed by the annotator
Transit Map Inter- section Points
The transit map images were Structured collected from Seed-Benen (Li et al., 2024c) and the Internet. Questions and answers are de- signed by the annotator
Exact Match, Sequence Equality Case Insensitive
String
Panel Images Sin- gle Question
Panel images from (Fan al., 2024). Ques- tions and answers designed by the annotator
collected
MC
Exact Match
String
Knowledge Graph Understanding
The large knowledge graph im- age was collected from the Inter- net. Questions and answers were designed by the annotator
Contextual
Set Equality
Panel Images
Panel images were collected from (Fan al., 2024). Ques- tions and answers were designed by the annotator
Structured
Exact Match
String
Multi Question
Mindmap Ele- Mindmap images were collected from Seed-Benen (Li et al., 2024c) and the Internet. Ques- tions and answers are designed by the annotator
Structured
Set Equality Case Insensitive
ments Parsing
Dvqa
Images were collected from Dvça dataset (Kafle et al., 2018). Questions and answers were re-designed by the annotator
Numerical
Multi Ref Phrase
Figureqa
Images were collected from Fig- ureQA dataset (Kahou et al., 2017). Questions and answers were re-designed by the annota- tor
MC
Multi Ref Phrase
Map Diagram Qa
Images were collected from MapQA dataset (Chang al., 2022). Questions and an- swers were re-designed by the annotator
Contextual
Simple Match
String
Chart Vga
Data were collected from Math- Vista (Lu et al., 2023) (statistics subset) and converted into specific task
Numerical
General Single Numerical Match
85

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Photo Sharing Im- age Retrieval
from the Pho- toChat (Zang et al., 2021) dataset. Questions and answers are designed by the annotator
Images
MC
Exact Match
String
Multi Load Type Prediction From
The temporal data were collected from Informer (Zhou al., 2021) and AutoFormer (Wu et al., 2021). The annotator re-processed the data to design more specific task
MC
Sequence curacy Insensitive
Ac-
Case
Plot
Stock Price Future Prediction
The annotator downloaded data from Yahoo! Finance's API, and processed data to design this task
Contextual
Normalized Rmse
Traffic Future Pre- diction From Line Plot
The temporal data were col- lected from AutoFormer (Wu et al., 2021). The annotator re- processed the data to design more specific task
Numerical
Normalized Rmse
Electricity Plot Future Prediction
The temporal data lected from AutoFormer (Wu et al., 2021). The annotator re- processed the data to design more specific task
col-
Numerical
Normalized Rmse
Ratio Future
The temporal data were col- lected from AutoFormer (Wu et al., 2021). The annotator re- processed the data design more specific task
Numerical
Normalized Rmse
Prediction
Paper Vqa
The annotator took high- resolution screenshots of a few papers on arXiv, and designed the questions and answers
Contextual
Simple Match
String
Doc Vqa
Data and open-ended QA pairs were converted from DocMatix (HuggingFaceMM4, 2024)
Open
GPT-4o as Judge
FunSD Document
Images were collected from FunSD (Jaume et al., 2019). Questions and answers were designed by annotator
Contextual
Simple Match
String
Qa
OCR Article Jour-
The article screenshots taken from various websites. Questions and answers were cre- ated by the annotator
Contextual
Simple Match
String
nal
IAM Line
Ocr
Images were collected from IAM handwritten
Structured
String Normal- ized Similarity Damerau Leven-
Exact Match,
And Locate
database (Marti & Bunke, 1999). Questions and answers were re-designed by the annotator
shtein
86

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
OCR Resume Ex- perience Plain
The resume screenshots were taken from various websites. Questions and answers were ated by the annotator
Contextual
String Set Equal- ity Line Break
Newspaper Ocr In Query Box
Images were collected from The Newspaper Navigator Dataset (Lee et al., 2020). Ques- tions and answers were adapted by the annotator into simple string answer format.
Contextual
Simple Match
String
OCR Skill Plain
Resume
The article screenshots were taken from various websites. Questions and answers were cre- ated by the annotator
Contextual
String Set Equal- ity Line Break
OCR Resume Employer Plain
The article screenshots were taken from various websites. Questions and answers were cre- ated by the annotator
Contextual
String Set Equal- ity Line Break
Finance Table Un- derstanding
Images were collected from MMMU (Yue et al., 2024a). Questions and answers were adapted by the annotator into direct numerical output format
Numerical
Exact Match
String
Monthly Weather Days Count
Images were collected from the Microsoft Weather by taking screenshots. Questions and an- swers designed by the an- notator.
Structured
Exact Match
String
Table standing Complex Question Answer- ing
Under-
Tables were collected from WikiTableeuustions (Pasupat & Liang, 2015) and TabFact (Chen et al., 2019). Questions and answers were designed by the annotator
Contextual
Simple Match
String
Table Understand- ing Fetaqa
Data were collected and con- verted from FetaQA (Nan et al., 2022)
Open
GPT-4o as Judge
Table Understand- ing Fact Verifica- tion
Tables were collected from WikiTableQuestions (Pasupat & Liang, 2015) and TabFact (Chen et al., 2019). Questions and answers were designed by the annotator
Contextual
Dict Precision
Electricity Future Prediction From Table
The temporal data were col- lected from AutoFormer (Wu et al., 2021). The annotator re- processed the data to design more specific task
Numerical
Normalized Rmse
87

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Video Detail De- scription
Video and description data were collected from VideoDetailCap- tion (Maaz et al., 2023) and con- verted into a specific task
Open
GPT-4o Judge
Guess Image Gen- eration Prompt
Examples were collected from various online exxt-to-image gen-
Open
GPT-4o as Judge
eration demos
Docci Image scription Long
De-
Data
collected from Open DOCCI (Onoe al., 2024)
GPT-4o Judge
Tweets Caption- ing
The annotator collected the data from by taking sereenshots and and the texts
Open
GPT-4o as Judge
Image Captioning With Additional Requirements
Images were collected from ious sources on the Web. The annotator used Claude 3.5 Son- net to generate reference answers and manually polished them
Open
GPT-4o Judge
Ad Count Detec- tion
Image were collected from var- ious websites by taking screen- shots. Questions and answers created by the annotator
Numerical
Exact Match
String
Adapted Cvbench Count
Data collected from CV- Bench's counting split (Tong et al., 2024) and adapted into a specific task by rewriting the question-answer pairs
Numerical
Exact Match
String
Av Vehicle Multi- view Counting
Images were collected from the nuScenes (Caesar et al., 2020) dataset. The annotator designed the questions and implemented a script to generate the answers from the raw annotation
Numerical
Exact Match
String
Counting Image
Multi
Data were collected from Man- tis (Jiang et al., 2024a) and adapted into direct numerical an- swer
Numerical
Exact Match
String
Av Human Multi- view Counting
Images collected from the nuScenes (Caesar et al., 2020) dataset. The annotator designed the questions and implemented script to generate the answers from the raw annotation
Numerical
Exact Match
String
Shape Composi- tion Shapes
Images were made by the anno- tator using Canva. Questions and answers were created by the an- notator
Structured
Positive Int Match
Counting Image
Single
Data were collected from Man- tis (Jiang al., 2024a) and adapted into direct numerical an- swer
Numerical
Exact Match
String
88

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
CLEVRER Video
Video data collected from MVBench (Li et al., 2024c). Questions and answers adapted for the direct numerical output
Numerical
Exact Match
String
Moving
Object
Count
Shape Composi- tion Colours
Images were created by the anno- tator using Canva. Questions and answers were created by the an- notator
Structured
Positive Int Match
Face Matching
Identity
collected from CelebA (Liu et al., 2015). Ques- tions and answers re-designed by the annotator for this specific task
Images
Numerical
Set Equality
Rocks Samples Identify
Images, questions, and answers were collected from the Web by the annotator
Contextual
Simple Match
String
Animal Pose Esti- mation
Images were collected from AP- 10K (Yu al., 2021). The anno- tator implemented script to pro- duce the answer from raw anno- tations for this task
Numerical
Sequence Coords
Similarity
License Plate
Images collected from the Web. Questions and answers created by the annotator
Exact
Exact Str Match Case Insensitive
Recognition
Image Recognition
Style
Images were collected from the Web. Questions and answers were created by the annotator
Exact
Exact Str Match
Case Insensitive
Long String Letter Recognition
Data were designed by the amno- tator and generated automatically with code
Exact
Exact Match
String
COCO Object De- tection By Query Property
Images were from MS- COCO (Lin et al., 2014). Questions and answers were
Numerical
Exact Match, Nbbox Iou Tuple
String
re-designed by the annotator and adapted manually
Widerface Face Count And Event Classification
Images were collected from WiderFace (Yang et al., 2016). Questions and answers were designed and produced by the annotator
Structured
Exact Match, Simple String Match
String
Handwritten
Data were collected from Contextual
Latex Expr Equal- ity
Math Expression Extraction
HME100K (Yuan et al., 2022)
Geometry soning Circled Letter
Rea-
Image were collected from Rah- manzadehgervi et al. (2024) are manually created. Questions and answers were re-designed by the annotator
Structured
Exact Match, Equality
String Sequence
89

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Av Multicamera
Images collected from the nuScenes (Caesar et al., 2020) dataset. The annotator designed the questions and implemented script to generate the answers from the raw annotation
Numerical
Nbbox quence
lou Se-
Tracking Predict
Bbox
ASCII Art Under- standing
Data and annotations were col- lected and created by the annota- for from various online resources
MC
Exact Match
String
Face Detection
Keypoint
Raw data were from CelebA (Liu et al., 2015). The annotator wrote script to produce the answers for this task
Structured
Sequence Coords Similarity
Extract Headline
Webpage
Images were collected from VisualWebBench (Liu et al., 2024a). Questions and answers were adapted by the annotator
Contextual
Simple Match
String
Waldo
Images and annotations were collected and created by the an- notator using various resources on the Web
Structured
Dict Nbbox Iou Tuple Agg Jac- card
Geographic mote
Re-
Images and annotations were collected and converted from SATIN (Roberts al., 2023)
Contextual
Sequence Equal- ity
Sensing Land Cover
Signboard Identi- fication
Images were collected from the Internet. The annotator created the question -answr pairs
Contextual
Simple Match
String
Long String Num- ber Recognition
Data were designed by the anno- tator and generated automatically with code
Exact
Exact Match
String
Waybill Num- Sequence
Images collected from the Internet. The annotator created the question-answer pairs
Contextual
Simple Match
String
ber Extraction
Single Person Pose Estimation
hello, this is Source Description
Structured
Sequence Coords
Similarity
COCO Person Detection
Images were from MS- COCO (Lin et al., 2014).
Numerical
Exact Match, Nbbox Iou Tuple
String
Questions and answers
re-designed by the annotator and adapted with script
Places365 Scene
Images were collected from Places365 (Zhou et al., 2017). Questions and answers were
Exact
Exact Match
String
Type Classifica-
tion
re-designed and generated by the
annotator
Visual Prediction
Images were collected using screenshots from arXiv papers' qualitative results. Questions and answers were created by the an- notator
MC
Sequence Equal-
Rater Openable
ity
Part Segmentation
90

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Visual Prediction
collected using screenshots from qualitative re- sults from the arXiv papers. Questions and answers were cre- ated by the annotator
Images
MC
Sequence curacy
Ac-
Rater Panoptic
Case
Segmentation
Insensitive
Visual Prediction Rater Semantic Segmentation
Images were collected using screenshots from the qualita- tive results the arXiv papers. Questions and answers were cre- ated by the annotator
MC
Sequence curacy Insensitive
Ac- Case
Video To Cam- era Trajectory Re- trieval
Data were collected from the project page of VD3D (Bahmani et al., 2024). Questions and an- swers designed and created by the annotator
MC
Exact Match
String
Sceneqa Scene Transition Video
Video data collected from MVBench (Li et al., 2024e).
Open
GPT-4o Judge
Questions and answers adapted by the annotator into open-ended format
Video Segments
Raw data come from UCF101 (Soomro et al., 2012).
Structured
Sequence Equal-
Reordering
ity
The annotator designed the task and re-organized the data to produce question-answer pairs
Action Sequence Understanding
Data were collected from MileBench (Song et al., 2024). Questions and answers were designed and created by the
Exact
Exact Match
String
annotator
Video Recognition
Action
Raw data come from UCFI0L (Soomro et al., 2012). The annotator designed the task and re-organized the data to produce the question-answer pairs
Structured
Exact Match
String
Google
The data were taken from Google Maps. Questions and answers were created by the annotator
Structured
Sequence Equal- ity
Streetview Line
Sorting
Next Action Pre-
Data were collected from MileBench (Song et al., 2024). Questions and answers were designed and created by the
MC
Exact Match
String
diction
annotator
Perception
Test Video Action
Video data collected from MVBench (Li et al., 2024e). Questions and answers are adapted by the annotator into direct numerical output format
Numerical
Exact Match
String
Count
91

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Google
The data were taken from Google Maps. Questions and answers created by the annotator
MC
Simple Match
String
Streetview
Line
Reasoning
Video Motion
Camera Descrip-
Videos were collected from Vi- dOR (Shang et al., 2019). Ques- tions and answers re-designed and created by the annotator
Exact
Exact Match
String
tion
Video Grounding Temporal
Videos were collected from Vi- dOR (Shang et al., 2019). Ques- tions and answers re-designed and created by the annotator
MC
Simple Match
String
Web Action Pre- diction
Data were collected from Visual- WebBench (Liu et al., 2024a)
MC
Exact Match
String
Cam
Traj
To
Data were collected from the project page of VD3D (Bahmani et al., 2024). Questions and an- swers designed and created by the annotator
Contextual
Simple Match
String
Video Selection
Sta Action Local- ization Video
Video data are collected from MVBench (Li et al., 2024e). Questions and answers are repur- posed for the contextual format- ted output format
Contextual
Simple Match
String
Contain
Contain
Images collected from the Web. Questions and constraints are designed by the annotator. This task has no reference an- swer
Open
Constrained Gen- eration
Images
Contain
Repeat
Images were collected from the Web. Questions and constraints are designed by the annotator. This task has reference swer
Open
Constrained Gen- eration
Length
Multi Contain Re- peat Position Only Length
Images were collected from the Web. Questions and constraints are designed by the annotator. This task has no reference an- swer
Open
Constrained Gen- eration
Contain Length
Images collected from the Web. Questions and constraints designed by the annotator. This task has no reference an- swer
Open
Constrained Gen- eration
Contain
Position
Images were collected from the Web. Questions and constraints are designed by the annotator. This task has no reference swer
Open
Constrained Gen- eration
Images
92

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Contain Position
Images collected from the
Open
Constrained Gen- eration
Length
Web. Questions and constraints designed by the annotator. This task has no reference an-
Xor Images
Images were collected from the Web. Questions and constraints are designed by the annotator. This task has no reference an- swer
Open
Constrained Gen- eration
Multi Contain Re- peat
Images were collected from the Web. Questions and constraints are designed by the annotator. This task has no reference an- swer
Open
Constrained Gen- eration
Contain Length
Contain
Images collected from the Web. Questions and constraints designed by the annotator. This task has no reference an-
Open
Constrained Gen- cration
Multi Contain Po- sition Only
Images were collected from the Web. Questions and constraints are designed by the annotator. This task has no reference an- swer
Open
Constrained Gen- eration
Relative Depth Of Different Points
Images were collected from BLINK (Fu al., 2024c).
MC
Exact Match
String
The annotator augmented each sample by adding one more reference point manually and adjusted the answers
Visual Prediction Rater Depth Esti- mation
Images were collected by taking screenshots from depth estima- tion papers on arXiv. Questions and answers were created by the annotator
MC
Sequence curacy Insensitive
Ac- Case
Visual Prediction Rater Novel View Synthesis
Images were collected by taking screenshots from novel view syn- thesis papers on arXiv. Questions and answers were created by the annotator
MC
Sequence Equal- ity
Pokemon
3d Images created by the Structured notator from the Pokemon Go game. Questions and answers were designed by the annotator
Exact Match
String
Recognition
Av View Identifi- cation
Images were collected from the nuScenes (Caesar et al., 2020) dataset. Questions and answers were designed and created by the annotator
Contextual
Sequence curacy Insensitive
Ac- Case
93

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Multiview Rea-
collected from BLINK (Fu et al., 2024c). Questions and answers were re-designed and augmented by the annotator
Images
Exact
Exact Match
String
soning Moving
Camera
3d Indoor Scene Text Bbox Predic- tion
The data is adapted from Multi3DRefer (Zhang et al., 2023). Questions and answers were designed by the annotator and dataset annotation.
Numerical
Nbbox Iou Single
Google
The data were taken from Google Maps. Questions and answers were created by the annotator
MC
Simple Match
String
Streetview Circle Reasoning
Google
The data were taken from Google StreetView. Questions and an- swers were created by the anno- tator
Exact
Exact Match
String
Streetview
Direction
Un-
derstanding
Video Motion Matching Real 3d
Videos were collected from the project page of Shen al. (2024). Questions and answers were created by the annotator
MC
Exact Match
String
Video Motion
Videos were collected from the project page of Shen et al. (2024). Questions and answers created by the annotator
MC
Exact Match
String
Matching 3d Real
Visual Prediction Rater 3d Assem- bled Quality Un- derstanding
Data were collected from the project page of Wang et al. (2024e). Questions and answers were designed and created by the annotator
MC
Sequence Equal- ity
Visual Prediction
Images were collected by tak- ing screenshots from surface mal estimation papers on arXiv. Questions and answers were cre- ated by the annotator
MC
Sequence curacy Insensitive
Ac- Case
Rater Surface
Normal Estima-
tion
Adapted Cvbench
Images were collected from CV- Bench (Tong et al., 2024). An- swers were adapted by the anno- tator into exact text
Exact
Exact Match
String
Depth
Visual Prediction Rater Plane Seg- mentation
Images were collected by taking screenshots from plane segmen- tation papers on arXiv
MC
Sequence curacy
Ac-
Case
Insensitive
3d Indoor Scene Text Bbox Selec-
Images were collected by tak- ing screenshots from 3D scene understanding papers on arXiv. Questions and answers were de- signed and generated by the an- notator
MC
Exact Match
String
tion
94

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Google
The data were taken from Google
Structured
Sequence Equal-
Streetview
Circle
Maps. Questions and answers were created by the annotator
ity
Sorting
Metrics
Paper
Review
Data collected from OpenRe- view's public paper reviews
Open
GPT-4o as Judge
Writing
Paper Rating
Review
Data collected from OpenRe- view's public paper reviews
Numerical
Number Rel Diff
Ratio
Paper Review Ac-
Data collected from OpenRe- view's public paper reviews
Exact
Exact Match
String
ceptance
Autorater Artifact
Images were collected from Ima- genHub (Ku et al., 2023). Ques- tions and answers adapted by the annotator
MC
Exact Match
String
Autorater Control
Images were collected from Ima- genHub (Ku et al., 2023). Ques- tions and answers adapted by the annotator
Exact
Exact Match
String
Autorater Artifact Reason
Images were collected from Im- agenHub (Ku et al., 2023). The annotator created open- ended reference answer manu- ally
Open
Constrained Gen-
eration
Autorater thetics
Aes-
Images were collected from Ima- genHub (Ku et al., 2023). Ques- tions and answers adapted by the annotator
Exact
Exact Match
String
Autorater mask
Un-
Images were collected from Ima- genHub (Ku et al., 2023). Ques- tions and answers adapted by the annotator
Exact
Exact Match
String
Autorater Subject
Images were collected from Ima- genHub (Ku et al., 2023). Ques- tions and answers adapted by the annotator
Exact
Fxact Match
String
Autorater Model Texturing
3d Resources are collected from the study of Perla al. (2024). Questions and answers were de- signed and created by the anno- tator
Contextual
Sequence ity
Equal-
Autorater Seman- tics
Images collected from Ima- genHub (Ku et al., 2023). Ques- tions and answers adapted by the annotator
Exact
Exact Match
String
Autorater Motion Guided Editing
Images were collected by taking screenshots from image genera- tion papers on arXiv
MC
Sequence Equal- ity
95

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Autorater Mask
Images were collected from Ima- genHub (Ku et al., 2023). Ques- tions and answers adapted by the annotator
Exact
Exact Match
String
Video Eval Visual Pref
Video frames were collected from ImagenHub (He al., 2024). Questions and answers adapted by the annotator
MC
Exact Match
String
Generated Video Artifacts
Videos were collected by run- ning various text-to-video diffu- sion models online. Open-ended reference answers by the annotator manually
Open
GPT-4o as Judge
written
Video Eval Fac- tual Pref
Video frames were collected from ImagenHub (He et al., 2024). Questions and answers adapted by the annotator
MC
Exact Match
String
Video Eval Dy- namic Pref
Video frames were collected from ImagenHub (He al., 2024). Questions and answers adapted by the annotator
MC
Exact Match
String
Vizwiz Quality Accessment For Blind
Images were collected from Chiu et al. (2020). Questions and answers were adapted and re- designed by the annotator
Contextual
Set Equality
Reward Models T2i Reward
Images were collected from RLAIF-V dataset (Yu et al., 2024a). Questions and answers were adapted by the annotator
Exact
Exact Match
String
Reward Models
collected from RLAIF-V dataset (Yu et al., 2024a). Questions and answers were adapted by the annotator
Images
Exact
Exact Match
String
12t Reward
Science
Biology Exams V
Data collected from EXAMS- (Das al., 2024) and MMMU- Pro (Yue et al., 2024b), and the questions and answers are adapted match strings
Contextual
Simple Match
String
Pmc Vqa Medical Image Qa
Data collected from NLVR2
Contextual
Simple Match
String
dataset (Suhr et al., 2018), and the questions and answers are adapted to match strings
Medical Content Based Retrieval
Data collected from ROCO dataset (Pelka et al., 2018), and the questions and answers adapted to match strings
MC
Exact Match
String
Radiology
96

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Medical
Ab-
Data collected from GMAI- MMBench (Chen et al., 2024b), and the questions and answers are adapted to match sequence accuracy
Contextual
Sequence curacy
Ac- Case
domen
MRI
Organ
Recogni-
Insensitive
tion
Medical Multi Or- gan Segmentation Rater
Data collected from pdf screen- shot, and the questions and answers are adapted to match strings
MC
Exact Match
String
Medical
Cell
Data collected from GMAI- MMBench (Chen et al., 2024b), and the questions and answers are adapted to match strings
Exact
Exact Match
String
Recognition
Medical
Im- age Artifacts
Data collected from GMAI- MMBench (Chen et al., 2024b), and the questions and answers are adapted to match strings
Exact
Exact Match
String
Indentification
Medical Blood Vessels Recogni- tion
Data collected from GMAI- MMBench (Chen et al., 2024b), and the questions and answers are adapted match strings
Structured
Exact Match
String
Healthcare
Info
Data collected from GMAI- MMBench (Chen et al., 2024b), and the questions and answers adapted match strings
MC
Exact Match
String
Judgement
Electrocardiogram
from MMMU (Yue et al., 2024a), and the answers are open-ended
Data collected
Open
GPT-4o Judge
Medical
Polyp
Data collected from pdf screen- shot, and the questions and swers are adapted to match se- quence equality
Structured
Sequence Equal- ity
Segmentation
Single Object
Rater
Ab- domen Endscopy Organ Recogni- tion
Medical
Data collected from GMAI- MMBench (Chen et al., 2024b), and the questions and answers are adapted to match sequence accuracy
Contextual
Sequence curacy Insensitive
Ac- Case
Key- words Based
Medical
Data collected from ROCO dataset (Pelka al., 2018), and the questions and answers are adapted match strings
Exact
Exact Match
String
Retrieval
Non
Radiology
Medical Parasite
Data collected from pdf screen- shot, and the questions and an- swers are adapted to match set equality
Structured
Set Equality
Detection
Medical Retrieval Given Surgeon Activity
Data collected from GMAI- MMBench (Chen et al., 2024b), and the questions and answers are adapted to match strings
MC
Exact Match
String
97

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Medical Counting Lymphocytes
Data collected from GMAI- MMBench (Chen et al., 2024b), and the questions and answers are adapted to match strings
Numerical
Exact Match
String
Chemistry Exams
Data collected from EXAMS- (Das al., 2024) and MMMU- Pro (Yue et al., 2024b), and the questions and answers are adapted to match strings
MC
Simple Match
String
V
Science Molecule Chemistry
Data collected IsoBench (Fu et al., 2024b), and the questions and answers are adapted by human annotator
from
Contextual
Simple Match
String
Mmmu Pro Exam Screenshot
Data collected from MMMU- Pro (Yue et al., 2024b), and the questions and answers are adapted to match strings
MC
Exact Match
String
Scibench W Solu- tion Open Ended
from Open Scibench (Wang et al., 2023b),
Data collected
GPT-4o as Judge,
General Single
and the answers are open-ended
Numerical Match
arXiv Vqa
Data collected from screen-
MC
Exact Match
String
shots by human annotator, and the questions and answers are adapted to match strings
Tqa Textbook Qa
Data collected from Dvqa (Kafle al., 2018), and the questions and answers are refractered from the original TQA dataset
Contextual
Multi Ref Phrase
Question Solution
Data collected from webpage screenshots by human annotator
Contextual
General
Single
Solving
Numerical Match
Quizlet Question
Data collected from wehpage screenshots by human annotator
Contextual
General
Single
Solving
Numerical Match
Scibench Funda-
Data collected from Numerical
General
Single
mental Wo Solu-
Scibench (Wang et al., 2023b)
Numerical Match
tion
Mmmu Physies Chemistry Mcq
Data collected MMMU (Yue et al., 2024a),
from
Exact
Exact Match
String
and the questions and answers are adapted to match strings
Data collected from webpage Circuit Diagram Understanding
Numerical
Fxact Match
String
screenshots by human annotator
Science
Basic
from IsoBench (Fu al., 2024b), and the questions and answers adapted by human annotator
Data collected
Contextual
Simple Match
String
Physics
98

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Physics Exams V
Data collected from EXAMS- (Das et al., 2024) and MMMU- Pro (Yue al., 2024b), and the questions and answers are adapted match strings
Contextual
Simple Match
String
Knowledge
Background Change
Images and labels come from the MFCBench (Wang et al., 2024c) dataset. Questions and annota- tions were adapted by human annotator.
MC
Exact Match
String
Out Of Context
Images and labels come from the MFCBench (Wang et al., 2024c) dataset. Questions and annota- tions were adapted by a human annotator.
MC
Exact Match
String
Text Entity Re- place
Images and labels come from the MFCBench (Wang et al., 2024c) dataset. Questions and annota- tions were adapted by human annotator.
MC
Exact Match
String
Text Style
Images and labels come from the MFCBench (Wang et al., 2024c) dataset. Questions and annota- tions adapted by human annotator.
MC
Exact Match
String
Face Edit
Attribute
Images and labels come from the MFCBench (Wang et al., 2024c) dataset. Questions and annota- tions were adapted by human annotator.
MC
Exact Match
String
Face Swap
Images and labels come from the MFCBench (Wang et al., 2024c) dataset. Questions and annota- tions were adapted by a human annotator.
MC
Exact Match
String
Interpret Perspective Illu sion
Force
Images come from various web- sites. Questions and annotations were created by human annota- tor.
Exact
Exact Match
String
Clip Stable Diffu- sion Generate
Images and labels come from the MFCBench (Wang et al., 2024c) dataset. Questions and annota- tions were adapted by human annotator.
MC
Exact Match
String
Unusual Images
Images come from various web- sites. Questions and annotations were created by human annota-
Open
GPT-4o as Judge
99

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Forensic Detec- tion Of Different Images
Images and labels come from the BLINK benchmark (Fu et al., 2024c). Questions and annota- tions were adapted by a human annotator.
MC
Exact Match
String
Veracity
Images and labels come from the MFCBench (Wang et al., 2024c) dataset. Questions and annota- tions were adapted by human annotator.
MC
Exact Match
String
Distinguish Generated Image
AI Images come from various websites and image generators. Questions and annotations were created by human annotator.
Exact
Exact Match
String
Cultural Vqa
Images and labels come from bench- the CulturalVQA mark (Romero et al., 2024). Questions and annotations adapted by a human annotator.
Contextual
Multi Ref Phrase
Human Relation- ship Reasoning
Images come from various web- sites. Questions and annotations were created by human annota- tor.
Contextual
Simple Match
String
Sign Language
Videos come from Dr. Bill Vi- cars' "Signs" YouTube channel. Questions and annotations were created by human annotator.
Contextual
Multi Ref Phrase
Ishihara Test
Images come from various web- sites. Questions and annotations were created by human annota- tor.
Structured
Set Precision
Llavaguard
Images and labels come from the LlavaGuard benchmark (Helff
Structured
Exact Match
String
al., 2024). Questions were cre- ated by human annotator.
Red Racial
Teaming
Images and labels come from the Red Teaming benchmark (Li
Open
GPT-4o as Judge
et al., 2024f). Questions were created by a human annotator or generated by GPT-4.
Red Captcha
Teaming
Images and labels the Red Teaming benchmark (Li et al., 2024f). Questions were created by human annotator or generated by GPT-4.
from
Open
GPT-4o Judge
Red Teaming Pol- itics
Images and labels come from the Red Teaming benchmark (Li et al., 2024f). Questions were created by human annotator generated by GPT-4.
Open
GPT-4o as Judge
100

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Mmsoc memes
Hateful-
Images and labels come from the MMSoc benchmark (Jin et al., 2024). Questions and answers were adapted by a human anno- tator.
MC
Exact Match
String
Red Teaming Vi- sual Order B
Images and labels come from the Red Teaming benchmark (Li et al., 2024f). Questions were created by human annotator or generated by GPT-4.
Open
GPT-4o Judge
Red Teaming Celebrity
Images and labels come from the Red Teaming benchmark (Li et al., 2024f). Questions were
Open
GPT-4o as Judge
created by human annotator. or generated by GPT-4
Mmsoc tion
Memo-
Images and labels come from the MMSoc benchmark (Jin et al., 2024). Questions and answers were adapted by a human anno- tator.
Structured
Exact Match
String
Misocc Misinfor- mation Politifact
Images and labels come from the MMSoc benchmark (Jin et al., 2024). Questions and answers were adapted by human anno- tator.
MC
Exact Match
String
Red Teaming Jail- break
Images and labels come from the Red Teaming benchmark (Li et al., 2024f). Questions were created by a human annotator or generated by GPT-4.
Open
GPT-4o as Judge
Red Teaming Vi- sual Order
Images and labels come from the Red Teaming benchmark (Li et al., 2024f). Questions were created by a human annotator or generated GPT-4.
Open
GPT-4o as Judge
Misoce Misinfor- mation Gossipcop
Images and labels come from the MMSoc benchmark (Jin et al., 2024). Questions and answers were adapted by human anno- tator.
MC
Exact Match
String
Red Teaming sualmisleading
Images and labels come from the Red Teaming benchmark (Li et al., 2024f). Questions were created by human annotator.
Open
GPT-4o as Judge
Video Follow Up
Content
Videos taken from YouTube. Questions and answers created by human annnotator.
Open
GPT-4o Judge
Meme Explain
Images come from various web- sites. Questions were created by a human annotator.
Open
GPT-4o as Judge
101

---

Images and labels come from Humor Understanding bench- mark derived from New Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Funny Image Title
Images come from various web- sites. Questions were created by human annotator.
Open
GPT-4o Judge
Emotion Recogni- tion
Videos and labels come from the CAER dataset (Lee et al., 2019). Questions and answers were adapted by human anno- tator.
Exact
Exact Match
String
Image Humor Un- derstanding
Images come from various web- sites. Questions created by a human annotator.
Open
GPT-4o as Judge
Humor Explana- tion
Images and labels come from Humor Understanding bench- mark derived from the New Yorker Caption Contest (Hessel et al., 2022). Questions were cre- ated by human annotator.
Open
GPT-4o as Judge
Mvsa Sentiment Classification
Images and labels come from the MVSA dataset (Niu al., 2016). Questions and answers were adapted by human anno- tator
MC
Exact Match
String
Video Recognition
Intent
Video and labels come from the MIntRec dataset (Zhang et al., 2022). Questions and answers were adapted by a human anno- tator.
Contextual
Simple Match
String
Humor stand Match
Under- Caption
Exact
Exact Match
String
Yorker Caption Contest (Hessel et al., 2022). Questions and an- swers were adapted by human annotator.
Figurative Speech Explanation
Images come from various web- sites. Questions were created by human annotator.
Open
GPT-4o as Judge
Muma Theory Of Mind Social Goal
Images and labels come from the MuMA-ToM dataset (Shi et al., 2024). Questions and answers were adapted by a human anno- tator.
Contextual
Simple Match
String
Muma Theory Of Mind Belief Of Goal
Images and labels come from the MuMA-ToM dataset (Shi et al., 2024). Questions and answers were adapted by human anno- tator.
Contextual
Simple Match
String
Hashtag Recom- mendation
Images and hashtags come from various social media websites. Questions were created by a hu- man annotator.
Structured
Set Precision
102

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Dish Match
Ingredient
Images and labels come from the HelloFresh website. Questions
MC
Exact Match
String
created by human tator.
Music Sheet Sen- timent
Images are music sheets posted to Noteflight. Questions and an- swers were created by human annotator.
Exact
Exact Match
String
Music Sheet Au- thor
Images are music sheets posted to Noteflight. Questions and an- swers were created by human annotator.
Exact
Exact Match
String
Music Sheet Note Count
Images are music sheets posted to Noteflight. Questions and an- swers were created by human annotator.
Numerical
Exact Match
String
Music Sheet For- mat Qa
Images music sheets posted to Noteflight. Questions and an- swers were created by human annotator.
Numerical
Exact Match
String
Orchestra Recognition
Score
Images come from various web- sites. Questions were created by human annotator.
Structured
Exact Match, String Match
String Simple
Music Name
Sheet
Images music sheets posted to Noteflight. Questions and an- swers were created by human annotator.
Exact
Exact Match
String
Insect Order Clas- sification
Images and labels come from the BIOSCAN-IM dataset (Gharaee et al., 2024). Questions and an- swers were adapted by human annotator.
Contextual
Simple Match
String
Signage tion
Naviga-
Images come from various web- sites. Questions and answers were created by human anno- tator.
Exact
Exact Match
String
Title Iden- tification From Lyrics
Song
taken by the human annotator on the Spotify Web Player. Questions and an- swers were created by the anno- tator.
Screenshots
Structured
Exact Match
String
Sign Recognition
Knowledge
Images come from various web- sites. Questions were created by human annotator.
MC
String Set Equal- ity Comma
Brand Logo Recognition And Elaboration
Images come from the Flick- rLogo (Romberg al., 2011) dataset and various websites.
Structured
Multi Ref Phrase
Questions created by hu- man annotator.
103

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Logo2k Type
Same Re-
Images come from the Logo2K+ dataset (Wang et al., 2020) dataset and various websites.
Structured
Exact Str Match Case Insensitive,
Logo
trieval
Set Equality
Questions were created by a hu- annotator.
Chinese Recognition
Idiom
Images come from various web- sites. Questions and answers were created by a human anno- tator.
Exact
Exact
String
Match
Multi Lingual Expla-
Some images and labels are from the COIG-CQIA dataset (Bai et al., 2024) and some images are from Baidu Tieba and annotated by human annotator.
Open
GPT-4o Judge
Ruozhiba nation French
Multi Lingual Ruozhiba Expla-
Some images and labels are from the COIG-CQIA dataset (Bai et al., 2024) and some images are from Baidu Tieba and annotated by human annotator.
Open
GPT-4o as Judge
nation Arabic
Multi Lingual Ruozhiba Expla- nation Spanish
Some images and labels are from the COIG-CQIA dataset (Bai et al., 2024) and some images are from Baidu Tieba and annotated by human annotator.
Open
GPT-4o as Judge
Multi Lingual Ruozhiba Expla- nation English
Some images and labels are from the COIG-CQIA dataset (Bai et al., 2024) and some images are from Baidu Tieba and annotated by human annotator.
Open
GPT-4o Judge
Multi Lingual Ruozhiba Expla- nation Japanese
Some images and labels are from the COIG-CQIA dataset (Bai et al., 2024) and some images are from Baidu Tieba and annotated by human annotator.
Open
GPT-4o as Judge
Multi Lingual Ruozhiba Expla-
Some images and labels are from the COIG-CQIA dataset (Bai et al., 2024) and some images are from Baidu Tieba and annotated by human annotator.
Open
GPT-4o as Judge
nation Russian
Font Recognition
Images and labels taken from Identifont. Questions are created by human annotator.
Exact
Exact Match
String
Traffic Accident Analysis
Images and labels taken from Jia Kao Bao Dian. Questions are created by human annotator.
Open
GPT-4o Judge
Multiple
States
Images come from various web- sites and were edited by the an- notator. Questions and answers created by human annota- tor.
Contextual
Sequence curacy Insensitive
Ac- Case
Identify Asia
104

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Multiple States Identify Americas
Images come from various web- sites and were edited by the an- notator. Questions and answers were created by human annota-
Contextual
Sequence curacy
Ac-
Case
Insensitive
States Identify Europe
Multiple
Images come from various web- sites and were edited by the an- notator. Questions and answers were created by human annota- tor.
Contextual
Sequence curacy Insensitive
Ac- Case
Multiple Identify Africa
States
Images come from various web- sites and were edited by the an- notator. Questions and answers were created by human annota- tor.
Contextual
Sequence curacy Insensitive
Ac- Case
Worldle
Images and labels are taken from Worldle Daily, a free Geoguessr alternative. Questions and swers are created by a human an- notator.
Structured
Exact Match
String
Location Vqa
Images and labels come from various websites. Questions were created by a human anno- tator.
Exact
Exact Match
String
Vibe Eval Open
Images and labels the Vibe-Eval dataset Padlewski et al. (2024). Questions were cre- ated by human annotator.
from
Contextual
Multi Ref Phrase
Vibe Eval Phrase
Images and labels come from the Vibe-Eval dataset Padlewski et al. (2024). Questions were cre- ated by human annotator.
Open
GPT-4o as Judge
Ancient Map Un- derstanding
Images and labels come from various websites. Questions were created by a human anno- tator.
Exact
Exact Match
String
Rocks Compare
Samples
Images and labels come from ChinaNeolithic.coms online rock store. Questions were created by human annotator.
Contextual
Simple Match
String
Painting Qa
Images and labels come from the MMMU benchmark Yue et al. (2024a). Questions and answers were adapted by human anno- tator.
Exact
Exact Match
String
Art Explanation
Images come from various web- sites. Questions created by a human annotator.
Open
GPT-4o as Judge
105

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Memorization Chinese Celebrity
Images and labels come from various websites. Questions
Structured
Multi Ref Phrase
created by human tator.
Memorization Pa- pers
Images and labels come from various websites. Questions were created by human anno- tator.
Structured
Simple Match
String
Memorization Fa- mous Treaty
Images and labels come from various websites. Questions were created by a human anno- tator.
Structured
Exact Match, Multi Ref Phrase
String
Memorization In- dian Celebrity
Images and labels come from various websites. Questions were created by a human anno- tator.
Structured
Exact Match, Multi Ref Phrase
String
Soccer Offside
Images from various web- sites. Questions were created by human annotator.
MC
Exact Match
String
Deciphering Ora- cle Bone
Images and labels come from the "Deciphering Oracle Bone Lan- guage with Diffusion Models" paper (Guan et al., 2024). Ques- tions were created by a human annotator.
Exact
Exact Match
String
Kvqa Knowledge Aware Qa
Images and labels come from the MapQA dataset (Chang et al., 2022). Questions and answers were adapted by human anno- tator.
Contextual
Simple Match
String
Character Recog- nition In Tv
Screenshots were taken by the Contextual human annotator the Amazon Prime Video webpage. Ques- tions and answers were created by the annotator.
Set Equality
Shows
Actor Recogni-
Screenshots were taken by the human annotator on the Amazon Prime Video webpage. Ques- tions and answers were created by the annotator.
Exact
Exact Match
String
tion In Movie
Landmark Recog- nition And Qa
Images and labels come from the Landmark v2 dataset (Weyand al., 2020). Questions and an- swers were adapted by human annotator.
Structured
Exact Match, Multi Ref Phrase, Near Str Match
String
Famous Building Recognition
Images and labels come from various websites. Questions were created by a human anno- tator.
Structured
Exact Str Match
Case Insensitive,
Exact String Match
106

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Landmark Check Two Images
Images and labels come from the Landmark v2 dataset (Weyand al., 2020). Questions and swers were adapted by human annotator.
Structured
Exact Str Match
Case Insensitive
Defeasible soning
Rea-
Images and labels come from various websites. Questions were created by a human anno- tator.
Open
GPT-4o Judge
Poetry Limerick
Images from various web- sites. Questions and evaluation constraints created by hu-
Open
Constrained Gen- eration
man annotator.
Poetry Shake- spearean Sonnet
Images come from various web- sites. Questions and evaluation constraints were created by hu- man annotator.
Open
Constrained Gen- cration
Poetry Custom Rhyming Scheme
Images come from various web- sites. Questions and evaluation constraints were created by hu- man annotator.
Open
Constrained Gen- eration
Acrostic Alliteration
Poetry
Images come from various web- sites. Questions and evaluation constraints were created by a hu- annotator.
Open
Constrained Gen- cration
Poetry Haiku
Images come from various web- sites. Questions and evaluation constraints were created by hu- man annotator.
Open
Constrained Gen- eration
Poetry
Petrar- Sonnet
Images come from various web- sites. Questions and evaluation constraints created by hu- man annotator.
Open
Constrained Gen- cration
chian Optional Meter
Poetry Acrostic
Images come from various web- sites. Questions and evaluation constraints were created by hu- man annotator.
Open
Constrained Gen- eration
Ascii Art 30
Images come from various web- sites. Reference ASCII images were created using the ASCII Art Archive's "Image
Contextual
ASCII Art GPT- 4o Judge
ASCII Art" tool.
Mathematics
Graph Path Kawai
Shortest Kamada
Data collected from Visual Graph Arena Dataset by human annotator, and the questions and answers are adapted to match strings
Numerical
Exact Match
String
107

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Graph Shortest
Data collected from Visual Graph Arena Dataset by human annotator, and the questions and answers are adapted to match
Numerical
Exact Match
String
Path Planar
strings
Graph Connectiv-
Data collected IsoBench (Fu et al., 2024b), and the questions and answers adapted by human annotator
from
Structured
Exact Match
String
ity
Graph Theory
Data collected from MathVi- sion (Wang et al., 2024b), and the questions and answers are adapted by human annotator
Exact
Exact Match
String
Graph phism
Isomor-
from IsoBench (Fu et al., 2024b), and the questions and answers are adapted by human annotator
Data collected
MC
Exact Match
String
Graph
Hamilto-
Data collected from Visual Graph Arena Dataset by human annotator, and the questions and answers are adapted match precision
Structured
Exact Match, Set Preci- sion
String
nian Cycle
Graph Hamilto- nian Path
Data collected from Visual Graph Arena Dataset by human annotator, and the questions and answers are adapted to match set precision
Structured
Exact Match, sion
String Sct Preci-
Chordless Cycle
Graph
Data collected from Visual Graph Arena Dataset by human annotator, and the questions and answers are adapted to match strings
Numerical
Exact Match
String
Topological Sort
Data collected from screenshots by human annotator
Structured
Set Equality
Graph Maxflow
Data collected IsoBench (Fu et al., 2024b), and the questions and answers adapted by human annotator
from
Numerical
Exact Match
String
Seibench Caleu- lus Wo Solution
Data
collected
from Numerical
General
Single
Scibench (Wang et al., 2023b)
Numerical Match
Clevr Arithmetic
Data collected from Clevr (John- son et al., 2017)
Numerical
Exact
String
Match
Iconqa Count And Reasoning
Data collected from IConQA (Lu et al., 2021), with annotation refractered from the original IConQA dataset
Numerical
Multi Ref Phrase
Number Compari- son
Data collected from screenshots by human annotator
Numerical
Exact Match
String
108

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Math Exams V
Data collected from MMMU- Pro (Yue et al., 2024b), and the questions and answers adapted to match numerical data
MC
General
Single
Numerical Match
Theoremqa
Data collected from screenshots by human annotator
Contextual
Boxed Single Nu-
merical Match
Math
Data collected from screenshots by human annotator
Numerical
Boxed Single Nu-
merical Match
Math Parity
Data collected IsoBench (Fu et al., 2024b), and the questions and answers are adapted by human annotator
from
MC
Exact Match
String
Math Breakpoint IsoBench (Fu et al., 2024b), and the questions and answers are adapted by human annotator
Data collected
from Numerical
Exact Match
String
Math Convexity Value Estimation
Data collected IsoBench (Fu et al., 2024b), and the questions and answers are adapted by human annotator
from
Structured
Exact Match, Rel Diff Ratio
String Number
Geometry Rea- Count
Data collected from Vision lan- guage models blind (Rah- manzadehgervi et al., 2024), and the questions and answers adapted by human annotator
Structured
Exact Match
String
soning
Line Intersections
Geometry Length
Data collected from MathVi- sion (Wang al., 2024b), and the questions and answers are adapted by human annotator
Contextual
General Numerical
Single Match
Geometry soning Squares
Rea- Nested
Data collected from Vision lan- guage models blind (Rah- manzadehgervi et al., 2024), and the questions and answers are adapted by human annotator
Structured
Exact Match
String
Geometry formation
Trans-
Data collected from MathVi- sion (Wang et al., 2024b), and the questions and answers are adapted by human annotator
Contextual
General Numerical Match
Single
Geometry soning lapped Circle
Rea- Over-
Data collected from Vision lan- guage models are blind (Rah- manzadehgervi et al., 2024), and the questions and answers are adapted by human annotator
Structured
Exact Match
String
Geometry Area
Data collected from MathVi- sion (Wang et al., 2024b), and the questions and answers are adapted by human annotator
Numerical
Exact Match
String
109

---

Published as a conference paper at ICLR 2025
Table 18 continued from previous page
Task Name
Source Description
Output Format
Metrics
Geometry Rea-
Data collected from Vision lan- guage models are blind (Rah- manzadehgervi al., 2024), and the questions and answers are adapted by human annotator
Structured
Exact
String
soning Grid
Match
Polygon
Interior
Data collected from screenshots by human annotator
Numerical
Angle Seq Float Rmse
Angles
Geometry Solid
Data collected from MathVi- sion (Wang et al., 2024b), and the questions and answers are adapted by human annotator
Contextual
General Single
Numerical Match
Geometry lytic
Ana- Data collected from MathVi- sion (Wang et al., 2024b), and the questions and answers are adapted by human annotator
Contextual
General
Single
Numerical
Match
Geometry scriptive
De-
Data collected from MathVi- sion (Wang et al., 2024b), and the questions and answers are adapted by human annotator
Contextual
Single Numerical Match
General
Counterfactual Arithmetic
Data collected from screenshots by human annotator
Numerical
Exact Match
String
Algebra
Data collected from MathVi- sion (Wang et al., 2024b), and the questions and answers are adapted by human annotator
Contextual
General Single
Numerical
Match
H AUTHOR CONTRIBUTION STATEMENT
All authors contributed at least 30 tasks to MEGA- A-BENCH and participated in task brainstorm- ing, annotation, and validation. They also engaged in discussions on data annotation and provided feedback. The following authors made additional contributions to various aspects of the project:
Jiacheng Chen co-designed the project with Wenhu Chen, led the benchmark construction process, and coordinated collaboration among all contributors. Jiacheng Chen created and maintained the annotation GUI tool, GitHub repository, draft task taxonomy tree, and results visualization page to facilitate data annotation and improve data quality. Jiacheng Chen led the development and main- tenance of the benchmark evaluation pipeline, including the VLM query pipeline, customized eval- uation metrics, HuggingFace Space for the project, etc., and conducted the main experiments and analyses. Jiacheng Chen also led the writing of the paper, coordinating contributors and incor- porating their input into the manuscript.
Tianhao Liang co-led the benchmark data organization, implemented most model query and evalu- ation pipelines under a consistent and unified framework, and conducted the experiments and anal- yses. Tianhao Liang also maintained the evaluation pipeline and implemented the code exccution metric. Tianhao Liang made significant efforts in data quality control, error analysis, and creating figures and tables for the paper.
Sherman Siu made contributions to the benchmark construction process, including task reviewing, annotator coordination, data quality control, metric implementation, and code maintenance. Sher- man Siu contributed and designed a bunch of complex and novel planning tasks. Sherman Siu also analyzed the number of examples per task to investigate the variance the benchmark score and contributed to writing the main paper.
110

---

Published as a conference paper at ICLR 2025
Zhengqing Wang contributed around 40 tasks, including several complex traditional computer vi- sion tasks. Zhengqing Wang organized the benchmark statistics for the Appendix, participated in enror case analysis, and developed the project page.
Kai Wang contributed around 40 tasks with diverse data sources and output formats. Kai Wang helped organize the benchmark construction process and actively checked the annotation quality of other annotators.
Yubo Wang assisted with the experiments of open-source models.
Yuansheng Ni helped organize the Appendix and polished šB to $F.
Wang Zhu implemented the evaluation metric for symbolic planning tasks and helped with the paper writing.
Hexiang Hu participated in discussions of the project's initial idea and continuously provided thoughts and resources for diverse tasks to facilitate the benchmark construction process. Hexi- ang Hu wrote a significant portion of the main paper, advised experimental design, and helped present tables and figures.
Xiang Yue discussed the high-level directions and goals of the project with Jiacheng Chen and Wenhu Chen. Xiang Yue provided insightful thoughts for multi-dimensional results analysis and assisted with the experiments of open-source models.
Wenhu Chen proposed the initial concepts of the project, continuously advised project progress while refining its strategic scope and direction, and called up and organized all contributors. Wenhu Chen contributed approximately 50 diverse tasks, generated ideas for new tasks, and distributed them to other annotators. Wenhu Chen wrote the initial draft of the main paper to establish the high-level structure and guided the organization and analysis of the results.
111


Reviews:
[{"review_id": "SnltjtjctU", "reviewer": "Reviewer_cTcJ", "summary": "This work presents a new benchmark for multimodal LLMs. The authors attempt to create a novel, diverse, comprehensive benchmark for vision-language reasoning using a several-stage process for designing the benchmark, refining the questions, and developing appropriate metrics. The authors conduct a comprehensive large-scale evaluation of current SOTA multimodal models using the benchmark.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "# Overall assessment\n\nThis work presents an interesting contribution in a much-needed space (benchmarks for multimodal large models). To address the current scattershot approach to multimodal model benchmarking, the authors attempt to create a single, highly diverse, comprehensive benchmark for a variety of image-language tasks (including video). To construct the benchmark the authors develop and refine a task taxonomy, but some details around the taxonomy and its construction are unclear. I have concerns about how the benchmark would be used in practice related to the 40 different evaluation metrics, and the distribution over various attributes (number of images, task type, etc.) but am willing to increase my score based on discussion with authors and other reviewers.\n\n# Major comments\n\n* The quality of the benchmark ultimately relies on the authors' proposed taxonomy, as this forms the basis for all data collection. However, I found the description of the annotation process somewhat disappointing; it effectively amounts to \"the Feynman Method\" (write down the problem, think hard about it, write down the solution). Critically, the authors provide no discussion or framing around the \"conceptualization stage\" for how they identified the top levels of the taxonomy (perception, planning, reasoning), nor how the annotators were selected or why they are representative of the whole of relevant multimodal knowledge (the sample of annotators could also bias the coverage in various ways). Please provide a clear discussion of (a) what the levels of the taxonomy are (please give the full list) and (b) how these levels were identified and why they comprise a holistic benchmark and (c) the disciplines of the annotators (since the authors state they are graduate or above from diverse disciplines).\n\n* The diversity of output formats is an interesting contribution. However, the diveristy of evaluation metrics (over 40 metrics?!) also makes this benchmark somewhat unwieldy, and raises concerns about usability. These issues arise even in the authors' main findings, stated at the end of Section 1. For example, it is very difficult to understand what it means that GPT-4o is 3.5% better than Claude 3.5? What makes this a \"significant margin\"? If Qwen2-VL is 10% better than other open source models, what does this mean? T\n\n* It is not clear whether all tasks in the benchmark have a single, objective answer. This makes it difficult to assess models' capabilities (for example, failure to write a latex equation may simply be due to a difference in formatting; writing a story containing two animals hinges on many different criteria which are difficult to assess).\n\n* The advantages of a single, diverse, high-coverage benchmark are outlined nicely in the introduction. However, the paper's contribution hinges on whether it does indeed  achieve strong coverage of a \"diverse\" suite of tasks. Ultimately, this is nearly impossible to assess, but I have some concerns about the \"concepttualization\" process above that make me unsure that this benchmark is as comprehensive as the authors claim. On the other hand, the existing benchmarks are also imperfect (and a direct comparison to existing benchmarks in terms of content and task design would make it easier to assess whther the benefits of the new benchmark outweigh the potential downsides and complexity).\n\n* It is unclear why certain task distributions are set as the authors designed them in the benchmark. For example, why are only 4% of tasks 6-8 images, while 8% are 9+ images? Why are 16% of tasks open-ended while 22% are structured? These design decisions can have significant effects when averaging over benchmarks, as will likely occur with this benchmark.\n\n* The empirical study is useful, appears comprehensive, and leads to some interesting conclusions.\n\n* It seems unlikely that the benchmark will last very long by relying on GPT-4o as judge. Is it possible to substitute the LLM judge in the benchmark if a future best frontier model emerges?\n\n\n# Minor comments\n\n* Another relevant multimodal baseline the authors may want to reference: Bitton, Yonatan, et al. \"Visit-bench: A dynamic benchmark for evaluating instruction-following vision-and-language models.\" Advances in Neural Information Processing Systems 36 (2023): 26898-26922.\n\n# Typos etc\n\n* \"these models have shown great potential to solve any desired task with a well-designed prompt\" - this is editorlaizing somewhat; please revise.\n\n* L111-113: \"comprehensive studies...have discovered\" passive voice, consider revising", "weaknesses": "see above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TUOGqb4Lpv", "reviewer": "Reviewer_d5yn", "summary": "The paper presents MEGA-BENCH, a comprehensive multimodal benchmark scaling up to over 500 real-world tasks, designed to assess the diverse capabilities of vision-language models. It offers a fine-grained analysis across various dimensions, including application, input type, output format, and skills, and provides customized metrics for different output formats. The benchmark reveals significant performance variations among state-of-the-art models, emphasizing the importance of task diversity over increasing examples per task for insightful model evaluation.", "questions": "1. Could you explain more about the background of your 16 annotators and how you make sure for all task instances, the instruction and solution align with each other?\n2. For the open-ended tasks, you mentioned using an LLM-assisted metric. How do you handle the potential for bias in the evaluation process, given that the scoring is dependent on a proprietary LLM? If we use different LLM as judges, will their ratings differ a lot from each other?\n3. What are the considerations and challenges you preview when scaling MEGA-BENCH even further? How do you plan to maintain the benchmark's relevance and diversity as new multimodal tasks emerge?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. MEGA-BENCH has a large scale and coverage, containing over 500 diverse real-world tasks, which allows for an in-depth assessment of multimodal models across various applications and skills.\n\n2. It offers a sophisticated, fine-grained analysis capability by categorizing tasks along multiple dimensions, providing a nuanced understanding of model performance in specific areas and revealing strengths and weaknesses that aggregate scores might obscure.\n\n3. The benchmark's design emphasizes cost-effectiveness and efficiency, demonstrating that increasing task diversity is more valuable for gaining performance insights than simply adding more examples per task.", "weaknesses": "1. While MEGA-BENCH offers a vast array of tasks, its large scale may lead to increased computational costs and complexity in evaluation, potentially limiting its accessibility for further research and extensive exploration.\n2. MEGA-BENCH's focus on breadth may result in some tasks being too specific or niche, which could limit the generalizability of the benchmark results to a broader range of multimodal problems and applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FKHgjS8Y0m", "reviewer": "Reviewer_8FfE", "summary": "The paper presents Mega Bench, a comprehensive benchmark for evaluating multimodal models on over 500 tasks. Mega Bench features a wide range of output formats and uses multiple metrics for evaluation. It includes a detailed capability report for popular language and vision models. The benchmark's assessment of leading models reveals significant performance differences and the importance of task diversity.", "questions": "Q1: There are many different input/output formats and metrics in Mega Bench. How does Mega Bench address the challenge of \"Unmanageable Setups\" mentioned in the introduction?\n\nQ2: Are there any copyright / privacy concerns for the tasks in the bench mark?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1: The proposed open-source benchmark includes a large number of diverse tasks for LLMs that can potentially address the limitations of existing benchmarks. It provides valuable resource for the community.\n\nS2: The paper also provides an extensive experiment and analysis of popular LLMs using Mega Bench. It yields many interesting findings.\n\nS3: This paper is well-written and easy to read.", "weaknesses": "### Major weaknesses\n\nW1: The rationale behind the task taxonomy tree is not well-explained. Section 3.1 can be strengthened by discussing the design considerations for the draft taxonomy tree. For example, why do we want perception, planning, reasoning? Are these the limitations of existing benchmarks? How do we know this taxonomy is comprehensive and reflects the real usage of LLMs?\n\nW2: The introduction highlights Mega Bench's contributions in multimodal tasks. However, there is limited information regarding non-text tasks in Section 3. I recommend adding a few non-text tasks in Figure 4 and discussing the image and video tasks included in Mega Bench in Section 3.\n\nW3: It is unconvincing that Mega Bench makes significant contributions over existing benchmarks. In the introduction, the paper lists four limitations of existing benchmarks: (1) limited output diversity, (2) lack of task coverage, (3) expensive inference cost, and (4) unmanageable setups. Section 3 and 4 explain how Mega Bench address limitations (1) and (2), but (3) and (4) remain unaddressed in the paper. I recommend discussing what makes Mega Bench less expensive and easier to run compared to other popular benchmarks.\n\n### Minor weaknesses\n\nM1: Replace $ (L83).\n\nM2: The claim that \"many examples or tasks are highly similar in the capabilities that they assess\" requires evidence to back it up (L83-83).\n\nM3: The tasks in Mega Bench have a lot in common with those in Big Bench. A detailed comparison to Big Bench would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rMcRzsX3YW", "reviewer": "Reviewer_9tqt", "summary": "The paper introduces MEGA-BENCH, a comprehensive multimodal evaluation suite that encompasses over 500 real-world tasks, addressing the diverse daily use cases of end users. Its goal is to optimize for high-quality data samples that cover a wide range of multimodal tasks while facilitating cost-effective and accurate model evaluation. The authors have compiled 507 realistic tasks with over 8,000 samples from 16 expert annotators, embracing various output formats and developing over 40 metrics to accommodate these formats. MEGA-BENCH provides a fine-grained capability report across multiple dimensions, enabling in-depth interaction with and visualization of model capabilities. The paper also evaluates various state-of-the-art vision-language models using MEGA-BENCH, revealing significant performance variations among models that were previously thought to be similar.", "questions": "Has the authors' team conducted any analysis on the environmental impact of the computational resources required for the benchmarking process? If so, could they share some insights?\n\nAre there plans to release the annotation tools, pre-processing pipelines, and evaluation metrics as open-source to facilitate community-wide reproducibility and further development?\n\nCould the authors discuss how the tasks in MEGA-BENCH map to real-world applications? Are there any tasks that are particularly relevant to current industry needs or future technological trends?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The creation of MEGA-BENCH is an original contribution to the field of multimodal AI evaluation. It scales up the number of tasks to an unprecedented level, offering a comprehensive assessment of model capabilities across a vast array of real-world applications. The approach of embracing diverse output formats and developing over 40 metrics to accommodate these is innovative, moving beyond the limitations of traditional multi-choice question-based benchmarks.\n\nThe quality of the work is evident in the meticulous construction of the benchmark. With 507 realistic tasks and over 8,000 samples collected from 16 expert annotators, the dataset is both extensive and rich in diversity. The rigorous annotation process, including the development of an annotation GUI and a taxonomy tree, ensures high-quality data that is well-suited for evaluating multimodal models.\n\nThe paper is well-structured and clearly articulated. The figures and tables are effectively used to convey complex information in a digestible manner. The taxonomy tree and the breakdown of tasks across different dimensions are particularly clear, aiding the reader in understanding the scope and organization of MEGA-BENCH.", "weaknesses": "The paper presents a snapshot of model performance but does not address how these benchmarks might be used to track performance over training time. A good benchmark should be verified by scaling laws.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)