MixEval-X: Any-to-any Evaluations from Real-world Data Mixture
Authors: Jinjie Ni
Yifan Song
Deepanway Ghosal
Bo Li
David Junhao Zhang
Xiang Yue
Fuzhao Xue
Yuntian Deng
Zian Zheng
Kaichen Zhang
Mahir Shah
Kabir Jain
Yang You
Michael Shieh
Venue: ICLR 2025 Spotlight (2025)
Keywords: Evaluation
Multi-modal Evaluation
Benchmark
Multi-modal Benchmark
Any-to-any
MixEval
Real-world
Data Mixture
Artificial General Intelligence
AGI
Primary area: datasets and benchmarks
TL;DR: We propose MixEval-X, the first any-to-any real-world benchmark optimizing benchmark mixtures for a wide range of input-output modalities.

Abstract: Perceiving and generating diverse modalities are crucial for AI models to effectively learn from and engage with real-world signals, necessitating reliable evaluations for their development. We identify two major issues in current evaluations: (1) inconsistent standards, shaped by different communities with varying protocols and maturity levels; and (2) significant query, grading, and generalization biases. To address these, we introduce MixEval-X, the first any-to-any, real-world benchmark designed to optimize and standardize evaluations across diverse input and output modalities. We propose multi-modal benchmark mixture and adaptation-rectification pipelines to reconstruct real-world task distributions, ensuring evaluations generalize effectively to real-world use cases. Extensive meta-evaluations show our approach effectively aligns benchmark samples with real-world task distributions. Meanwhile, MixEval-X's model rankings correlate strongly with that of crowd-sourced real-world evaluations (up to 0.98) while being much more efficient. We provide comprehensive leaderboards to rerank existing models and organizations and offer insights to enhance understanding of multi-modal evaluations and inform future research.

Published as a conference paper at ICLR 2025
MIXEVAL-X: ANY-TO-ANY EVALUATIONS FROM REAL- WORLD DATA MIXTURES
Jinjie Nist, Yifan Song", Deepanway Ghosal"; Bo Li", David Junhao Zhang", Xiang Yue", Fuzhao Xue", Zian Zheng", Kaichen Zhang", Mahir Shah", Kabir Jain", Yang You", Michael Qizhe Shich
"National University of Singapore, "Nanyang Technological University, SPeking University, CCarngge Mellon University, University of Waterloo, 'Independent Researcher
https://mixevalxx.ithbbi github..io
Image Text
Ratio 1.31%
Video Text
Ratio 0.61% 5%
Audio Text
MMU Tasks
cabject
Normtl// Hard
Normal
pointig enongyy
typical
wiidh
puilore?
Clyllenc
Optinns:
Optiane Lod Zepeiia
The prisspllo muventiiin
Kapid axcesi
Daligeemmn Maxisn
Oueers
ceabo Eped-
D. O.ACDCC
Text Image
Ratio on web: 0,7 0,73%
Text to Video
Ratio on web: 0.23%
Text to Audio
Ratio on 2. 2.84%
MMG Tasks
Then Cross image a
Tam Reraow flooes digroedd
Caussaa Fecinas vidao Thrn oyde qaoeraed mrazeseedd saccse fall....
Produce sada chiqu eficci
Yern Allaasa the arli backgrooed
doreek
wholfict...
Eaanspli
Exanaplc
lmage
Juticnaaa image
Text to Action
Ratio per web: 19.5%
Image to Action
Ratio on web: 1.1 7%
Frontier Organizations on MixEval-X
Agent Tasks
Instrustoon Puckace ssak day pard de likkk books
Instrustion Hove la aacoe Nettindan
(Pickuddejeet), [Furthject]]; [Usedbject), "Speakl,
BOC
Sertigg thmm best
[oweigetel [Tap],
Dajecte
[Roud] [Screll
ceroact,
raphy ...
Already Freeated [Rrvigate)
Already Excested Siegi
<hotsinga,, [Sc (ScoollDown)
[Hsrigstton]] crestiual, entranceo
Auseen
Mow
Tap] capos fictfoicct
[Icentt [LcenttFydbject) cescoat beothr,
Tee
Te07
Figure 1: Mi xEval -X encompasses eight input-output extended. Its data points reflect real-worid task distributions. The last grid presents the scores of frontier organizations' flagship models on MixEval 1-x, normalized to a 0-100 scale, with MMG tasks using win rates instead of Elo. Section D presents example data samples and model responses.
combinations and can be further
ABSTRACT
Perceiving and generating diverse modalities are crucial for AI models to effec- tively learn from and engage with real-world signals, necessitating reliable eval- uations for their development. We identify two major issues in current evalua- tions: (1) inconsistent standards, shaped by different communities with varying protocols and maturity levels; and (2) sigmificant query. grading, and generaliza- tion biases. To address these, we introduce Mi MixEval-X. the first any-to-any. real-world benchmark designed to optimize and standardize evaluations across di- verse input and output modalities. We propose multi-modal benchmark mixture and rectifiitonnti pipelines to reconstruct real-world task distributions, ensuring evaluations generalize effectively to real-world use cases. Extensive meta-evaluations show our approach effectively aligns benchmark samples with real-world task distributions. Mcanwhile, MixEval-2 model rankings corre- late strongly with that of crowd-soureed real-world evaluations (up to 0.98) while being much more efficient. We provide comprehensive leaderboards to rerank ex- isting models and organizations and offer insights enhance understanding of multi-modal evaluations and inform future research.
1 Corresponneece to: Jinjie Ni <jinsjenninus edu.sg> Now at Google DeepMind.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Evaluations are crucial in the Al community for two main reasons: (1) they provide early signals to developers, assisting in the refinement of data and model, and (2) they guide users in selecting appro- priate models for specific tasks. Thus, evaluations offer feedback signals to the entire community, driving model optimization.
Recently, models with diverse input (Achiam et al., 2023; Xue et al., 2024; Chu et al., 2024) and out- put (Betker et al., 2023; Yang et al., 2024; Majumder et al., 2024) modalities have been developed, with evaluations tailored to each. However, these communities often evolve in isolation, resulting in significant disparities in evaluation standards and methods. For example, while the large lan- guage model (L.LM) community has hundreds of multi-task evaluations spanning various domains and methodologies, the audio language model community still relies heavily on task-specific bench- marks (Chu et al., 2024). This fragmentation results in inconsistent evaluation signals, misleading and bottlenecking the overall progress of various modalities.
Additionally, existing evaluations exhibit significant biases in query, grading, and generalization. Query bias occurs when evaluation tasks deviate from real-world task distributions, leading to dis- crepancy in evaluation results and real-world performance; grading bias arises from unfair scoring paradigms, and generalization bias stems from evaluation contamination. These biases skew eval- uation signals, hindering model development. To address these issues, MixEval (Ni et al., 2024) proposes low-bias paradigm for LLM evaluations (Text22eext). MixEval aligns benchmarks with real-world task distributions by matching web-mined queries to similar benchmark tasks. Its ben- efits include: (1) comprehensive, less biased task distribution based on large-scale web corpus, (2) fair grading due to the ground-trutte truth--atu--- paradigm, (3) dynamic benchmarking via a low-effort update pipeline, mitigating generalization bias, (4) accurate model ranking with 0.96 correlation to Chatbot Arena, (5) fast, cost-effective, and reproducible execution, requiring only 6% of MMLU's time and cost, and (6) challenging problems with significant room for improvement.
To this end, to optimize and standardize evaluations across AI communities, we propose MixEval-X, the first any-to-any real-world benchmark optimizing benchmark mixtures for wide range of input-output modalities. MixEval- X consists of eight subsets, each with distinct input- output modality combinations, categorized into three types: multi-modal understanding (MMU), multi-modal generation (MMG), and agent tasks (Figure I). These modalities are not only the dominant ones web queries but also central to various communities.
Specifically, we first use MixEval's web user query detection pipeline to gather a well-distributed set of real-world queries spanning diverse input-output modalities. For MMU tasks, we construct large-scale multi-modal benchmark pools from existing community benchmarks, prioritizing query- based ones like question-ansee answering and examination tasks due to: (1) the query- nature of real-world tasks, and (2) the convergence of AI tasks toward natural language queries for multi- task learning (Wei et al., 2021). We then match web queries to similar query-based tasks from the benchmark pool to reconstruct the benchmark distribution. This is followed by an automatic quality control step to climinate the wrong or extreme samples. Additionally, we perform rejection sampling to select more challenging MMU tasks while preserving real-world distribution alignment. For MMG tasks, which are open-ended, we implement an iiaapaatio-rrectffcaaii adaptation step creates real-world tasks from web queries using frontier models, ensuring alignment with real-world task distributions, and the rectification step automatically fixes errors and distribu- tion deviations. For agent tasks, lacking general-don general-domain benchmarks, we use similar adaptation- rectification pipeline to recreate task distributions and annotate reference answers. Optional human inspection was adopted to increase the annotation quality. The efficient evaluation creation pipelines for MMU, MMG, and agent tasks allow periodic data refreshes to mitigate contamination. Our meta- evaluations show that: (1) Mi xFva 1-x data closely aligns with real-world task distributions, and (2) Mi xExxl-XXXX evaluation results strongly correlate with real-world user-facing evaluations (up to 0.98), while being significantly more efficient.
pipeline where the
Why use MixEval-X? (1) It extends all the benefits of MixEval to multi-modal evaluations, in- cluding comprehensive and less biased query distribution; fair grading (except open-ended tasks); dynamism: accurate model ranking; fast, cost-effective, reproducible execution; and challenging nature. (2) It establishes unified, high standards across modalities and communities. For single- modality models, it ensures its evaluation keeps up with the sate-of-the-an standards; for multi-
2

---

Published as a conference paper at ICLR 2025
modality models, it ensures consistent, high-standard evaluations across modalities, preventing any from becoming a bottleneck. (3) Beyond model evaluation, Mi xEval-X benchmarks different or- ganizations (Figure 1) with balanced dimensions (modalities), unlocking
level of evaluation.
Research Contributions (1) We propose the multi-modal benchmark mixture and adaptation- rectification pipeline to optimize Al evaluations, providing an efficient approach create low-bias any-to-any benchmarks with real-world distributions. (2) We introduce Mi Mixval--X -x, the first high- standard, unified real-world benchmark with diverse input-output modalities, reducing the bias and heterogeneity in Al evaluations. (3) We present comprehensive evaluation results, reranking models (4) We conduct extensive meta-evaluations, offering valuable insights for guiding Al evaluations and future research.
and organizations for wide range of communities.
2 MIXEVAL-X
Train & Test
Modality Annotation
Multi-modal Web Queries
LLM Classifier
Filter
EKIM
Quality Control
Benchmark
image text
Sampling
Mixture
video to
(ormelbann))
audio to text
text to image
Reel)
Querent
(Annotation &) Inspection
text to video text to audio text tn action image to action
Rectification
Adaptation
MixEval-X
Figure 2: The overall pipeline for creating MixEval-2
In this section, introduce the methods used construct the various subsets of Mi xEva 1-x and their respective grading mechanisms. As shown in Figure 2, MMU tasks are built with benchmark mixture, while MMG and agent tasks are created with claptation-rectfficatii pipeline, both de- signed to align evaluation tasks with real-world distributions. The grading for MMU tasks is robust due their soound-truthhbaat data creation pipelines. Moreover, we carefully refine annotation accuracy and task difficulty to ensure the quality and usage potential of MixEval-X.
nature. All nub-benchmarks are dynamic, enabled by the efficient
2.1 WEB QUERY DETECTION
We create the same web query detection pipeline as MixEval to detect real-world user queries from Common Crawl (Computer, 2023). Both recall and precision crucial to ensure the query distri- bution reflects real-world scenarios. Therefore, we developed two benchmarks to evaluate our query detector's performance. The first benchmark includes self-collected in-the-wild user queries pos- itive samples, with non-query datasets such as Wikipedia (Foundation, 2022) as negative samples. The second, higher-quality benchmark contains positive and negative samples hand-picked by our authors from in-the-wild query and non-query datasets. In preliminary experiments, direct prompt- ing of open-source language models performed poorly on our benchmarks. Thus, we devised loop to ensure high recall and precision cost-effectively. We started with a detection phase to gather training data. Testing various open-source LLMs,, Vicuna 33B (Chiang al., 2023) achieved a high recall (>99%) on our test sets with careful prompt engineering ensuring that very few positive sam- ples were missed initially, In this phase, we detected around 20k queries using Vicuna 33B over a subset of Common Crawl. We then used GPT-4 to more accurately label these data positive or negative samples, and used the resulting data to train Vicuna 33B. The trained Vicuna 33B achieved high recall (>99%) and precision (>98%) our benchmarks and detected 2M user queries from the Common Crawl subset. Finally, we prompted GPT-4 Turbo to further filter and classify them into various modalities, extracting well-distributed multi-modal queries for Mi xEva 1 -X creation.
3

---

Published as a conference paper at ICLR 2025
2.2 MMU TASK CREATION
Benchmark Mixture We perform benchmark mixture to mitigate query bias in MMU tasks. As illustrated in MixEval (Ni et al., 2024) and further Figure 9, current benchmark query distri- butions deviate from real-world use, limiting the generalizability of evaluation outcomes. Using the MixEval web query detection pipeline, which trained precise query classifiers to extract well- distributed queries from the web, we crawl multi-modal user queries from web and map this web query distribution onto the constructed benchmark pool containing numerous ground-trute souunddtrutt-bseee benchmarks. We sample problem-answer pairs from this benchmark pool by selecting the most sim- ilar one given a web query, constituting a new benchmark with natural ground- process is based the similarities between the sentence embeddings (Reimers & Gurevych, 2019) of benchmark text queries and web queries. As such benchmarks exist only for MMU tasks, we apply benchmark mixture exclusively these modalities. Due varying community maturity, benchmark pools for certain modalities, such as Audio2Text, are less extensive compared to more established ones like Text2Text and Image2Text. However, a key advantage of MixEval- -X is its capacity for self-refinement, enabling the benchmark pool to grow as the community develops. The benchmark pool composition is detailed in Section G.
The matching
Challenge Set Sampling and Dynamism The rapid advancement of frontier and open com- munities introduces two key challenges in model evaluation: saturation, where further score im- provements are limited, and contamination, where models overfit to the test data. To enhance model differentiation, we applied rejection sampling (Ni et al., 2024) to select more challenging MMU tasks while preserving real-world distribution alignment. The effectiveness of this strategy is demon- strated later by the low scores on the hard split in Section 3.1, and the close distance between hard split queries web queries in Figure 9. Since Mi xEval-X. benchmark mixture pipeline is fully automated and updatable within minutes, refreshing data points is efficient and helps mitigate testset overfitting. Additionally, the benchmark pool also integrates newly released benchmarks to mitigate contamination. However, this dynamism alleviates but does not fully resolve contamination. Further discussions on benchmark mixture contamination can be found in Ni (2024).
Quality Control An inspection step is used to enhance the benchmark quality. We focus on the entries where frontier models erred most, excluding those that most models gave the same answer different from the ground-truth. Cases with extreme inputs were removed to streamline evaluation.
2.3 MMG AND AGENT TASK CREATION
Copntattonn-eetificattini Pipeline For MMG and agent tasks lacking natural general-domain benchmarks, alternative methods are needed to recreate real-world task distributions. MMG tasks simpler construct, being open-ended with no reference answers. Since web user queries are not clean and challenging enough for MMG models, we developed an retttattion-ettifiaattonnn pipeline that transforms them into well-formatted, challenging tasks. In adaptation, language model mod- ifies the query to match the required complexity and format while maintaining user intent. In rec- tification, the model inspects and corrects the task's logie, complexity, correctness, and alignment with the original task. The resulting MMG tasks have two turns-a generation turn that instructs the model to generate content and an edition turn that instructs the model to edit the generated content in the last turn.
Constructing agent tasks is more demanding, requiring careful annotation of reference answers. The task design follows the MMG approach, using the ceappaaaioo-rctfficatatee pipeline. To annotate answers for Text2Action and Image2Action tasks, frontier LLMs and VLMs provide initial annota- tions, refined through automated rectifications. Both MMG and agent tasks have an optional human review step. These sub-benchmarks are also dynamic due to the efficient data update pipeline. De- tailed pipeline prompts are shown in Section E.
2.4 GRADING
Grading bias undermines evaluation accuracy, even for ground-truth tasks with narrow answer spaces. As noted in Ni (2024), rule-based parsers unstable when grading across multiple models and ground-truth-bur benchmarks, while language model parsers provide a more reliable alterna- tive. We use model-based parsers to grade tasks with narrower answer spaces, such MMU and
4

---

Published as a conference paper at ICLR 2025
Table 1: The statistics of MixxEvl--i -X subsets. All MMU tasks have both free-form and multiple- choice tasks except Audio2Text and Hurrd2Text-Hard.
Tasks
Turns
Avg. per Query
Avg. # Inputs
Avg. Input Length
Min Imput
Input Length
English Ratio
Type
Length
Image2Text
MMU
2,000
12.1
1.0
99.2%
Image2text Hard
MMU
1,000
14.7
1.0
99.4%
Video2Text
MMU
2,000
10.2
1.0
56.5
(.5
238.4
100.0%
Video2Text-Har Text-Hard
MMU
1,000
10.8
1.0
70.7 (8)
(.4
238.4 (6)
100.0%
Audio2Text
8.2
(6)
(c)
Andio2Text-Hard
MMU
500
9.2
1,0
54.6
5.6
149.5
100.0%
Text2Action
Agent
100
14.3
1.0
139.7 (toks)
35 (toks)
214 (toks)
99.0%
Image2Action
Agent
100
14.2
1.0
61.7 (toks)
34 (toks)
100 (toks)
100.0%
Text2lmage
MMG
200
2
31.5
100.0%
Text2Video
MMG
48.0
100.0%
Tox2Audio
MMG
200
54.5
100.0%
agent tasks. For MMU, small language models assess answers given the problem, model response, and reference answers. For agent tasks, which have comparatively broader answer spaces, we use frontier LLMs to grade on scale of 0-10, given the reference answer. MMG tasks, which are more open-ended, are harder to evaluate, where traditional automated metrics such as FID, CLIP, and FVD fail to capture nuanced quality and user preference (Jiang et al., 2024). Thus, we employ crowd-sourced workers to rank model pairs, computing Elo ratings using the Bradley-Terry (BT) model (Bradley & Terry, 1952), which is statistically robust for open-ended tasks (Chiang al., 2024; Jiang et al., 2024). Alternative grading methods are also possible for MMG tasks. In Sec- tion 4.2, we explore the correlation between model evaluations and human preferences, advocating for more research into model-based MMG grading. Note that for MMG tasks, MixEval X of- fers a task set that highly representative of real-world use cases while remaining flexible in the grading methods. Users may choose between automatic metrics, model-based evaluation, human judgment depending on the specific application. The grading prompts are presented in Section F.
Table presents the statistics for the Mi xEva 1-x benchmarks. We regulate task count and input lengths for efficiency, especially for MMU tasks, which often require longer inputs. Input means the input contents other than the textual query. Input lengths are measured in seconds for Video2Text and Audio2Text, and in text tokens for other modalities (NLTK tokenizer (Loper & Bird, 2002)). MMG tasks generally have longer queries by design. Figure 40-45 details the distribution of the benchmark pool across MMU subsets.
3 EVALUATION
We present the evaluation results, and the settings are detailed in Section C.
3.1 MMU TASKS
Image2Text In Image2Text tasks, models generate language responses based on user-provided im- ages and text. We evaluate a broad range of Image2Text models due to its established community. Figure presents the leaderboard. Proprietary models like Claude, GPT, Gemini, and Reka series outperform open-source models, with Qwen and Llama leading the latter. Our analysis shows that input resolution limits and model size are key factors in rankings, while model architecture, training methods, and input formatting also influence performance. Most models use encoder-decoder architecture, as decoder-only models remain underexplored in vision-language tasks.
Video2Text In Video2Text tasks, models generate language responses based user- user-provice rer-provided videos and text. All models are evaluated with the same number of frames, except for specialized models with frame limitations. The results are shown in Figure 4, with proprietary models again dominating. In addition to factors like model size and architecture, the maximum supported input frames significantly impact Video2Text performance. Models with limited frame capacity perform worse, especially on long video datasets like ActivityNet and EgoSchema, highlighting the need for further research in long video understanding.
5

---

Published as a conference paper at ICLR 2025
Claude 5,5 Sonnet
76.9
46.2
76.0
15.1
94.6
90.3
62.5
78.8
31.0
48.9
GPT-4-
76.6
GPT-4V
75.0
44.6
75.6
68.0
92.1
89.
53.7
79.2
$1.9
40.6
Qwen2 VL-12B
74.8
43.4
71.5
67.5
90.6
90.3
80.4
25.4
27.8
Gemint 1.5 Pro
74.2
42.2
72.2
77.2
85.6
86.88
63.7
76.7
29.7
44.4
1amm 32 92
13.0
40.6
73.3
62.9
92.7
90.9
61.6
89.8
28.9
50.1
Intens LL2-258
71.5
41.5
71.5
90.3
91.2
58.2
70.2
52.3
28.6
Cluude Opus
69.5
41.1
72.0
66.55
844.2
86.77
56.9
66.9
344.9
44.4
833.1
LL-VA-16-34B
68.1
37.5
70.4
60.4
71.0
$1..8
4856
58.8
31.9
56.8
Clande
67.8
80.3
78.6
32.3
30.8
Reka Core
67.4
37.3
66..5
71.1
76..5
79.9
56.9
59.6
25.0
39.1
Raka Fash
67.4
36.6
73.6
53.8
71.3
76.8
62.5
32.8
23.3
IntmmVL-Chr-V1222
67.2
36.0
70.7
54.8
51.8
76.3
60.0
59.2
25,4
33.8
Quen-VL-PLUIS
67.0
35.9
66.2
56.9
84.1
83.1
57.5
52.7
19.8
27.1
Clande
66.1
37.5
88.33
32.88
45,9
Gemini 1.0 Pro 66.1
35.0
67.66
60.9
70.3
81.3
55.7
51.8
29.3
19.8
54.7
V--V--34
$8.5
30.6
68.0
53.8
21.5
59.7
53.3
41.4
27.6
29.3
OM.iLMM-11BB-
58.2
54.8
42.3
70.2
48.6
32.3
DeepSeck VL-rB-Chut
56.7
26.5
61.3
41.1
39.4
69
50.8
32.0
21:1
14.3
Y:-L--B
55.4
30.1
65.6
45.7
23.6
62.3
28.0
27.6
19.5
InfMMM-Zephyy-T11
53.7
29.4
62.5
44.2
21.9
46.1
46.1
27.6
26.7
25.6
MinCCMM-V 51.5
25.9
59.1
32.0
53.2
76.6
40.8
32.2
23.7
18.0
Matco-VL- 50.5
24.1
19.0
27.8
LL 1.5 13B 50.2
26.0
$6.9
32.5
22.4
53.7
42.9
24.3
19.0
24.8
PLIGG-WW.2 48.9
22.5
$7.5
28.9
26.9
$9.7
39.8
29. 4
28.0
10.5
SPHINX 47.5
23.8
16.4
$1.0
24.5
19.8
EItraBBLIP XXL 46.2
21.5
$8.0
31.0
11.2
41.7
44.3
24.5
19.4
28.6
InstrucBLLP TXXL- 45.5
22.9
53.1
32.0
14.5
44.5
12.9
21.1
18.8
BLIP2 FLAN-TSXXI-- 45.2
21.6
55.1
33.0
13.5
46.3
42.2
29.6
22.8
17.3
BLIP2 FLAN-TSXL
43,0
20.0
52.5
33.5
16.3
40.9
39.2
9.4
23.3
11.3
Adapt Hayy-
41.1
21.6
LLaMA Adapter2 7D
36.6
20.4
42.5
12.5
15.6
23.7
44.5
25.1
18.1
14.3
Miss GiT
32.1
15,8
38.2
25.4
15.4
23.4
33.7
18.4
15.5
13.5
ImageZText
ImsgeT Text Hard
SEED (Mixed)
MMMU (Misel)
DocVOA (Mised)
TearVOA (Mixal)
VisWi: (Mixed)
InfvemphicYOA (Mized))
SEED -Hrd
MMMU (Mixed)
-Hard (Mixai)
Figure 3: The evaluation results of prominent models on Mi MixEval- 1-X Image Image2Text- Hard, and their subsets. Proprietary models are highlighted in blue. See Section H for details.
Claude 3.5 Sonndt
74.2
45.5
73.3
76.6
64.8
79.4
76.4
78.9
60.4
39.4
72.7
Gemini 1.5 Pro
71.8
38.1
65.2
64.8
82.6
82.9
74.4
75.7
43.2
68.5
GPT-4V 71.0
40.0
63.4
78.2
69.5
77.9
69.5
78.5
37.2
37.8
78.5
27.3
Gemini 1,5 Thab
66.3
33.9
59.0
67.4
70.3
73.8
61.4
72.3
26.7
$1.2
LLaVA Oiieeinnnnnni 22B-VV
64.7
77.0
64.4
71.2
70.6
35.6
28.3
Qwent VL-B
64.2
31.9
54.3
14.7
52.1
74.9
62.6
68.9
27.2
26.0
LLaVA-Neit LaaVA-Nat-Vibo.3tt
V4B
63.1
28.4
56.1
68.6
62.7
74.0
62.8
68.0
26.7
38.6
58.7
LLaVANN Vidco-B
58.7
27.2
53.2
62.1
44.3
72.5
61.0
74.4
25.9
33.1
Rela eda:
58.7
27.3
$1.7
72:4
46.6
69.1
59.3
65.2
29.0
22.8
LLaMA-VID
55.6
23.8
$2.9
60.9
36.0
72.8
67.1
19.1
17.3
VideoLLaVA
55.3
22.6
$1.7
64.0
39.4
66.7
61.9
64.7
18.2
26.0
Vidao-ChatCPP-
46.4
20.7
25.4
24.7
mPLLG--idee
39.1
17.8
41,5
36.4
23.3
71,9
56.7
61,8
22,7
7.9
VideoZText
Video2Tent Hard
AaivitsNa-OA
HowToOA
TVQA (Mised)
MSYD-ON (Mixed)
Ned droetas (Misel)
TGIF-ON
AntivityNar-0 -A
TVQA -Hard
(Mised)
(Mixed)
(Mise)
Hard
(Mised)
(Mixed)
Figure 4: The evaluation results prominent models on MixEval-2 Video2Text, Video2Text- Hard, and their subsets. Proprietary models are highlighted in blue. See Section H for details.
Audio2Text In Audio2Text tasks, models generate language responses based on user- user-provided audio and text inputs. This field less developed compared to vision tasks, leading to smaller benchmark pool and fewer models. As shown in Figure 5, the Gemini series is the only proprietary model natively supporting Audio2Text, with Gemini 1.5 Pro ranking first at 62.7% accuracy on the general split. Owen2-Audio- leads
split but showing room for improvement the Audio2Text-Hor the Audio2Text Hard split. Rankings are primarily influenced by language model quality and input formatting strategies.

---

Published as a conference paper at ICLR 2025
3.2 MMG TASKS
The results for MMG tasks are shown in Figure 6. We em- ployed hundreds of human eval- uators from Amazon Mechani- cal Turk to assess model outputs using pairwise-ranking ap- proach, as automatic metrics fail to capture the nuances in output quality (Jiang al., 2024). We report only the overall scores, while MMG tasks consist of two turns-a generation turn and an edition turn. Turn-level scores are presented in Figure 46. See Section H for model details.
Gemini 1.5 Pro
62.7
24.0
67.4
53.4
26.8
21.7
Gemini 1.5 Flash
60.1
23.0
67.1
46.9
27.4
19.7
Soon2-Audo-7B-Insteee
64.7
22.5
56.6
24.6
63.1
44.0
29.9
20.0
SALMONN-I3B
52.5
20.9
57.6
41.4
14.9
25.4
Qwen-Audio
52.4
16.0
61.5
33.8
19.0
12.8
Qwen-Audio Chat
50.2
20.0
55.7
39.4
19.8
19.7
SALMONN-7B
38.9
46.6
20.6
11.6
Pengi
22.6
26.9
14.4
12,5
Audio2Text
AudioTText
Clotbe-AQA
DAQA (Mixed)
Clotho-AQA -Hard (Mised)
DAQA -Hard (Mixed)
-Hard
(Mixad)
Figure 5: The results of prominent models on MixEval-X Au- dio2Text, Audio2Text-Hard and their subsets. Proprietary mod- els are highlighted in blue. See Section H for details.
Text2Image In Text21mage tasks, models generate images based human prompts. Like Image2Text, Texr2Image has well-developed community, with Flux (BlackForestl abs, 2024) achieving the highest Elo score among the evaluated models, as shown in Figure 6(a). Our analysis reveals that image quality, particularly realism, significantly impacts human pairwise evaluations. Although DALL-E HD (Betker et al., 2023) shows high quality, it ranks lower in realism. A case study is shown in Figure D. Instruction-followim ability also strongly influences human evaluations.
Video In Text2Video tasks, models generate videos based textual prompts. Figure 6(b) presents the Elo rankings for various Text2Video models. Human evaluators tend to prefer videos with higher quality, realism, smoothness, and adherence the prompt. Most models struggle with generating realistic human faces, and unnatural or distorted faces greatly reduce human preference. In contrast, video length has less influence; for example, Show-1 generates longer sequences, but face distortions significantly lower its preference among crowd-sourced evaluators.
Text2Audio In Texi2Audio tasks, models generate audio based textual prompts. Figure 6(c) shows the Elo rankings for Text2Audio models. AudioLDM2 ranks first in human pairwise eval- uations, followed by Make-An-Audio-2 and Stable Audio. However, we find that our tasks are generally too challenging for most Text2Video and Text2Audio models, especially Text2Audio. Consequently, the Elo scores reflect only relative rankings, with even AudioL.DM2 failing to pro- duce high-quality audio or follow instructions effectively. This highlights significant performance gap across different communitics.
Workers: 768
# Workers: 832
Workers: 742
1085
1047
1025
1035
1038
1028:024
1020
1019
1014
1014
1010
1006
993
997
995
961
984
937
917
Flux
DALL-E E.LL-E HD
PixArtAlpha
PlayGround V2.5
PlayGround v2
SD3
Stable Cascade
SD1.5
SDXL
HetShot-Xt
CogVideoX-5e SB
LaVie
VidenCCafter2
ZeroScope ModelScope
Show-1
AudioLOM 2
Makre- An-Aiii
Stable Audio
Tango 2
ConsistencyTTA
AudioGen
Magnet
V2
(a) Texi2
(b) Tex12
(c) Text2Audio
Figure 6: The overall Elo scores of MMG models the MixEval-X MMG subsets, with error
bars representing the 95% confidence intervals for the ratings. These scores are derived using the Bradley -erry model, based crorddsourced user preferences. Additionally, the number of human evaluators per subset is provided for reference. The turn-eevel scores are shown in Figure 46. 46.
7

---

Published as a conference paper at ICLR 2025
3.3 AGENT TASKS
Text2Action Text2Action tasks involve models planning API- level actions based on textual inputs describing the environ- ment and a user prompt. This setup simplifies weal- agent tasks to evaluate the action- planning capabilities of L.Mss, offering more flexibility for op- timizing task distribution. Fig- ure 7 presents the results for the Text2Action subset. The model rankings differ from Text2Text tasks (Ni et al., 2024), suggesting that strong text understanding does not guarantee proficiency in textual agent tasks.
7,54 7.49, 7,046966 6.67.477 6.6661336611 6.8 6.14 6.12
4.477444 4.10
CVT-4-Jarta
Gonnle 1.5 Fre
Mannll Lagess GPT-4e
Clasde 3.5 Stoeet
Rele Cere
Gemave t-t Jnstrct
Reka Fiss
Claste Hell
Mital-Mfediun Mediimn
Graini Flats
Qwen 2720-2222liin Arri--nsssss Instust
LL 1-788 Jnaraei
Mitral Soail
CPT-4o OPT-4o-Mall Miil
LLaMA 1-88 Insinie
Yos 1.1-8--- Chi
GFF L-T-S--arra
Qwean 2-an----B Datarri Yo 1--9 Chat
Roka Edge
Figure 7: The evaluation results of prominent models on Text2Action. See Section H for details.
Image2Action In Image2Action tasks, models with both image and text input capabilities plan API-level actions based on the observed envi- ronment (presented as an image) and the user prompt. Figure presents the evaluation results for the Im- age2Action subset of Mi xEva 1-x. The rankings of vision-language models (VLMs) differ significantly from those in Image2Text tasks. Notably, some open-source models, not aligned with RLHF similar techniques, often produce shorter or repeated action sequences, leading lower scores.
8.5 5,24 7.92
7.38.1166777
7.5
5.545.94
5.0
99 3,103,03
Clask 3,5 Somer
CPT4V
GPf--oo
Chuste Opuss
Cizute Somer clask Hakk
Gemine 1.5 Pro
Liauu 1.2 wit
Inum Chat V2
Quam- Qem--V--Mte VL-MAX
Owee PLL1
IaiM Zaplo---
Dospleak VI. Clas
Mank
Intrrd XrC--CCCmmaine Comgwer222
VE-VL 34
Qrea V-V 7222
La Latthhll. 1M1
EL 6-M4
Figure The evaluation results of prominent models on Image2Action. See Section H for details.
4 META EVALUATION
4.1 DISTRIBUTION ANALYSIS
Setup We aim to analyze the task distributions of MixEval-X. While many benchmarks and datasets are documented, we focus on their actual distributions in practice and their comparison real-world tasks. In Figure 9, we randomly sample 1000 task queries from each dataset, reduce their sentence emheddings to 2D using t-SNE, and visualize the distributions. Dimensionality re- duction is performed in the same space for benchmarks with the same modality, using identical color schemes to facilitate direct comparison, i.c., datasets of the same modality are comparable in terms of their distributions. To further examine topie distributions, we segmented the aggregated queries of each modality (e.g., Image2Text) into 16 spatial patches in the 2D space. From each patch, we uniformly sampled 100 queries and used GPT-4 summarize the topics (Figure 35-39). MMG benchmarks are not analyzed as their prompts are caption-like, making them non-comparable.
MixEval-X tasks closely align with real-world task distributions while being the most com- prehensive and diverse. In Figure 9, C-Dist measures the cluster distances between each bench- mark and corresponding web queries. Benchmarks are ranked by proximity to web queries, with closer ones ranked higher. MixEval-2 sub-benchmarks, including Image2Text, Video2Texi, Au- dio2Text, Text2Action, Image2Action, and their hard versions, are the closest to web queries, indi- cating their distributions strongly resemble real-world tasks. Moreover, Mi xEval-X benchmarks visually cover more diverse topics than others. This also illustrates that both the multi-modal bench- mark mixture and the ddlaptationrrectffcattoo pipeline effectively aligns benchmarks with real-world
8

---

Published as a conference paper at ICLR 2025
Imade
Imap22Tett
MMBench
VieW
MM.Vet
DoeViA
Image2Text Open
IntographicKOA
Clat 3.79
C Dis
Dittt: (042
COA
Sclenca
C-Dint:
G-Ditt:
G-Diet: 12
C-Diee:
G-Dist:
Hallusion@enchh
Char1Q4
N2D
Q-Gench
MathVista
Video2Text Web Queries
Diei:: 19.44
DisE20.11 T8
DIE
VideoZText
2odeo2Toxt-Hard
Video MME
VideoVista
MSRVTT-QA
HONTOQA
Dist:
Diat 5:77
C Diait 7.00
C: Disri 7:75
C-DGE: R:51
Video2Textt-
NectOA-choic
TVDA
9-D
Social-Q-2.0
IGIF-QA QA
EgoSchhema
MSVD-QA
STAR
Perception lest
Diet: TE.
Dist: 7.
Dist: 19.72
DiDE 17
DIES: 50
Dist:26.es 26.6
Audio2Text Hard
Audio2Text Open
10.38
DADA
Ci FAR
TextZAction Web Queries
Text2Action
WebArens
TooiBench
Dist:
Dietio0 8g
0.t: 22
Ciutt: 12.47
D2 13
Dist
PlanBench
Action Web Querles
Imaco2Action
Android In The Zoo
AUFRED
VizualWeb/veen
Figure 9: Task distribution of various modality benchmarks, with each modality uniquely color- coded. Benchmark data points (orange dots) are plotted against the detected web queries (blue dots) for the corresponding modality. The sentence embeddings of the queries were dimensionally re- duced into unified 2D space, enabling direct comparison of topic distributions across benchmarks. Benchmarks are sorted by their cluster distance (C-Dist) from the corresponding web queries.

---

Published as a conference paper at ICLR 2025
distribution. Note that WildVision, the only real-world dataset available for these modalities, also aligns closely with web queries, Image2Text, and Image2Text-Hann reinforcing this conclusion.
The task distributions of most existing benchmarks deviate from real-world task distributions. This deviation expected, as most benchmarks are not designed with real-world tasks mind. This creates challenge since we expect evaluation results to generalize real-world applications where models are deployed (Ni, 2024). Nonetheless, existing benchmarks, though deviated, can still be useful for evaluating particular aspects of model. However, some benchmarks' actual distributions may not match their creators' claims, risking the use of inappropriate benchmarks for specific tasks. To better understand these benchmarks, users can refer Figures 35-39, which use GPT-4 to summarize task topics from specific 2D grid locations in Figure 9, enabling more interpretable and accurate benchmark selection.
4.2 CORRELATION ANALYSIS
MixEval-X demonstrates strong correlation with real-world user-facing evaluations. A key feature of MixEval-X its alignment of benchmark task distributions with real-world tasks. Beyond distribution analysis, we sess this alignment by evaluating the correlation between MixEva--X results and real-world evaluations. While many communities lack stable real-world evaluation leaderboards like Chatbot Arena (Chiang al., 2024), the Image2Text community has two comparatively stable user- facing leaderboards: Vision Arena (Lu et al.. 2024b) and Arena (Vi- sion) (Chiang et al., 2024). Our Image2Text results show strong Spearman's model ranking correla- tion with these, with 98.1% % corre- lation to Vision Arena and 96.3% to Arena (Vision); Image2Text-HHe shows 94.5% and 95 % respectively. These high correlations, along with findings in Ni al. (2024), highlight the effectiveness of our benchmark mixture approach. Although corre- lations for other modalities can't be verified present due to the lack of stable real-world evaluations! we will assess them once suitable evaluations are available.
1050
Model Judge Score
7
1025
6
Elo Score
1000
I
975
950
3
Elo
GPT
Gemini
Claude
DALL HD
Flux
PixArtAlpha
Play PlayGr V2.5
Plound V2
Stable Cascade
SD3
SDL.5
Figure 10: Model judge scores and crowd-soureing Elo scores of the Text2lmage subset. The upper and lower er- bars represent the 1st and 2nd turn scores, respectively. Each data point an average of five different runs.
Multi-modal language models evaluate MMG tasks differently from crowd-soureed human preferences. In this study, we employed 700-800 crowd-sourced workers for pairwise human pref- erence evaluations MMG tasks. Large-scale human evaluations provide meaningful assessments due to the Wisdom of the Crowd effect (Yi et al., 2012) and their relevance to real-world applica- tions, but they are time-consuming and costly (Ni et al., 2024). Thus, we are exploring cheaper alternatives, like L.M--s-judge evaluations, which, though considered biased (Zheng et al., 2023; Ni et al., 2024), have shown promise in open-ended Text2Text tasks. We compared frontier model judges against crowd-sourced results (Figure 10). Nearly all model judges judges showed low correlations with human preferences (78% on average, see Table 2), suggesting that multi-modal models eval- uate MMG tasks differently from humans, consistent with Zhang al. (2024b). We focused on Image2Text results due to a lack of reliable judge models for video and audio tasks. Interestingly, correlations between model judges were relatively high (85%-95%) indicating a potential shared bias. These findings highlight the need for further research into cost-effective, low-bias grading methods for MMG tasks.
"GenAT-Arema real-world platform for MMG models, and the Video2Text leaderboard in Vision Arena remain unstable due limited votes and models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
We thank Graham Neubig, Yuntian Deng, Jason Phang, Piotr Nawrot, Luca Soldaini, Guanzhi Wang, Qinghong Lin, and Fanyi Pu for insightful discussions and pointers.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican. al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
Anthropic. Claude 3.5 sonnet. 2024a. UBL.ttpss//wwwantrrpp......l claude- 3-5-sonnet.
AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024b.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308.12200 2023.
Yatong Bai, Trung Dang, Dung Tran, Kazuhito Koishida, and Somayeh Sojoudi. Consistencytta: Accelerating diffusion- based text-to-audio generation with consistency distillation. arXiv preprint arXiv:2309. 2024.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/dalle-33 pdf. 2(3):8, 2023.
BlackForestLabs. Announcing Black Forest Labs, 8 2024.
URL https://
blackforest lab..ai.aa///.....a..........l.a haw.rfresstaaaaaaaaammnnnnnnnnna
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-355 1952.
Santiago Castro, Naihao Deng, Pingxuan Huang. Mihai Burzo, and Rada Mihalcea. Wildqa: In-the- wild video question answering. arXiv preprint arXiv:2209.0666 2022.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
7310-7320 2024.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-ing text-to-image synthesis. arXiv preprint arXiv:2310.00422 2023a.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qing- long Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. In- ternvl: Scaling up vision foundation models and aligning for generic risual-linguisti tasks. arXiv preprint ar 221::22312.1 132...4288 2023b.
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng. Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li. Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, al. Chatbot arena: An open platform for evaluating Ilms by human preference. arXiv preprint 24X.v22403.04132, 2024.
11

---

Published as a conference paper at ICLR 2025
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audie audio technical report. arXiv preprint arXiv:2407.10759, 2024.
Together Computer. Redpajama: an open dataset for training large language models, 2023, URL https: http:uuccommottgtherooooeoe /Reda ama Data.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA. December 10 htt.e///pppee/23////// harpr:/606000eaaaaaaaa-ooooorrrr
16, 2023, 2023. URL 9a 6.635455
Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems, 16:18090-18108 2023.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Intemlm-xcomposee Mastering free- form text-image composition and comprehension in lasounlanguage large model. arXiv preprint arXiv:2401.16420, 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis, march 2024. URL http:/arrxiv. org/abs/2403.03226.
Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. arXiv preprint arXiv:2407.14333 14358, 2024.
Haytham M Fayek and Justin Johnson. Temporal reasoning via audio question answering. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2283 2294, 2020.
Wikimedia Foundation. Wikimedia downloads, 2022. URL /tt.tttpps///uuupu..iiiiie..ooo.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint 2304 15010, 2023.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen. Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Pro- ceedings of the IEFE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14375- 14385, 2024.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3608-3617, 2018.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.00 003000009003300,, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2025
Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio Temporal-enhanced text-to-audio gen- halv:2305.18774 2023.
eration. arXiv preprint
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Yunseok Jang. Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2758-27665 2766, 2017.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Dongfu Jiang, Max Ku, Tianle Li, Yuansheng Ni, Shizhuo Sun, Rongqi Fan, and Wenhu Chen. Genai arena: An open evaluation platform for generative models. arXiv preprint arXiv:2406.04485, 2024.
Kyung-Min Kim, Min-Oh Heo, Scong-Ho Choi, and Byoung- Tak Zhang. Deepstory: Video story qa by deep embedded memory networks. arXiv preprint prXiv:1177.00666 2017.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprini arXiv:2401. 13649, 2024.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209.153: 2022.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. Tvqa: Localized, compositional video question answering. arXiv preprint 180.00800900166 2018.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.0372M 2023a.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.033 03326.. 2024a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023b.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed- bench 2: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13299-13308, 2024b.
Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient vision- iision-language learning by cross- modal skip-connections. arXiv preprint arXiv:2205. /2005, 2022.
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.17245 2024c.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19742. PMLR, 2023c.
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hierarchical encoder for video+ language rmni-representttion pre-training. arXiv preprint arXiv:2005.000 00200,, 2020.
13

---

Published as a conference paper at ICLR 2025
Yanwei Li, Chengyao Wang, and Jiaya Jia. vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.1744,, 2023d.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-languaa models. arXiv preprint arXiv:2305. 2023e.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023a.
Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear benchmark: Continual learn- ing on real-world imagery. In Thirty-fifth conference neural information processing systems datasets and benchmarks track (round 2), 2021.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint 1xxiv:23111.75777 2023b.
Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho- aqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Pro- cessing Conference (EUSIPCO), pp. 1140-1144 1144. IEEE, 2022.
Feng Liu, Tao Xiang, Timothy M Hospedales, Wankou Yang, and Changyin Sun. ivqa: Inverse visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8611-8619, 2018.
Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm Learning holistic audio generation with sulf-supervised pretraining, JEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating ilms as agents. arXiv preprint arXiv:2308. 03688, 2023a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, al. Mmbench: Is your multi-modal model all-around player? arXiv preprint arXiv:2307.06281, 2023b.
Edward Loper and Steven Bird. Nltk: The natural language toolkit. arXiv preprint cs/0205028, 2002.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek- vl: Towards real-world vision- langunelanguage understanding, 2024a.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wild- vision: Evaluating lasion-language models in the wild with human preferences. arXiv preprint MrXiv:2406.11066 2024b.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023.
Navonil Majumder, Chia- Yu Hung. Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Sou- janya Poria. Tango 2: Aligning diffusion based text-to-audio generations through direct prefer- ence optimization. arXiv preprint 4xXiv:240..09956 2024.
14

---

Published as a conference paper at ICLR 2025
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very long-form video language understanding. Advances in Neural Information Process- ing Systems, 36:46212 42214444,44 2023.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 1200-2209 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvga. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Meta. Introducing llama 3.1: Our most capable models to date. 2024a. URL https::/
ai.meta.com/blle htwpaaccooo/ooo///eeeeeeeee -wt.://ettataaaaeee--
Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. http:/llaaaa---cmmaeee 2024-vision-
2024b. URL htttp::/aa..mta..oo. ecde-mobile-deevices:
John Mullan, Duncan Crawbuck, and Aakash Sastry. Hotshot-XL. October 2023. URL https:
// www.:/tt....o//tttttttoc///////e
Jinjie Ni. Don't Build Random Evals: Principles for General-Purpose Model Evaluation, 2024. URL https://beneficial/ chips
Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. Mixeval: Deriving wisdom of the crowd from Ilm benchmark mixtures. arXiv preprint arXiv:2406.000 06565, 2024.
OpenAI. Hello GPT-40, 2024a. URL htttt:t/ htt.httt..////geem.....o////////
OpenAl. Gpt-4o mini: advancing cost-efficient intelligence. 2024b.
URL https:
fttee///pp-oomiiiiii httpee///g----emaiiaaaaaaaaninro intelligence/.
/
Aitor Ormazabal, Che Zheng, Cyprien de Masson d' Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, al. Reka core, flash, and edge: A series of powerful multimodal language models. arXiv preprint 2404v 12387, 2024.
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Ba- narse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36, 2024.
Pablo Pernias, Dominic Rampas, Mats L. Richter, Christopher Pal, and Mare Aubreville. Wuer- stchen: An efficient architecture for large-scale text-to-image diffusion models, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01995, 2023.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert- networks. arXiv preprint arXiv: 1908. 10084, 2019.
15

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Sciencequ: A novel resource for question answering on scholarly articles. International Journal Digital Libraries, 23(3):289-301, 2022.
Dustin Schwenk, Apoory Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint wwXiv:2310.13286 2023.
Adept Team. Adept Fuyu-Heavy: A new multimodal model, 1 2024a. mayu-heavy.
URL https://
www.adept.ai/blog/ee
Adept Team. Large Enough 7 2024b. URL https://nieerralaa///e/ttttttttt
large- 2407/.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at a practical size. arXiv preprint 2rXiv:2408.00/ 18, 2024.
InfIMM Team, Infimm: Advancing multimodal understanding from flamingo's 's legacy through di- verse Ilm integration, 2024c. URL https:
Infi -MM/
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope -ideo technical report. arXiv preprint arXiv:2308. 06571, 2023a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing Aision-languag" model's 's perception of the world at any resolution. arXiv preprint arXiv:2409. /2191, 2024.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.155 15103, 2023b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. arXiv preprint 210.022009..162, 2021.
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. arXiv preprint 2rXiv:2405.09711, 2024.
Haoning WU, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, al. Q-bench: A benchmark for general- foundation models on low-level vision. arXiv preprint arXiv:2309.141888 2023.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on com- pufer vision and pattern recognition, pp. 9777-9786, 2021.
16

---

Published as a conference paper at ICLR 2025
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Pro- ceedings of the 25th ACM international conference on Multimedia, pp. 1645-1653, 2017.
Li Xu, He Huang, and Jun Liu. Sutd-trafficqa: A question answering benchmark and efficient network for video reasoning over traffic events. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 9878-9888, 2021.
Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang. Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-tovvideo diffusion models with an expert transformer. arXiv preprint wrXiv:2408006077 2024.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 01000, 2024.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-modal large language model with modality collabo- ration. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13040-13051, 2024.
Sheng Kung Michael Yi, Mark Steyvers, Michael D Lee, and Matthew J Dry. The wisdom of the crowd in combinatorial problems. Cognitive science, 36(3):452 470, 2012.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.0466 04652, 2024.
Tianyu Yu. Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grate mraigrii correctional human feedback. arXiv preprint 12.0v::3/22.08849 2023a.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308. N00:22308.02490, 2023b.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9127-9134, 2019.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Patten Recognition, pp. 9556-9556 2024.
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong. and Louis Phill-Ph Morency. Social- iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8807-8811 2019.
David Junhao Zhang, Jay Zhangjic Wu, Ha- Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to- video generation. arXiv preprint arXiv: 15818, 2023.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024a. URL http::///llaaa.....ggttth...//b/ 2204 - vixt-
17

---

Published as a conference paper at ICLR 2025
Zicheng Zhang. Haoning Wu, Chunyi Li, Yingjie Zhou, Wei Sun, Xiongkuo Min, Zijian Chen, Xiaohong Liu, Weisi Lin, and Guangtao Zhai. A-bench: Are Imms masters at evaluating ai- generated images? arXiv preprint arXiv:2406.00 03070, 2024b.
Bo Zhao, Boya Wu, Muyang He, and Tiejun Huang. Svit: Scaling up visual instruction tuning. arXiv preprint arXiv:2307 00000007777.... 2023.
Lianmin Zheng, Wii Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm-as-a-judge with mt-hench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46633 2023.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. arXiv preprint arXiv:2307. 13854, 2023.
Deyao Zhu, Jun Chen, Xiaogian Shen, xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing rision-language understanding with advanced large language models, 2023.
Alon Ziv, Itai Gat, Gael Le Lan, Tal Remez, Felix Kreuk, Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. Masked audio generation using a single non-autoregressive transformer. arXiv preprint ar arXiv:2401.0457) 2024.
A FREQUENTLY ASKED QUESTIONS
A.1 WHY ARE THERE ONLY EIGHT INPUT AND OUTPUT MODALITY COMBINATIONS COVERED?
The eight input and output modalities represent the most common cases identified in our web query analysis and are also widely regarded as central by the AI community. Expanding the scope to include more modalities could dilute the overall quality of the work. Hence, we have chosen to focus on the key modalities for now, leaving combinations like Image2Video for future exploration.
A.2
WHY DO YOU MATCH TEXTUAL WEB QUERIES WITH TEXTUAL BENCHMARK QUERIES IN THE BENCHMARK MIXTURE OF MMU TASKS, INSTEAD OF MATCHING REAL-WORLD INPUT-QUERY PAIRS WITH BENCHMARK INPUT-QUERY PAIRS (WHERE INPUT DENOTES THE MULTI-MODAL INPUT, SUCH AS IMAGE OR VIDEO)?
practical reason is the difficulty in obtaining large-scale, multi-modal tasks that reflect real-world distributions. In this paper, we detect such tasks from the web to capture real-world task distributions using MixEval's (Ni et al., 2024) detection pipeline, which is challenging but feasible. However, matching input-query pairs poses three significant challenges: (1) If the data source is the web, as in this work, multi-modal corpora with video or audio inputs are not readily available. (2) If the data source is users actively engage with to collect input-query pairs with real-world distributions-this is time- intensive, costly, and difficult to control in terms of distribution. Moreover, without serving usable models supporting these modalities, such user inputs will not scale up. (3) Even if multi-modal input-query pairs were obtained, representing these tasks effectively remains a challenge due the lack of robust representation models for such modalities.
we would need to create platforms, like ChatGPT or Chatbot Arena, that
Another reason is that, in most tasks, the textual query captures a substantial portion of the task information, making the benchmark mixture meaningful. Our goal is to optimize task distributions rather than achieve perfect representations.
A.3 WHY NOT EVALUATE INTERLEAVED OPEN-ENDER MMU BENCHMARKS?
The primary reason is the lack of capable judges for video and audio tasks (and other modalities). Accurate evaluation requires a sufficiently robust judge model, such as GPT-4 for Text2Text tasks, which was employed in MT MT-Bench (Zheng et al., 2023). However, even with advanced models as judges, these models exhibit significant biases, including preference and length biases (Zheng et al., 2023; Ni et al., 2024).
18

---

Published as a conference paper at ICLR 2025
A.4 WHY NOT DEDUPLICATE THE MMU BENCHMARKS?
In this work, we match each web query with the most similar benchmark query and its corresponding ground truth. Given the finite size of the benchmark pool, it is likely that multiple web queries will be paired with the same benchmark entry. This duplication is expected, and deduplication would distort the real-world distribution.
B RELATED WORK
LLM Evaluations The LLM community is the most advanced among different Al com- munities. In practical LLM evaluations, three primary biases compromise impartiality: (I) proper distribution, (2) grading
query bias-evaluation queries that lack comprehensiveness bias- bias-significant bias or error in the grading process, and (3) generalization ttas-model overfitting to the evaluation data. Current benchmarking approaches either automatic user-facing. Au- tomatic benchmarks often use traditional, ground-truth- based frameworks like MMLU (Hendrycks al., 2020), which fail capture the complexity and nuance of real-world queries, though they of- fer a relatively unbiased grading process. A Alternatively, open-ended benchmarks that employ LLMs as graders, such as MT-Bench (Zheng et al.. 2023), face issues of grading bias and query incom- pleteness due to preference biases and the high cost of cutting-edge LLM judges. Furthermore, the static nature of automatic benchmarks introduces contamination over time, exacerbating general- ization bias. These biases lead to significant deviations from gold-standard evaluations, hindering model development. In contrast, large-scale user-facing benchmarks like Chatbot Arena (Chiang et al., 2024) provide more reliable metrics for model development and address the three biases more effectively. (1) They capture a diverse array of real-world queries, ensuring better query comprehen- siveness and distribution. (2) Their evaluation of varied model responses benefits from the "wisdom of the crowd" effect (Yi et al., 2012), where individual judgment noise is averaged across numer- ous samples, reducing grading bias. (3) Continuous influx of user queries minimizes benchmark contamination. Moreover, this approach steers model optimization towards practical applications, aligning models more closely with user needs. However, Chatbot Arena is costly, slow, and irre- producible. It is also not publicly accessible, limiting practitioners' ability to conduct quick and straightforward evaluations. Recently, MixEval (Ni al.. 2024) was introduced as an efficient, dynamic, and low-bias evaluation framework for LLMs, addressing the aforementioned biases.
MMU Evaluations Compared to the LLM (Text2Text) community, evaluations in the MMU domain remain underexplored. Among the modalities, the Image2Text community is relatively more ma- ture, closely following the evaluation paradigms established in the L.MM field. Existing evaluation approaches include ground-truth-based methods (Yue et al., 2024; Liu et al., 2023b), VLM-as-judge frameworks (Yu et al., 2023b), and user-facing evaluations (Jiang et al., 2024). However, user- facing evaluations in this domain tend to be unstable due to the limited availability of user votes. In contrast, the Video2Text and Audio2Text communities have only seen a limited number of ground- truth- evaluations thus far (Yu et al., 2019; Mangalam et al., 2023; Xu et al., 2017; Lipping et al., 2022; Fayek & Johnson, 2020; Lin et al., 2021). Mi MixEval 1 -X performs benchmark mixture for large MMU benchmark pool to achieve golden- saandad MMU evaluations.
MMG Evaluations MMG tasks are inherently open-ended, making ground-truth-base evaluations ineffective. Traditional automatic metries (Wang et al., 2004: Zhang et al., 2018; Heusel et al., 2017) fail to capture the subtleties of generation quality. Zhang et al. (2024b) has shown that current vision-language models are not reliable judges for MMG tasks. Currently, user-facing evaluations are regarded as the most reliable approach for MMG, either through crowdsourcing platforms or expert panels. GenAI-Arena is a crowdsourced, user-facing platform that evaluates MMG tasks via user votes, similar to Chatbot Arena. However, as with other modalities beyond Text2Text, the insufficient number of votes leads to instability model rankings. Mi Mixavl-22 MMG subsets offer several centralized sets of real-world generation and editing tasks to enable more unbiased and reproducible human evaluations, climinating the need for arena-style platforms to collect de- centralized user queries. Arena-like platforms require significantly more human resources and are inherently non-reprodueible due to user randomness. In the future, MixEval-X MMG tasks may also utilize reliable model judges to enhance evaluation efficiency.
(Multi-modal) Agent Evaluations Current agent benchmarks (Liu et al., 2023a; Zhou et al., 2023; Koh et al., 2024) are typically tightly integrated with specific environments obtain evaluation
19

---

Published as a conference paper at ICLR 2025
signals (e.g., successful task completion), making them highly domain-specinie As result, these benchmarks often fail to represent real-world task distributions, limiting their ability to generalize to certain real-world tasks. The MixEval-X agent subset introduces general-purpose (multi- modal) agent benchmark that avoids relying on environment-specife signals to assess model actions. Instead, tasks adapted from real-world user queries and framed API-calling tasks. In this setup, the model is not required to complete the tasks directly but is evaluated based on its ability to invoke various provided API actions achieve task goals. This approach expected generalize more effectively to real-world tasks due to its alignment with real-world task distributions.
C EVALUATION SETTINGS
In this section, we provide comprehensive evaluation results to offer more precise rankings for mod- els and organizations in the field. We follow official settings for all open-source models ensure fairness, For proprictary models, we use their official APIs. To avoid task-specific biases, we stan- dardize the benchmark imput formats, including prompts. Models supporting interleaving receive interleaved entries as input. Since current MMG models only accept caption-lik, prompts, we use caption rewriter (GPT-4) convert user instructions into caption- like inputs for MMG tasks.
D EXAMPLE DATA SAMPLES AND MODEL OUTPUTS
Image2Text Free-form Tasks
Query: What is the fence meant to block? Reference Answers: ["balls", "tennis balls"]
GPTT-4:: Tennis balls
Claude 3.5 Sonnet: Tennis balls
Gemini-1.5-Pro: Tennis balls
Image2Text Multiple-choice Tasks
20

---

Published as a conference paper at ICLR 2025
Query: Which work was most directly influenced by the painting?
Options:
Rude's La Marseillaise
Goya's Third of May
C. Picasso's Guernica
D. Johns's Three Flags
Reference Answers: A
GPT-4o: A. Rude's Marseillaise
Claude 3.5 Sonnet: A. Rude's La Marseillaise
The painting shown is "Liberty Leading the People" by Fug00e8ne Delacroix, which depicts scene from the French Revolution of 1830. This work, with its allegorical figure of Liberty leading lutionaries, most directly influenced Fran Rude's sculpture "L.a Marseillaise" on the Are de Triomphe in Paris. Both works share similar themes of French patriotism, revolutionary spirit, and the personification of liberty female figure. Gemini-L.5-Pro:
Image2Text Open-ended Tasks
BAMSLER
alamy
<image 1>
<image
Turn 1: <image 1> This black-and and--niitiiee image shows the logo from which manufacturer? Turn that the logo the car shown <image 2>?
Open-ended MMU datasets are not evaluated in this work. We only release the data instcad. See Section A.3 for details.
Video2Text Free-form Tasks
See the full video here.
Query: What happened to the referee after the first long jump? Reference Answers:
("measuring distance"]
GPT-4o: Cheering
GPT-4V: The referee was accidentally hit by long jumper during the event. Gemini-1.5-Flast He walked away.
21

---

Published as a conference paper at ICLR 2025
Text Multiple-choiee Tasks
See the full video here.
Query: Identify the primary theme present throughout the video, and provide key actions scenes that support your assessment of this theme. Options:
A. The primary theme present throughout the video anxicty. this is evident in the fact that e stares at her hand, touches her shoes, and looks around.
In the video, the primary theme consistently present throughout is nervousness. this is clearly evident due
fact that character e repeatedly stares at her hand, fidgets with her shoes, and constantly looks around.
C. The primary theme present throughout the vidco is clcanliness, this is evident in the fact that e washes her hands multiple times, interacts with the man in friendly way, and wipes her hands tissue paper.
The predominant primary theme present consistently throughout entire video friendliness. this aspect is clearly evident in the fact that character interacts with the man in genuinely friendly way.
E. The most noticeable primary theme present throughout the entire video is politeness. this aspect is clearly evident in the that character courteously shakes the man's hand.
Reference Answers:
GPT-4o:
GPT-4V: E.
Gemini-1.5-Flashn
Vixeo2Text Open-ended Tasks
Matane
930am Offccs
<vdee 1>
<video 2>
See the full video here.
See the full video here.
Turn 1: What is the of <video 1>? Describe the appearance of the boss. Turn 2: Is scene <video 2>e same that <video 1>? Are they gathering by organization or just at random?
Open-ended MMU datasets are not evaluated in this work. We only release the data instead. See Section A.3 for details.
22

---

Published as a conference paper at ICLR 2025
Tudio2Text Free-form Tasks
Listen the full audio here.
Query:
What you hear before the phone ringing?
Reference Answers: ("human whistling")
Qwen2-Audio-7B: dog barking
SALMONN-13B: dog barking.
Pengi: motorcycle
Audio2Text Open-ended Tasks
<audio 1>
<audio 2>
Listen the full audio here.
Listen to the full audio here.
Turn 1: <audio >>Is it possible to transcribe the words in <audio 2> subtitles? Why? Turn What the man <audio I>takknng about? Try recover whole content talking (including those not covered by the audio).
Open-ended MMU datasets Section A.3 for details.
evaluated in this work. We only release the data instead. See
Text21mage Tasks
1st Turn Query (generation): Design a vibrant cityscape banner featuring the iconic Sydney Opera House and Harbour Bridge under a dazzling sunset, with the City of Sydney2019, logo prominently displayed in the foreground, ensuring it catches the eye against the vivid backdrop. 2nd Turn Query (edit): Remove the City of Sydney2019 logo.
Flux
23

---

Published as a conference paper at ICLR 2025
of Sydney
1t turn generation.
2nd turn generation.
DALL-E 3 HD
INA
Generic city.
Generic city.
1st turn generation.
2nd turn generation.
PlayGround V2.5
1st turn generation.
2nd turn generation.
24

---

Published as a conference paper at ICLR 2025
Texe2Video Tasks
1st Turn Query (generation): Create dynamic video showcasing the energy and excitement live event. Focus vihrant crowd reactions, close-ups of performers speakers engaging passionately, and visually stunning moments that capture essence heing there live. Ensure include diverse camera angles to give a full experience of the event's s atmosphere. 2nd Turn Query (edit): Highlight the vibrant crowd reactions.
CogVideoX V5deoX-5B
3
2nd turn generation. See the full video here.
1tt turn generation. See the full video here.
HotShot-XT.
1st turn generation. See the full video here.
2nd turn generation. See the full video here.
VideoCrafter2
1st turn generation.
2nd turn generation. See the full video here.
See the full video here.
Text2Audio Tasks
1st Turn Query (generation): Craft brief audio snippet featuring clear, firm voice stating the necessity of paying fees exclusively within Russia, highlighting the rule's strictness and the policy's geographical exclusivity, without mentioning specific numbers or amounts. 2nd Turn Query (edit): Remove the geographical exclusivity detail.
AudioLDM 2
25

---

Published as a conference paper at ICLR 2025
1tt turn generation.
2nd generation. Listen to the full audio here.
Listen to the full audio here.
Make-An-Audio 2
1tt turn generation. Listen to the full audio here.
2nd turn generation. Listen to the full audio here.
Stable Audio
1st turn generation.
2nd turn generation.
Listen to the full audio here.
Listen to the full audio here.
Text2Action Tasks
Task Description: Find and send copy of the Donor Privacy Policy requesting donor. Allowed Actions: [Navigation], [PickupOOhect)) [PutOhject], [OpenOhject], [CloseObject], [ReadText], [PrintObject], [ScanObject], [SendEmail]] [TypeText], [AttachFile]. [OpenEmailClienj,,
26

---

Published as a conference paper at ICLR 2025
IScarchForObject), [IdentifyObject), [UseComputer]
Visible Objects: <'donor privacy policy document>, <computer>> <printer>, <scanner>, <email
client>, <file cabinet>, <desk>, <envelope>, <stamp>,, <mail drop box>, <recipient address label>, <office supplies>, <trash can>, <outbox tray>
Already Executed Steps: [Navigation] <office>>, [OpenObject] <file cabinet>>, [SearchForOhject]
<donor privacy policy document>, [PickupObject] <donor privacy policy document>, [CloseObject] <file cabinet>>
Reference Answer: [Navigation] <scanner>> [ScanObject] <donor privacy policy document>>
[Navigation] <computer>, [OpenEmailCCienn] <email client>>, [TypeTexx) <recipient address>, [AttachFile] <donor privacy policy document?>, [SendEmail]] <email client,>
GPT-4o: Based the given task and the steps that have already been executed, here is the plan for the remaining action-object sequences to complete the task: [Navigation] <scanner>> [PutObject] <donor privacy policy document> <asanl>>>><aanr>>>> [ScanObject] <donnr privacy policy document><4 >4. [Navigation] <computer>5.. [UseComputer] <computer>6. [OpenEmailClient) <emmail client>>7. [TypeeTxx]] Please find attached copy of the Donor Privacy Policy you srquested.<cmmatn client>>8. [AttachFile] <donor privacy policy document semn><<mmil client>99, [SendEmail] <email client>
Based on the task description and the already executed action-object se- quences, here's the plan to complete the task of sending copy of the Donor Privacy Policy requesting donor: [Navigation] (computee>>>UseCCopppue
Claude 3.5 Sonnet:
"computer>(ppenEmalll <email client> ObeanO <donor privacy policy [AttcchFie <donor privacy policy document>> [TypeTexx]] <email Dessage>>SSndEmane
<donor email>> This sequence of actions will accomplish the task efficiently: 1. Navigate to the computer to use for sending the email. 2. Use the computer access necessary software. 3. Open the email client prepare the email. 4. Scan the donor privacy policy document to create a digital copy. Attach the scanned file the email. 6. Type an appropriate message
email body. Send email with the attached donor privacy policy to the requesting donor. This plan assumes that the donor's 's email address is known or can be easily retrieved. If that information is not readily available, additional steps might be needed to locate the donor's 's contact information before sending the email.
Gemini-1.5-Pro: Your Planning: 1. [[[aavigation] <scanner>> *M[OpenOjject] <scanner>> 3. **[PutObject] <donor privacy policy document><<scannee
4. [SScanObbecc] <doonr pri- vacy policy document>** 5. **[ClosObjecc] <scanner> ** 6. **[avvigaioon] <computer> ** 7. Faapphniaall <email client>** 8. **[TypeTTex]] <email client> +* (Compose email to donor) [*[tttchFille] <scanned document>> <email>** ** **[SenEEmail] <email>**
Image2Action Tasks
Task Description: Find and purchase wrench.
Allowed Actions: [RecognizeOOjeet), [SelectObject), (DeselcctObject), [ConfimmSelection,
[Navigate], [ext] [ProcessList), [IdentifyObjecr), [Search]. [PickUp), [Transaction]
Visible Objects:
Already Executed Steps: [Search] <wrench>>, [[dentifyObject] wrench>>
Reference Answer: [Navigate] <wrench>>, [PickUp) <wrench>, [Navigate] <checkout counter>>,
[PickUp] <cash>> [Transaction] <eashh>>
GPT-4o: [PickUp] <wrench>>, [Navigate] <checkout counter>>, [Transaction] <wrench>
Claude 3.5 Sonnet: Based on the task description, allowed actions, visible objects, and already executed action-object sequences, here's the plan for the remaining action-object sequences to find
27

---

Published as a conference paper at ICLR 2025
and purchase wrench: [Navigate] <wrench (ecttion>/Rccognizzeee <wrench>/ReadTeen <price tag=/[oofiiSSelectinn Swrench>(PickUpp Swrench>/Navigeen <checkout counter>[Tnann [Transacton] <wrench>> Swvench>-Explanatioo of the plan: 1. Navigate to the wrench section, which we can see among the various tools. 3. Select the desired wrench. 4. Read the price tag to ensure it's within budget. 5, Confirm the selection of the wrench. 6. Pick up the wrench. 7. Navigate to the checkout counter (assumed be present in the store). 8, Complete the transaction to purchase the wrench. This sequence of actions should successfully complete the task finding and purchasing a wrench in the hardware store environment shown the image.
(rrench>(SeleetOhicee
image. Recognize the specific wrench we want to purchase
Gemini-1.5-Pro: <wrench>
Your Planning: [Navigate] <wrench>>,
[PickUpl <wrench>>
[Transaction]
E
PROMPTS
Text2Image Adaptation- Prompt Adaptation
System:
In this task, I want you to act instruction rewritter, and imagine that you rewritting
instructions for image generation.
The rewritten instruction will be used to instruct an image generation model that generates images. You will be provided with noisy user instruction that decides the topic content of the image generation task and you need rewrite the instruction make clearer and specific. Meanwhile, should practical for the image generation model to generate the corresponding image. The below examples are simplified, while your rewritten instructions could be either more detailed or more concise.
Main:
Example
Raw User Instruction: Eligible artwork. The artwork MUST wildlife. The rewritten instruction: Create an image of a serene forest at sunrise, with deer by stream, birds on branches, and rabbits in the underbrush, all bathed in the soft, golden light of the morning.
Example
Raw User Instruction: Next up, how to draw a Lamborghini the inner parts and details. The rewritten instruction: How about exhibiting Lamborghini with open hood and doors, detailing the engine, dashboard, and leather seats in brightly lit setting, emphasizing the car's luxury and mechanical complexity?
Given a noisy raw user instruction that implies the topic or content of the image generation task, brainstorm and rewrite the instruction to make it more clear and specific. You should be imaginary and the rewritten instruction should be concrete and specific the objects elements contained, even though they may
raw user instruction. The rewritten instruction not required to always use a fixed format starting from "Draw an image..." or "Create an image...". Instead, the format of the rewritten instruction should be more diversified and it should sound like a real instruction from user. However, it's important to always include the request for image creation in the rewritten instruction. Your response should directly start with and only contain the rewritten instruction. Do not generate anything else.
specified
Raw User Instruction:
The rewritten instruction:
Text2lmage Adaaptation-Rectiieattion Prompt- Adaptation
In this task. I want you to act as image edit task designer, and imagine that you are instructing image edit model to edit images.
System:
Main: You will provided with the image caption of an image to edit and you need to randomly pick some editing aspects to formulate an editing instruction. Besides that, you should provide the image
28

---

Published as a conference paper at ICLR 2025
caption for the edited image. Below are some examples.
Example
Image Caption: A serene forest sunrise, with by a stream, birds on branches, rabbits in the underbrush, all bathed in the soft, golden light of the morning.
The editing json dict: "editing instruction" "Channe the time to night.", "caption of the edited im- age":"A A serene forest night, with deer by stream, birds branches, and rabbits in the underbrush, bathed the soft, golden light of the morning.".
Example 2:
Image Caption: A Lamborghini with open hood and doors, detailing the engine, dashboard, and leather seats in brightly lit setting. emphasizing the car's luxury and mechanical complexity. editing json diet: "editing image":"A Ferrari with open hood and doors, detailing the engine, dashboard, and leather seats in a brightly lit setting, emphasizing the car's luxury and mechanical complexity.".
+ Ferrari.", "caption the edited
You should randomly pick thing/feature in the original image caption to edit. Note that the editing instruction must be practical, i.e., the things/features edit must contained in the provided image caption. The caption of the edited image should be exactly aligned with that of the original image, i.e., the only differences between the two captions should be those things/features that need to be edited. Your response should exactl follow the json dictionary format shown in the examples. Do not generate any thitinn else.
Image Caption:
The editing json dict:
Text2Image Mlaptation-Rectifcaatioo Prompt Rectification
System: In this task, I want you to act as an inspector of the image generation task, and imagine that you are inspecting the quality of the designed image generation tasks.
be provided with task dictionary which is designed instruct image generation models to generate and edit images, and you need to judge whether each component valid. The task contains turns, cach with a user prompt and its corresponding image caption, and there is no answer generated images in the task dictionary. The first-turn user prompt mainly designed to instruct the image generation model generate images, and the first-turn caption is the corresponding caption format of first-turn user prompt. which contains user prompt. The second-turn user prompt is a short prompt designed to instruct the image generation model to edit images generated in the first turn, and the second-turn caption is the caption designed for the edited image, which is almost the same the first-turn caption except for the items/features being edited. In the inspection json dictionary, answer 'Yes' a component correct; answer 'No' and specify the reason it's not. Below examples only illustrate the format and some simple situations, while it does not cover all conditions.
Main:
same features/elements as that the first-turn
Example
Task json dictionary: -promppp:"""" "Illustrate a close-up of car's front, focusing on detailing the hood's contours and shapes, alongside the intricate designs of the headlights. under clear, bright light to enhance the features.". "turnl "tural-caption":: "A close-up of car's front, focusing on detailing the hood's contours, alongside the intricate designs of the headlights, under clear, bright light enhance the features.", turm2-prompe
"Change the time dusk.", "tum22caattoo": "A close-up of a car's front, focusing on detailing the hood's contours and shapes, alongside the intricate designs of the headlights, under a clear, bright dusk light to enhance the features." Your inspection json dictionary: "turnl-promp" correct": "Yes", "turnI-ca -caption correct": "No. The hood's 's shapes are missed from the caption.", "'urn2-prompt correct": "No. The time is not contained the turn-1 caption.", "turn2 caption correct": "Yes"
Example 2:
Task json dictionary: "turn :urn--prompt": "Generate interactive digital photo album interface, with the 'Photos' tab highlighted. Upon clicking, display organized index page showing thumbnails of various albums categorized by events and dates.". "turn suattionppion":: "An interactive digital photo album interface. with the 'Photos' tab highlighted, displaying an organized index page showing thumbnails
29

---

Published as a conference paper at ICLR 2025
of various albums categorized by events and dates upon clicking.", "tum2--rompp".: "Change the highlighted tab from 'Photos' to "Vidcos".", "tum2-caption": "An interactive digital photo album interface, with the "Videos" tab highlighted, displaying fancy index pages which calegorized by events and dates." Your inspection json dictionary: "turnl-po -prompt correct": "Yes", -caption correct": "Yes", "tur22-romppt correct": "Yes", "lurn2-caption correct": "No. The "displaying fancy index pages which are categorized by events and dates." in turn2-caption is not aligned with tuml-caption"
Given a task dictionary, check whether each component correct logic and format. Especially check the helow four aspects: 1. Whether the turn tumm-prompt is valid image generation request; Whether the turn -caption is precisely the caption version of the tuml-prompt, without changing other components; 3. Whether the pumn2-proppt is valid image editing request, with all its editing Whether the tum22caatiion
components exactly being contained in the turtl-captio; caption for the edited image, which is supposed to be exactly aligned with turn aurn-ccation, ie.. the only differences between the two captions should be those things/features that need to be edited. Your response should exactly follow the json dictionary format shown in the examples. Your response should only contain the json dictionary and generate anything else.
a valid
Task json dictionary:
Your inspection json dictionary:
Text2Image Task Rewritter Prompt
System: this task, want you to act as caption extractor, and imagine that you are extracting
captions from user instructions image generation.
Main: You will be provided with user instruction that asks image generation model to generate images. Below are some examples.
Example 1:
User Instruction: Create image of serene forest at sunrise. with deer by a stream, birds on branches, and rabbits in the underbrush, all bathed in the soft, golden light of the morning. The Extracted Caption: A serene forest sunrise, with deer by a stream, birds on branches, and rabbits the underbrush, all bathed in the soft, golden light of the morning.
Example 2:
User Instruction: How about exhibiting Lamborghini with open haod and doors, detailing the engine, dashboard, and leather seats brightly setting, emphasizing the car's luxury and mechanical complexity? The Extracted Caption: A Lamborghini with open hood and doors, detailing the engine, dashboard, and leather seats in a brightly lit setting, emphasizing the car's luxury and mechanical complexity.
The extracted caption must contain exactly same content as the user instruction, i.c., not add or reduce any image content/feature description that is included in the original user instruction. You only need to change the format from the instruction format to the caption format. Your response should directly start with and only contain the extracted caption. Do not generate anything else.
User Instruction:
The Extracted Caption:
Text2Video Adaptation- Prompt Adaptation
In this task, I want you to instruction rewritter, and imagine that you rewritting instructions for short video generation ovision-only, without audio).
System:
The rewritten instruction will be used to instruct video generation model that generates videos. You will be provided with noisy raw user instruction that decides the topic content of the video generation task and you need rewrite the instruction make clearer and specific. Meanwhile, should he practical The below examples are simplified, while your rewritten instructions could be either more detailed or
Main:
video generation model to generate corresponding video.
30

---

Published as a conference paper at ICLR 2025
more concise.
Example
Raw User Instruction: How to Use Your Smartphone to Project Holograms? The rewritten instruction: Create a concise video illustrating the construction of hologram projector from a smartphone. Highlight key steps: selecting materials (CD case, tape, scissors), craftig trapezoid from the CD case, forming pyramid structure, and positioning it atop the smartphone project hologram. Focus on clear, visual steps, ending with hologram projection.
Example 2:
Raw User Instruction: Produce action film? The rewritten instruction: Give thrilling action sequence showcasing chase in urban setting. Highlight intense character expressions, swift movements through crowds. jumps over obstacles. and clever escape. Use narrow alleyways bustling streets, and iconic landmarks to enrich the visual narrative.
instruction that implics the topic content of the video generation task, brainstorm and rewrite the instruction to make it more clear specific. You should imaginary and the rewritten instruction should concrete and specific the objects or elements contained, even though they may not be specified in the raw user instruction. The rewritten instruction is not required to always use fixed format starting from "Create an video..." or "Generate an video...". Instead, the format of the rewritten instruction should be more diversified and it should sound like real instruction rewritten
Given noisy
from user. However, it's important to always include the request for video creation instruction. The rewritten instruction should not request for generating videos with too long content, e.g., hour-level videos. Instead, they should be second-level, while you should not directly mention the target length of the video in the rewritten instruction. The videos to generate vision-only and do not include audios. Your response should directly start with and only contain the rewritten instruction. Do not generate anything else.
Raw User Instruction:
The rewritten instruction:
Text2Video Adaptation-Rectiffaatao Prompt Adaptation
System: In this task, I want you to act as an video edit task designer, and imagine that you are instructing an video edit model to edit videos (vision-only, without audio).
Main: You will be provided with the video caption of an video edit and you need randomly pick some editing aspects formulate an editing instruction. Besides that, you should provide video caption for the edited video. Below are some examples.
Example
Video Caption: concise video illustrating the construction hologram projector from smart- phone. Highlight key steps: selecting materials (CD case, tape, pen, scissors), crafting a trapezoid from the CD case, forming a pyramid structure, and positioning it atop the smartphone to project a hologram. Focus clear, visual steps, ending with the hologram projection. The editing json diet: "editing "iaruction"""Remoons illustrating the construction a trapezoid from the CD case, forming a pyramid structure, and positioning it atop the smartphone to project hologram. Focus on clear, visual steps. ending with the hologram projection.".
the material selection". "caption of the edited video":" concise video hologram projector from smartphone. Highlight key steps: crafting
Example
Video Caption:A thrilling action sequence showcasing a chase seene in an urban setting. Highlight intense character expressions, swift movements through crowds, jumps over obstacles, and a clever escape. Use narrow alleyways, bustling strects, and iconic landmarks to enrich the visual narrative. The editing json dict: "editing instruction" "arructio""""mmphaatt ""Emhhasize edited video"."A thrilling action sequence showcasing chase scene in urban setting. Highlight intense character expressions, swift movements through crowds, and clever escape. Use narrow alleyways, bustling streets, and iconic landmarks to enrich the visual narrative. Emphasize the jumps over obstacles.".
jumps over obstacles". "caption of the
31

---

Published as a conference paper at ICLR 2025
You should randomly pick a thing/feature in the original vidco caption to edit. Note that the editing instruction must be practical, i.c., the things/features to edit must be contained in the provided video caption. The caption of the edited video should exactly aligned with that of the original video, i.e. the only differences between the two captions should be those things/features that need to edited. The videos to edit are vision-only and do not include audios. Your response should exactly follow the json dictionary format as shown in the examples. Do not generate anything else.
Video Caption:
The editing json dict:
Text2Video Adaptation- Rectiiicaiin Prompt- Rectification
System: In this task, I want you to act as inspector of the short video generation task (vision-only, without audio), and imagine that you inspecting the quality of the designed vidco generation tasks, Main: You will be provided with task dictionary which designed to instruct vidco generation models to generate and edit vidcos, and you need to judge whether each component valid. The task contains turns, each with a user prompt and its corresponding video caption, and there is no answer generated videos in the task dictionary. The first-turn user prompt mainly designed to instruct the video generation model generate videos, and the first-turn caption the corresponding caption format of the first-turn user prompt, which contains user prompt. The second-turn user prompt a short prompt designed to instruct the video generation model to edit videos generated in the first turn, and the second turn caption is the caption designed for the edited video, which is almost the same as the first-turn caption except for the items/features heing edited. In the inspection json dictionary, specify the reason s's not, Below examples only illustrate the format and some simple situations, while it does not cover all conditions.
same features/elements as that the first-turn
"Yes' component correct; answer 'No' and
Example
Task json dictionary: "turn -prommpp"pp" "Generate a compilation vidco of the most captivating moments from the linked content. Focus on visually striking scenes, dynamic actions, and key highlights that can stand alone without audio explanation. Aim for scamless transition between segments to maintain viewer engagement.", "turn "tur1Iccaption" -caption"" "A compilation video of the most captivating moments from the linked content. Focus on visually striking scenes and key highlights that stand alone without audio explanation. Aim for seamless transition between segments to maintain viewer engagement.", fuum2-promptt "Highlight the facial expression more prominently". "turn2-caption"" "A compilation video of the most captivating moments from the linked content. Focus facial expression prominently, dynamic actions, and key highlights that stand alone without audio explanation. Aim for seamless transition between segments maintain viewer engagement." Your inspection json dictionary: "turn -prompt correct": "Yes", "turn -caption correct": "No, the 'dynamic actions' is missing compared to turn turnI-prun -prompt.,. "tum22-rompt correct": "No. The facial expression does not exist in turn I l-caption, thus not being eligible for edit.", "turn2-caption correct": "No. The facial expression does not exist in turnl ccaptcnp,ion, thus being eligible for edit."
Example 2:
Task json dictionary: "turnl "turnI-prome" -prompt": "Craft 15-min video demonstrating the enhancement of website using Drupal 8. Focus visual guides for integrating categories, enabling and managing comments, applying custom styles, and adding unique features the site. Display step-by-step actions taken within the Drupal interface achieve each enhancement, ensuring clarity precision each visual step demonstrated.", "turnI-cap -caption"" "A 15-min video demonstrating the enhancement of a website using Drupal 8. Focus visual guides for integrating categories, enabling and managing comments, applying custom styles, and adding unique features the site. Display step-by-step actions taken within the Drupal interface to achieve each enhancement, ensuring clarity and precision each visual step demonstrated.", "tum2-caption": "A 15-min vidco demonstrating the enhancement of a website using Drupal 8. Focus visual guides for integrating categories, enabling and managing comments, applying custom styles with increased clarity, and adding unique features the site. Display step-by-step actions taken within the Drupal 8 interface achieve each enhancement, ensuring clarity and precision each visual step demonstrated." Your inspection json dictionary: "turn (urmmppoompt correct": "No. The expected video is too long- It should be less than 120 seconds instead.", "turn "turnI-caption conrect": "Yes", "tum22-rroppt correct": "Yes", "turn2 caption correct": "Yes"
"Increase clarity on applying custom styles",
32

---

Published as a conference paper at ICLR 2025
Given a task dictionary, check whether cach component is correct in logic and format. Especially check the below four aspects: 1. Whether the turn -urmmpt is valid video generation request that instructs model to generate second-level videos less than 120 seconds (the explicit length restriction is not required to included generate); 2. Whether the turn acrnttcttiion is precisely the caption version of the turn turnI-promm -prompt, without changing other components: 3. Whether the
turn1prompp; only inspect the estimated length of content
is valid video editing request, with all its editing components exactly being contained the turl-ccaptio;; 4. Whether the tum2-caption is valid caption for edited video, which is supposed to exactly aligned with turn numl-catiion ie.. the only differences between the two captions should be those things/featurre
need to edited. Besides, the vidcos to generate are vision-only and do not include audios. Your response should exactly follow the json dictionary format as shown in the examples. Your response should only contain the json dictionary and do not generate anything else.
Task json dictionary:
Your inspection json dictionary:
Video Task Rewritter Prompt
System: In this task, want you to act caption extractor, and imagine that you extracting captions from user instructions video generation.
Main: You will be provided with user instruction that asks an video generation model to generate videos. Below some examples.
Example
User Instruction: Create a concise video illustrating the construction of hologram projector from smartphone. Highlight key steps: selecting materials (CD case, tape, pen, scissors), crafting a trapezoid from the CD case, forming pyramid structure, and positioning atop the smartphone project hologram. Focus on clear. visual steps, ending with the hologram projection. The Extracted Caption: A concise video illustrating the construction of a hologram projector from a smartphone. Highlight key steps: selecting materials (CD case, tape, pen, scissors), crafting trapezoid from the CD case, forming pyramid structure, and positioning it atop the smartphone project hologram. Focus on clear, visual steps, ending with hologram projection.
Example 2:
User Instruction: Give thrilling action sequence showcasing chase scene in urban setting. Highlight intense character expressions, swift movements through crowds, jumps and clever escape. visual narrative. The Extracted Caption: A thrilling action sequence showcasing a chase scene in an urban setting. Highlight intense character expressions, swift movements through crowds, jumps over obstacles, and clever escape. Use narrow alleyways, hustling streets, and iconic landmarks to enrich the visual narrative.
obstacles,
narrow alleyways, bustling streets, and iconic landmarks to enrich the
The extracted caption must contain exactly the same content as that of the user instruction, i.e., do not add reduce any video content/feature description that is included in the original user instruction. You only need to change the format from the instruction format the caption format. Your response should directly start with and only contain the extracted caption. Do not generate anything else.
User Instruction:
The Extracted Caption:
Text2Audio Adaptation-RRectiiaatioo Prompt Adaptation 1
System:
In this task, want you to act an instruction rewritter, imagine that you are rewritting instructions for short audio generation.
The rewritten instruction will be used to instruct audio generation model that generates audios. You will be provided with noisy raw instruction that decides the topic content of the audio generation task and you need to rewrite the raw instruction make clearer more specific. Meanwhile, should be practical for the audio generation model to generate the corresponding audio.
Main:
33

---

Published as a conference paper at ICLR 2025
The below examples are simplified, while your rewritten instructions could be cither more detailed or more concise.
Example
Raw User Instruction: just read how you will dress. The rewritten instruction: Create an audio clip of calm and composed voice describing outfit choice for formal event in detail, including the color and material of the clothing, any accessories being worn, and the reasons behind these choices.
Example
Raw User Instruction: Scarch for similar music: funky, upheat, ambient, excitement, business, pop, atmospheres, Bass-Guitar The rewritten instruction: Give audio track that blends elements of funk and pop with an upbeat and exciting rhythm, featuring prominent bass guitar lines. The composition should include ambient textures to create dynamic atmosphere suitable for business environment. The track should evoke sense of motivation and energy, making use of synthesizers and drum beats to enhance its lively mood.
Given a noisy raw user instruction that implics topic content of the audio generation task, brainstorm and rewrite the instruction to make it more clear specific. You should imaginary and the rewritten instruction should be concrete and specific in the objects or elements contained, even though they may not be specified in the raw user instruction. The rewritten instruction is not required always use fixed format starting from "Create audio..." "Generate format of rewritten instruction should more diversified and it should sound a real instruction from a user. However, it's important to always include the request for audio creation in the rewritten instruction. The rewritten instruction should not request for generating audios with too long content, e.g., hour-level audios. Instead, they should be second-level, while you should not directly mention the target length the audio the rewritten instruction. The rewritten audio generation instruction can either human speech generation or general audio generation. When lopic is possible, the rewritten instruction should b a general audio generation instruction without human speech or human voice generation. Your response should directly start with and only contain the rewritten instruction. Do not generate anything else.
audio..." Instead, the
Raw User Instruction:
The rewritten instruction:
Text2Audio Cilapttation-Reitticatl Prompt Adaptation 2
System: In this task, I want to act as an audio edit task designer, and imagine that you are instructing audio edit model to edit audios.
Main: You will be provided with audio caption of an audio edit and you need to randomly pick some editing aspects to formulate an editing instruction. Besides that, you should provide the audio caption for the edited audio. Below are some examples.
Example
Audio Caption: A calm and composed voice describing an outfit choice for a formal event detail, including the color and material of the clothing, any accessories being worn, and the reasons behind these choices. The editing json dict: "editing instruction" "Remmoee the description of the color and material of the clothing". "caption of the edited audio":" A calm and composed voice describing outfit choice for a formal event detail, including any accessories being worn, and the reasons behind these choices.".
Example 2:
Audio Caption: An audio track that blends elements of funk pop with an upbeat and exciting rhythm, featuring prominent bass guitar lines. The composition should include ambient textures create dynamic atmosphere suitable for a business enviromment. The track should evoke sense of motivation and energy, making of synthesizers and drum beats enhance its lively mood. The editing json dict: "editing instruction" "Remmovee the ambient textures enhance the synthesizers". "caption of the edited audio":" audio track that blends elements of funk and pop with upbeat and exciting rhythm, featuring prominent bass guitar lines. The composition should be suitable for business environment. The track should evoke sense of motivation and energy. making use of enhanced synthesizers and drum beats enhance its lively mood.".
34

---

Published as a conference paper at ICLR 2025
You should randomly pick a thing/feature in the original audio caption to edit. Note that the editing instruction must be practical, i.c., the things/features to edit must be contained in the provided audio caption. The caption of the edited audio should exactly aligned with that of the original audio, i.e. the only differences between the two captions should be those things/features Your response should exactly follow the json dictionary format as shown in the examples. Do not generate anything else.
need to edited.
Audio Caption:
The editing json dict:
Text2Audio Adaptation- Prompt Rectification
System: In this task, I want you
inspectir of the short audio generation task, and imagine that you are inspecting the quality the designed audio generation tasks.
Main: You will be provided with task dictionary which is designed to instruct audio generation models generate and edit audios, and you need judge whether each component is valid. The task contains turns, each with a user prompt and corresponding audio caption, there is no answer or generated audios the audio generation model to generate audios, and the first-turn caption the corresponding caption format of the first-turn prompt, which contains the same features/elements as that of the first-tun user prompt. The second-turn user prompt short prompt designed instruct the audio generation model edit audios generated
task dictionary. The first-turn user prompt mainly designed to instruct
first turn, and the second-ture caption is caption designed for the edited audio, which is almost the same as the first-turn caption except for the items/features being edited. In the inspection json dictionary, answer "Yes" if a component is correct; answer "No' and specify the reason if it's not. Below examples only illustrate the format and some simple situations, while does not cover all conditions.
Example
Task json dictionary: "turn1-pre -prompt": "Craft audio message that succinctly conveys the essential details might include in SMS, ensuring the tone is informative yet brief, suitable for mobile notification sounds.", "turnl carnl-capioon" "An audio message that succinetly conveys details one might include an SMS, ensuring the tone is informative, suitable for mobile notification sounds.", "tum2-prompp" "Make the background music louder.", "turn2-caption": "An audio message that succinetly conveys the essential details one might include in SMS, ensuring the tone is informative, suitable for mobile notification sounds. Make the background music louder." Your inspection json dictionary: "turnl "turnl-prompt correct": "Yes", "turnl "tuml-caption correct": "No, the "brief" missing from the turn -prompt.. "turn2-prompt correct": "No. 'background music' is not contained in the turn-l caption.", ", "lurn2-caption correct": "No. 'background music' not contained in the turn-1 caption, this change is invalid."
essential
Example 2:
Task json dictionary: "turnl -prompt": "Produce a 15-min audio snippet featuring a stern yet professional voice explaining the necessity of paying fees exclusively within Russia, emphasizing the nule's importance and potential consequences of non-compliance." ftuml-caption" "A 15-min audio snippet featuring a stern yet professional voice explaining the necessity paying fees exclusively within Russia, emphasizing the rule's importance and potential consequences of non-compliance." "turn2-prowd prompt": "Remove the emphasis on the rule's importance and potential consequences of non-compliance", "tum2-caption": "A 15-min audio snippet fcaturing stemn yet professional voice explaining the necessity of paying fees exclusively within Russia." " Your inspection json dictionary: "turn I- prompt correct": "No. The expected audio too long. It should be less than 120 seconds instead.", "turnI-caption correct": "Yes". "turn2-prompt correct": "Yes". "turn2-caption correct": "Yes"
Given a task dictionary, check whether each component correct logic and format. Especially check the below four aspects: 1. Whether the turnl turnl-prompt a valid audio generation request that instructs a model to generate second-level audios less than 120 seconds (the length restriction is not required to be included in the turn turn1-prompt, only inspect the estimated length of the content generate); 2. Whether the turn l-caption is precisely the caption version of the turn I -prmmpt without changing other components; 3. Whether the turn2-prompt is a valid audio editing request, with its editing components exactly being contained in the turn luml-caption; Whether the turn2-caption is
35

---

Published as a conference paper at ICLR 2025
valid caption for the edited audio, which is supposed to be exactly aligned with turn -caption, i.c., the only differences between the captions should be those things/featuree that need to be edited. Your response should exactly follow the json dictionary format shown in the examples. Your response should only contain json dictionary and not generate anything else.
Task json dictionary:
Your inspection json dictionary:
Text2Audio Task Rewritter Prompt
System: this task, want you to act as a caption extractor, and imagine that you are extracting captions from user instructions for audio generation.
Main: You will be provided with instruction that asks audio generation model generate audios. Below are some examples.
Example I:
User Instruction: Create audio clip of a calm and composed voice describing outfit choice for formal event in detail, including the color and material of the clothing, any accessories being worm, and the reasons behind these choices. The Extracted Caption: an outfit choice for a formal event in detail, including the color and material of the clothing, any accessories being worn, and the reasons behind these choices.
and composed voice describing
Example
User Instruction: Give me an audio track blends elements of funk and pop with an upbeat exciting rhythm, featuring prominent bass guitar lines. The composition should include ambient textures to create dynamic atmosphere suitable for business environment. The track should evoke sense of motivation and energy, making of synthesizers and drum heats enhance its lively mood. The Extracted Caption: audio track blends elements funk and pop with upbeat and exciting rhythm. featuring prominent bass guitar lines. The composition should include ambient textures to create dynamic atmosphere suitable for a business environment. The track should evoke sense of motivation and energy, making of synthesizers and drum beats to enhance its lively mood.
The extracted caption must contain exactly the same content as that the user instruction, i.c., do not add or reduce any audio content/feature description that is included in the original user instruction. You only need to change the format from the instruction format to the caption format. Your response should directly start with and only contain the extracted caption. Do not generate anything else.
User Instruction:
The Extracted Caption:
Text2Action Adaptation-Rectife
Prompt Adaptation
System:
In this task, want you task designer, and imagine you designing real-world planning tasks that are not too complicated break down, be executed by embodied agents or robots in text.
You will only be provided with instruction, which implies the topic of the task content, and you need make the task content complete by designing the task description, the allowed actions, the visible objects, and the already executed actions. The designed task content will be used to test the planning abilities of embodied agents or robots.
Main:
The below examples simplified, while your completed task description could be either detailed concise, depending on context.
Example 1:
<Instruction:>: How about egg feast? The task json dictionary: "task description":"PPe heated egg in the sink.", "allowed "(OpenOeii)),) [CloseOhject], [PickupObjeet], [PutOhject], [ToggleOOjecrOnn, (ToggleObjectOff], [SliceOhjeet), [Navigation]" "iriooonn""vvviiilliee vvisible "miiiiee"""""""""""""""""""""" <sink>, <toaster>> <coffee maker'>, <fridge>>, <blender>, <potato>, <bows>, <egg>>.
<microwave>,
36

---

Published as a conference paper at ICLR 2025
<fridge>> [PickupObjeer] <.gg>, [CloseObject] <fridge>, [Navigation] microwave>, [PutObject] <egg><<microwave>>>>>
<garbagecan>>" "aalccdy exccuted "ctions":"[NNvieeee" <fridge>,
[OpenOhject]
Example 2:
<Instruction.>: rush for the ticket! The task json dictionary: "task description": "Purchase tickets in that hall." "allowed actions": "[Navigation), [InteractWtthhbeec]], [PickupObject), [PutOb- ject]. [UseObject). [Speak]. [Listen]. [PaymentTransaction]] [IdentifOObeaal", "visible objects": "<ticket booth>, <information desk>,, <scats>>, <hall entrance?>, <hall exit>, <ticket machine>, <posters>, <map?>, <cash>, <credit card,>, <other visitors?> <staff members?", "already executed actions": "[Navigation] <hall entrance,>, [IdentifyOhjeet] <ticket booth?>, [Speak] <sttff members>"
Example 3:
<Instruction>: Click here for the L2TP setup The task json dictionary: "task description": "Set up L2TP VPN connection in the Network Preferences.", "allowed actions": "[Navigation], [Click), [InpurText], [TTogleSwitch), [ConfirmAction), [ReadText), [Scroll), [OpenApplication), [CloseAppli- [EnterCredentials)] [SaveSettingsl". "visible objects": "<computer>>, <network preferences menu>> <VPN option>> <I.2TP option>>, <server address field>>, <account name field>> <password field>, <shared secret field>, <save button?>, <cancel button.> <status indicators>>, <dropdown menus>> <text fields>, <checkboxes>", "already executed actions": "[Navigation] <computer>, [OpenMenu] <network preferences menu>, [Click] <VPN option>>, [Click] <L2TP option>. [InputText] <server address field>, [InputText] <account name field?"
cation], [OpenMenu]. CChooseNetworlyypee
Considering the above examples, given <Instruction>>hat implics the topic of the task content, brainstorm complete the task design. each planning task you should randomly pick setting that has similar topic to the given <Instruction>> 'task description' should be natural and sound like user instruction and make sure the task is not too complexed to break down and is practical execute for embodicd agents robots. The provided 'allowed actions" and "visible objects' should be highly practical and they should be more than those required to plan this task, so that it more challenging for the embodied agents to plan. Each element in the 'allowed actions' should he placed in a pair of square brackets '[]', and each element in the 'visible objects' should he placed in a pair of angle brackets <<>". The same applies the elements the 'already executed actions' which elements from 'allowed actions" 'visible objects', sometimes an element in 'already executed actions' may be combination of an element from 'allowed actions' and multiple elements from 'visible objects', e.g. [[uu[bbect]] <egg>> <microwave>'" The 'already executed actions' contains action-object sequenees that assumed to have been completed by the embodied agents for the designed task and you should control an appropriate number of elements in the 'already executed actions' as a hint for the embodied agents to plan the remaining actions for the designed task. Note that the 'already exccuted actions" should not contain the plans to be completed by the embodied agents. Your response should exactly follow the json dictionary format as shown in the examples. Your response should only contain the json dictionary and do not generate anything else.
directly generate the specific information. The
the designed planning task,
combinations of
<IInstrccton>>
The task json dictionary:
Text2Action Adaptation-Reetifcatiioo Rectification Prompt Rectification
System: want you to act as a task verifier and rewriter, and imagine you are verifying real-world action planning tasks to be executed by embodied agents or robots in text.
Main: You will be provided with task json dictionary. which formulates real-world action planning task. json dictionary contains four keys: "task description', 'allowed actions", 'visible objects' and 'already executed actions'. The 'task description' a user instruction that instructs the embodied agents or robots to complete the task. The 'allowed actions' is a list of actions that are allowed to be used by the embodied agents rohots to complete the task. The 'visible objects' is list of objects that visible the embodied agents or robots when they completing the task. The 'already executed actions' a list action-object sequences are assumed to have been completed by the embodied agents or robots at the time of task designing. You need to verify whether the task is a valid
37

---

Published as a conference paper at ICLR 2025
action planning task in terms of the following requirements: Requirement I: The task format should be correct. should contain the above-mentioned four keys and their corresponding values; each element in the "allowed actions' should be placed in pair of square brackets '[]'.
each element the "visible objects' should be placed pair of angle brackets <>"; each element of the 'already executed actions' is exactly combination of one action element from the 'allowed actions' and one more object elements from the "visible objects'. .
Requirement The task description should be natural user instruction. It should not be too complicated to break down should be practical to execute for embodied agents or robots.
Requirement 3: The provided 'allowed actions' and 'visible objects' should be highly practical for the designed planning task, i.e., the actions in the 'allowed actions' and the objects in the 'visible objects' should be commonsensical in the designed task and its environment.
Requirement 4: The provided 'allowed actions' and "visible objects' should be redundant, ie., they should be more than the real requirement Requirement 5: The "alredyy executed actions' should have an appropriate number of steps and should be correct in logic as part of the actions of the designed task.
designed task.
Requirement 6: The 'already executed actions' should not contain the plans be completed by the
embodied agents, i.c., the task completely solved given the "already executed actions'.
Below are two simplified examples.
Example 1:
Task json dictionary: "task description"""Put heated egg in sink.". "allowed ac- "ions::((peenO)ject]] [CloseObjectt, [PickupObject), [PutObject), [ToggleObjectOnn. [ToggleOb- <<mmcrowave>,
[Navigation]"" "viisblll <<hirrss""<mmrroavvve>> <sink>>, <toaster>, <coffee maker>>, <fridge>, <blenner>>, <potato>>, <hows>, <egg>, <garbagecan>>""" "aaleedy executed aetiooss:""Navviaatioon <fridge>, [0penOhject] <fridge>, [PickupObject] <egg>, [CloscObject] <fridge>, [Navigation] <microoave>> [PutObject] <egg> <eg>><mmeowwav>>>>>>> Your verification (and rewriting): "Requirement I": "Yes", "Requirement 2": "Yes", "Requirement 3"; "Yes", "Requirement 4": "Yes". "Requirement 5": "Yes", "Requirement 6": "Yes", "need rewrite?": "No", "rewritten version":
Example 2:
Task json dictionary: "task description": "Purchase tickets that hall.", "allowed actions": "Naviga- tion, InteractWithOhjee PickupObject, PutObject, UseObject, Speak, Listen, PrymenTTransaction, IdentifyOhjeet, CloseApplication. OpenMenu, ChooseNetworKTypee" "visible objects": "<fridge>, <blender>, <potato>, <ticket booth>, <information desk>, <seats>, <hall entrance>, <hall exit>> <ticket machine>, <credit card>,, <other visitors>>, <staff members>", "already executed actions": "Navigation <hall entrance>>, Identify Object <ticket booth>>, Speak <staff members>>" Your verification (and rewriting): "Requirement 1"; "Yes", "Requirement 2": "No", "Requirement 3": "No", "Requirement 4": "Yes", "Requirement 5": "Yes", "Requirement 6": "Yes", "need rewrite?": "Yes", "rewritten version": ""task description": "Purchase tickets in that hall.", "allowed actions": "[Navigation], [IntcractWithObere [PaymeniTransaationn] [ddentifybjeecl",, "visible objects": "<ticket booth>, <information desk>>, <seats>> <hall entrance>> <hall exit>, <ticket machine>, <posters>, <map>> <cash>> <eredit card>, <other visitors>, <ttaff members>". "already executed actions": "[Navigation] <hall entrance>, [IdentifyObeect] <ticket booth>, [Speak] <staff members>"
[PickupObjectl, [PutOhject), [UseObject), [Spcak), [Listen],
In the above two examples, the first one qualified task, and the second one isn't because does not meet Requirements 2 and Any unqualified task requires it qualified, the 'rewritten version' should be empty. Considering the above context and examples, verify (and rewrite) the below task json dictionary. Please exactly follow the json format shown in the above examples. Your response should only contain the verification json dictionary, do not generate anything else.
modified rewritten, and if you think
Task json dictionary:
Your verification (and rewriting):
Text2Action
Prompt Reference Answer Annotation
System: In this task, want you to real-world agent, and you will plan action-object sequences for the real-world tasks.
38

---

Published as a conference paper at ICLR 2025
You will be provided with a task json dictionary, which formulates a real-world action planning task. The json dictionary contains four keys: "task description', 'allowed actions', 'visible objects', and "already executed action-object sequences'. The 'task description' is instruction that instructs you to complete the task. The 'allowed actions' a list of actions are allowed to be used by you to complete the task. The 'visible objects' is list of objects that are visible to you when you are completing the task. The "already executed action-object sequences' is a list of action-object sequences that assumed to have been completed you. You need plan the remaining action-object sequences to complete the task.
Main:
Below are two simplified examples.
Example 1:
Task json dictionary: "task description"."Put a heated egg in sink.". "allowed ac- iions:"[(penOOjject,, [CloseObject]. (PickupObject), [PutObject], [ToggleObjectOn] [Tog- gleObjectOfff, ISliceObject), [Navigation|]."visii objects";" ""birss""<mmcrooavve>>>" <sink>>, <toaster>, <coffec maker>, <fridge>, <blender>>, <potato>>, <bows>, <egg>, <garbagccan>"""alreaa executed action-object "oquuenes"""""aviiaaaiii
<fridge>, [OpenObject] <fridge>, [PickupObject] <egg>, [CloseObject] <fridge>>, [Navigation] <microwave>> [PutObject] <egg> <microwave>> Your planning: [ToggleOhjectOn] < <microwave>, [ToggleObjectOff] <microwave>, [PickupOhject]
(iggeoooaaee.TTTggglhhhceocooo <egg>>, [Navigation] <sink>> [PutObject] <egg><<ink
Example
Task json dictionary: "task description": "Purchase tickets that hall.". "allowed actions": "[Navigation], [LneeracWithhbject]]. [PickupObject], [PutObject]. [UseObject]. [Speak]. [Listen]. [Payment(ransactioon) [ddentifyObjeet|", "visible objects": "<ticket booth>,, <information desk>>, <scats>> <hall entrance>>, <hall exit>,, <ticket machine>>, <posters>, <mma>>, <cash>>, <credit card>, <other visitors?,, <staff members>, <tickets>", "already executed action-ohject sequences": "[Navigation] <hall entrance>,, [[dentifyObjec]] <ticket booth>, [Speak] <staff members>"" Your planning: [Listen] <staff members>, [Navigation] <ticket booth>, [IdentifYbbecc]] <ticket machine>, [[nterac[Wtthhbjee]]] <ticket machine>, [PickupObject] <eash>> [PayaennTransaction]] <ticket machine>, [PickupObject] <tickets>.
Considering the above examples, given task json dictionary, plan the remaining action-objed object sequences to complete the task. Each element in the 'allowed actions' are placed in pair of square brackets '[I', and each element in the 'visible objects' placed in pair of angle brackets '<>", ; the same applies to the elements in your planning, where each element is combination of an element from 'allowed actions' and an element from 'visible objects'. Sometimes an element in your planning may be a combination of special element from 'allowed actions' and multiple elements from 'visible objects', e.g., '[PutObject] <egg>><< microwave>>' where the '[PutObject]' action special action that should be followed with two objects. Your planning should be efficient complete the task description, do not plan irrelevant steps or miss crucial steps. Your response should directly start with and only contain the planned action-object sequences, do not generate anything else.
Task json dictionary:
Your planning:
Image2Action cnppitationnRectiiictttiii Prompt- A
System: In this task, want you to act as practical visual description designer.
Main: You will only be provided with a user prompt. The user prompt refers to an external image. Specifically, the user prompt is asking an embodied agent perform a task based on the content of the image. and the image content is the whole picture of what the embodied agent sees. The content by the embodied agent is not provided and sometimes the user prompt may be unclear or noisy. but your job is to design the description of the content seen by the embodied agent that is specific. You need to imagine the seen content and write the concise description for it based on the user prompt. The below examples are simplified.
Example
User Prompt: How do we dig in this landscape? The description of the content seen by the embodied agent: A rugged, mountainous landscape under clear blue sky. A range of tall moun- tains with jagged peaks extends into the distance, suggesting challenging environment for excavation.
39

---

Published as a conference paper at ICLR 2025
Example 2:
User Prompt: Click the pic below to get full-size. The description of the content seen by the embodied agent: A computer screen displaying image thumbnail within digital photo gallery. The thumbnail shows picturesque landscape, possibly hinting
larger, more detailed image. Surrounding the thumbnail are user interface elements like a 'Click to Enlarge' button, and other thumbnails showcasing different images, indicative of typical photo viewing or editing software environment.
Pick a practical and common setting for possible seen contents based on User Prompt directly descrihe the content. You should be imaginary and written description should concrete and specific in the objects or elements contained, even though they may not he specified in the user prompt. Try to be concise, i.e., do not descrihe too many unrelated elements, descrihe those important instead. And make the described seen content practical for the embodied agent perform the task specified the user prompt. Your response should directly start with only contain the content description. Do not generate anything else.
User Prompt:
The description of the content seen by the embodied agent:
Image2Action Adaptation- Prompt Reference Answer Annotation
System: You will be provided with task json dictionary and its corresponding task image, which formulates real-world action planning task with image input.
Main: The json dictionary contains three keys: 'task description', "allowed actions', and 'already executed action-object sequences'. The "task description' is user instruction that instructs you to complete the task. The 'allowed actions' is list of actions that are allowed to be used by you to complete the task. The 'already executed action-object sequences' a list of action-object sequenees that are assumed have been completed you. The corresponding task image contains the visible hidden objects for you complete the task and indicates the task environment. need to plan the remaining action-object sequences to complete the task.
Below are two simplified examples.
Example
Task json dictionary: "task dessrription:""Ger the egg from the fridge, and put the heated egg in the sink.", "allowed actionss:""0peemhheeccll [CloseObject), [PickupObjectl, [PutObject], [TTogleOOb- jectOn), (ToggleOhjcctOfff), ISliccOhjectt, INavigation|! aalleeady executed steps": "NaviittnNi <fridge>, [bpenObject] <fridge>, [PickupObject] <egg>>, [CloseObject] <fridge>, [Navigation] <microwave>>, [PutObject] "cogg><<icoowvee""" <taas image>:
Your planning: [ToggleObjectOn] <microwave>, [ToggleObjectO0f]] <microwave>> [Navigation]
<sink>, [PickupObjeet] <egg>, [PutObject] <egg><<ink>>
Example 2:
Task json dictionary: "task description": "Purchase tickets from the counter.", "allowed actions": "[Navigation], [IneraerWithhhjeet)), [PickupObject), [PutOhject], [UseOhject]. [Speak], [Listen], [PaymenfTransaationn] [IdentifyOhjee]]", "already executed steps": "[[dentifyObject] <counter>>, [Navigation] <counter>, [Speak] <staff members>" <task image>: Your planning: [Listen] <ttaff members>, [PickupObject] <cash>, [Paymen(Transaction]] <sstaff' members>> [PickupObject] <tickets>>
Considering the above examples, given task json dictionary and its corresponding task image, plan the remaining action-object sequences to complete the task. Each action the 'allowed actions' are placed in a pair of square brackets '[I', and each object is placed in pair of angle brackets <<>; the same applies to the elements in your planning, where each element is the combination of an action from 'allowed actions' and visible hidden object in the image. Sometimes planning may be a combination of special action from "allowed actions' and multiple visible or hidden objects in the image, e.g., '[PutObject] <egg> <egg><<icrowave>>>>>>>>> where the '[PutObject]' action is a special action that should be followed with two objects. Note that the objects you can use are not only limited to those visible ones, there also some hidden objects that exist for sure in the environment of the provided image and you can also them based on your commonsense. Your planning should be efficient to complete the task description, do not plan irrelevant steps or miss crucial steps. Your response should directly start with and only contain the planned action-object
element in your
40

---

Published as a conference paper at ICLR 2025
sequences, do not generate anything else.
Instruction>:
<task image>:
Your planning:
F MODEL PARSE PROMPTS
Image2Teext Free-form
this task, want you act a judge.
You will be provided with a question, its golden answer(s), and the model's answer, while the context of the question, which is one or more images, is not given here. Your task is judge how correct the model's answer is based the golden answer(s). without seeing the input images of the question, and then give correctness score. The correctness score should one of the below numbers: 0.0 (totally wrong). 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, or 1.0 (totally right). Your should first briefly give your reasoning process regarding how the model's answer conforms to contradicts the golden answer(s). and then give the correctness score. The correctness score must strictly follow this format: [[scorc]]: The correctness score: [[0.5]]: Below are some examples.
Example
Question: what advertising?
Golden Answer(s): <answer L>garden annual; <answer seeds:: <answer 3>ssedss: < <answer
4>>seeds: <answer 5>>seds;; <answer 6>seeds; <answer 7>seeds: <answer s>seeds; <answer
9>>eeds; <answer 10>cole's garden anmual
Model's Answer: Seed
Your Judgment: The golden answers consistently mention "seeds" suggesting an advertisement for a seed catalog. The model's answer, "Seed", aligns exactly with this description. The Correctness
Score: [[1.0]]
Example
Question: Who making face?
Golden Answer: <answer I 1>>hhld
Model's Answer: A
Your Judgment: The golden answer specifies "child" making face, but the model answered "A man", which incorrect as refers to a different age group. The Correctness Score: [[0.0]]
Example 3:
Question: what road the right?
Golden Answer: <answer Irouubblso valley rd; <answer 2>troublesome valley rd.; <answer
3> troublesome valley; <answer trouubllse valley road; <answer 5>valley road; <answer trrooubleeomme valley; <answer 7>troublesome valley road; <answer 8>>troublesome valley <answer 9>>troublesome valley rd; <answer 10>-rrublesome valley rd.
Model's Answer: troublesome road
Your Judgment: The golden answers specify name of road as "troublesome valley rd" or variations of this phrase with consistent reference to "troublesome valley." The model's answer, "troublesome road," captures the "trouhlesome" aspect but omits the critical "valley" part of the name, which is crucial for full accuracy. Thus, the model's answer partially matches the golden answer but lacks complete specificity. The Correctness Score: [[0.6]]
Note that each one of the golden answers is considered correct. Thus if the model's answer matches any of the golden answers, it should be considered correct. Judge the below case, give the bricf reasoning process and the correctness score.
Question: prompt
Golden Answer(s): gold ans
Model's Answer: response
Your Judgment:
41

---

Published as a conference paper at ICLR 2025
Tmate2Text Multiple-choice
In this task, want you to act as an option extractor.
You will be provided with multiple-choiee question, its options, and the model's answer, while the context the question, which one more images, not given here. Your task is extract or judge which option is chosen by the model based on its response, without seeing the context of question. The extracted option should be one of the provided option letters. Your should first briefly give your reasoning process, and then give the extracted option letter. The extracted option must strictly follow this format: [[option letter]]: e.g.. The option chosen by the model: [[A]] Below are some examples.
Example I:
Question: Where the cast of the television show located in the image?
Options:
In the foreground
B. In the background
C. In the center
At the edges
Model's Answer:
center
Your Judgment: The model's answer clearly states "C. In the center", indicating that the correct option, according to the model, is in the center. The option chosen by the model: [[C]].
Example 2:
Question: <image 1,zn the was painted during
Options:
A. first or second century C. E.
sixth seventh century C. E. tenth eleventh century C.E.
fourteenth or fifteenth century E.
Model's 's Answer: The correct answer is option D. the fourteenth or fifteenth century C.E. Your Judgment: The model's response specifies "option D, the fourteenth or fifteenth century C.E." The option chosen by the model: [[D]].
directly the correct
Example
Question: what does the diagram show's you information about
Options:
Photosynthesis
The plant getting fed
C. A picture of the plant
D. What happens to plant daily
Model's Answer: The diagram shows the process of photosynthesis, which the process by which plants convert sunlight, carbon dioxide, and water into oxygen and glucose. Judgment: The model's answer mentions "the process photosynthesis, which directly corresponds to option A, "Photosyuthesis" Therefore, the correct option according to the model is photosynthesis. The option chosen by the model: [[A]].
Give the reasoning process the extracted option the helow case:
Question: prompt
Options:
options
Model's Answer: response
Your Judgment:
Video2Text Free-form
In this task, I want act as a judge.
You will be provided with question, its golden answer(s), and the model's answer, while the context the question, which is one or more videos, is not given here. Your task to judge how correct the model's answer is based on the golden answer(s), without seeing the input videos of the question, and
42

---

Published as a conference paper at ICLR 2025
then give correctness scorc. The correctness score should be one of the below numbers: 0.0 (totally wrong), 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0 (totally right). Your should first briefly give your conforms contradicts the golden answer(s).
reasoning process regarding how the model's and then give correctness score. correctness score must strictly follow this format: [[score]]: examples.
e-g-, The correctness
[[0.5]]: Below
Example
Question: what does this video want to express
Golden Answer(s): <answer i>introduce method of playing
Model's Answer: Volleyball serve
Your Judgment: The model's answer "Volleyball serve" suggests specific action, which may be part of what the video demonstrates. However, it misses the broader educational intent implied by the golden answer "introduce method of playing". Therefore, the answer is partially correct. The Correctness Score: [[0.5]]
Example 2:
Question: who do two other boys with surprised looks assist up? Golden Answer(s): <aaswwer 1>bbyy
Model's Answer: Boy.
Your Judgment: The model's answer "Boy." precisely matches the golden answer which states the two
other boys assist "boy". The Correctness Score: [[1.0]]
Example 3:
Question: what the lady
end of video after their performance Golden Answer(s): <answer 1>picks up her phone
Model's Answer: Nothing.
Your Judgment: The model's answer "Nothing." directly contradicts the golden answer which states that the lady "picks up her phone" the end of the video after their performance. Since the model's response completely misses the specific action described in the golden answer, incorrect. Correctness Score: [[0.0]]
Note that each of the golden answers is considered correct. Thus if the model's answer matches any one the golden answers, should considered correct. Judge the below case, give brief reasoning process and the correctness score.
Question: prompt
Golden Answer(s): gold
Model's Answer: response
Your Judgment:
Video2Text Multiple-choice
In this task, want you to act as an option extractor.
You will be provided with multiple-chnice question, its options, and the model's answer, while the context the question, which is one or more videos, not given here. Your task extract or judge which option is chosen by the model based on its response, without seeing the context of the question. The extracted option should be one of the provided option letters. Your should first briefly give your reasoning process, and then give the extracted option letter. The extracted option must strictly follow this format: [loption letter]], c.g., The option chosen by the model: [[A]] Below are some examples.
Example 1:
Question: What did he do the car?
Options:
A. Paint the car
B. Put plastic over the car
C. Put metal over the car
Cut the
Model's Answer: put plastic over car.
Your Judgment: The model's response directly aligns with option B, which is "Put plastic over the
43

---

Published as a conference paper at ICLR 2025
car." The response given is paraphrase of this option without deviating in meaning. The option
chosen by the model: [[B]]
Example
Question:
How did Eddie know Pam and Justin before Justin was killed?
Options:
They part of the theater company
They were high school friends
They went to college together
D. They were cousins
F. They were siblings Model's Answer: A.
Your Judgment: The model's answer directly provides the option letter "A." The option chosen by the model: [[A]]
Example 3:
Question: why the people move the same manner
Options:
A. uniform
B. dancing with the baby
C. exercising together
stay together
singing and dancing
Model's 's Answer: sing and dance
Your Judgment: The model's response "sing and dance" closely aligns with option E, which is "singing and dancing." The response provided is direct paraphrase of this option, modifying only slightly the form of words (from gerund to infinitive) but maintaining the same core activities described in the option. The option chosen by the model: [[E]]
When you think that the model's answer does not match any of the given options, please choose the option that the closest to the model's answer. Give the brief reasoning process and the extracted option for the below case.
Question: prompt
Options:
options
Model's Answer: response
Your Judgment:
Audio2Text Free-form
In this task, want you to act as a judge.
You will be provided with question, its golden answer(s), and the model's answer, while the context of the question, which is model's answer is based
more audios, is given here. Your task judge how correct the golden answer(s), without seeing the input audios of question, then give a correctness score. The correctness score should be one of the helow numbers: 0.0 (totally wrong). 0. 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8. 0.9. 1.0 (totally right). Your should first briefly give your reasoning process regarding how the model's and then give correctness score. correctness score must strictly follow this format: [[score]]: examples.
conforms
contradicts the golden answer(s).
e.g., The correctness
[[0.5]]: Below
Example
Question: Are the people isolated?
Golden Answer(s): <answer 1>-no
Model's Answer: yes
Your Judgment: The model's answer contradicts the golden answer directly. The question asked the people are isolated, to which the golden answer is "no," indicating that the people are not isolated. However, the model's answer is "yes," implying that the people are isolated. The correctness score: [[0.0]]
44

---

Published as a conference paper at ICLR 2025
Example 2:
Question: Who is speaking?
Golden Answer(s): <answer 1>man
Model's Answer: men
The model's answer is almost correct but imprecise. The question asked about the identity of the speaker, to which the golden answer specifies singular "man." However, the model's answer is "men," which suggests multiple individuals rather than one. This small pluralization emoor suggests misunderstanding the query about the exact number of people speaking. The correctness score: [[0.6]]
Your Judgment:
Example 3:
Question: What did you hear after the door slamming?
Golden Answer(s): <answer 1>dog making noise
Model's Answer: dog
Your Judgment: The model's answer "dog" matches the golden answer's essential element, "dog making noise," by correctly identifying the dog. Although it omits "making noise," it captures the key information needed. The correctness score: [[1.0]]
Note that each one of the golden answers is considered correct. Thus if the model's answer matches any of the golden answers, it should be considered correct. Judge the below case, give the brief reasoning process and the correctness score.
Question: prompt
Golden Answer(s): gold_ans
Model's Answer: response
Your Judgment:
Text2Image turn
this task, you will act as an impartial judge image generation tasks.
Please act impartial judge and evaluate the quality of image generated by an Al assistant given the provided user prompt caption.
You must first analyze the generated image based on the provided prompt carefully. After providing your analysis, you must give the final score on a scale of to 10 (1 means extremely bad and 10 means extremely good), and the rating must strictly follow this format: [[rating]]> for example: Rating: [[S]],
should consider following core aspects when analyzing image:
1. **Aligmmen**; Assess how accurately the image reflects the given prompt. Check if all elements and requirements correctly represented.
2. ** *ealism***; Judge if the image looks realistic and natural.
3. ** **Quality**;; Identify there's any flaw image, such distortion, blurriness, or illogical representation facial features, limbs, fingers, objects, or text. In addition, evaluate the overall quality of the image.
Analyze and judge the below case:
Generation Promp/CCaption: prompt1
Generated Image: imagel
Your Analysis and Judgment:
Text2Image turn 2
task, you will act as an impartial judge an image editing task.
You will be provided with image edit, the user prompt to edit the image, and the edited image. Your task is to evaluate the quality of the edited image based the given information.
45

---

Published as a conference paper at ICLR 2025
You must first analyze the edited image based on the provided editing prompt and the image to edit carefully. After providing your analysis, you must give the final score scale of to 10 (1 means extremely bad and 10 means extremely good), and the rating must strictly follow this format: [[rating]]: for example: Rating: [[S]]]
You should consider the following core aspects when analyzing the image:
I. *" "Aligmment**; Assess how accurately the edited image reflects the changes indicated in the given editing prompt. Check if all elements and requirements are correctly meeted. **Consistency*#: Evaluate the edited image is consistent with original image terms details, style, color, overall appearance, etc.
3. **Realism**, ** Judge if the edited image looks realistic and natural after the editing process.
4. **Quality#*; Identify if there's any flaw in the edited image, such distortion, blurriness, illogical representation facial features, limbs, fingers, objects, or text. addition, evaluate the overall quality of the image.
Analyze and judge the below case:
Editing Prompt/Caption: promptl The Image to Edit:
Text2Action
In this task, you will act as impartial judge for real-world planning task.
Your job is to evaluate quality the action-object sequences planned by an assistant a real-world task. You will be provided with the Task Description, Allowed Actions, Visible Objects, Already Executed Action-Object Sequences, the target, and the model's response. The "Task Description' is the task. The Alllowed Actions' is list of actions that are allowed be used by the assistant complete the task. The 'Visible Objects' a list objects that are assumed be visible to the Al assistant when it's completing the task. The AAlraady Executed Action-Objes Sequences' is list of action-object sequences that are assumed to have been completed by the AI assistant at the moment of starting the planning. The 'Reference Answer' is example action-object sequence output for your reference, which is annotated by a human and may not be the only correct answer. The 'Model Response' is the output of the Al assistant you are evaluating.
instruction that instructs the Al assistant, which is being evaluated, to complete
Your task is analyze the model's response and evaluate how well it plans given the information and the reference answer. After providing your analysis, you must give the final score a scale of 10 (1 means extremely bad and means extremely good), and the rating must strictly follow this format: [frating]]: for example: Rating: [[5]]:
Below is simplified example of how judge the model's response:
**Start Example**
Task Description: Put heated egg in the sink.
Allowed Actions: [OpenObject), (CloseObject), [PickupObject], [PutObject], [ToggleObjectOn). [ToggleOhjectOf1,, [SliceObject), [Navigation]
Visible Objects: <microwave>> <sink>> <toaster>, <coffee maker>, <fridge>, <blender>,
<potato>>, <bows>, <egg>>, <garbagecan>>
Already Executed Action-Objes Sequences: [Navigation] <fridge>, [OpenObject] <fridge>,
[PickupObject] <egg>>. [CloseObject] <tridge>, [Navigation] <microwave>, [PutObject]
<egg> <egg><<nicrowave>>>>>>>
Reference Answer: [ToggleOOjcctOn]] (aairrowwve>,/Toggoooooo
<microwave>, [PickupOb-
ject] <egg>,, INavigatiinn <sink>>, [PutObject] <sink>, <egg><<ink>>
Model Response: [PickupObject] <egg>>, [Navigation] <sink>> [PutOhject] <egg><<ink>>>>> Your Analysis and Judgment: The model's response omits crucial steps for heating the egg, assuming it already heated without evidence from prior actions. correctly performs the transport and placement of the egg, using appropriate actions and objects. However, by neglecting the heating process essential to the task description, the response is incomplete. My Final Rating: [[3]]. ** End of Example**
46

---

Published as a conference paper at ICLR 2025
With the above description and example, analyze and judge the below case:
Task Description: task description
Allowed Actions: allowed.actiin actions
Visible Objects: visible.objectt
Already Executed Action Object Sequences: already executed steps
Reference Answer: target
Model Response: response
Your Analysis Judgment:
Image2Action
In this task, will act as impartial judge for real-world planning task.
Your job evaluate quality of the action- action-object sequences planned by Al assistant visual perception for real-world task. You will be provided with the Task Description, Allowed Actions, Visible Objects, Already Executed Action-Ohject Sequences, the target, and the model's response.
The 'Task Description' is user instruction that instructs the AT assistant, which is being evaluated, complete the task. The 'lllowed Actions' list of actions that allowed he used by the AT assistant to complete the task. The "Visible Objects' is of objects that assumed to be visible the Al assistant when it's completing the task. Note that some invisible objects may still be usable to the Al assistant, but their existence must be consistent with the commonsense. The AAready Executed Action-Object Sequences' is list of action-object sequences that assumed have been completed by the Al assistant the moment of starting the planning. The 'Reference Answer' is example action-object sequence output for your reference, which annotated by a human and may not be the only correct answer. The 'Moeel Response' is the output of the AI assistant you are evaluating.
Your task is analyze the model's response and evaluate how well it plans given the mentionen information and the reference answer. After providing your analysis, you must give final score on a scale of to 10 (1 means extremely bad and 10 means extremely good), and the rating must strictly follow this format: [[ratin]]]: for example: Rating: [[5]]:
Below simplified example of how to judge the model's response:
Starar of Example***
Task Description: Get the egg from the fridge, and the heated in the sink. Allowed Actions: [OpenObject], [CloseObject], [PickupObject), [PutObject], [ToggleObjectOn] [ToggleObjectOI1], [SliceObject], [Navigation] Visible Objects: imagel
Alrcady Executed Action
Action-Object Object Sequences: [Navigation] <fridge>, [OpenObject] <fridge>,
[PickupObjeet] <egg>> [CloseObject] <fridge>, [Navigation] <icroowvee>, [PutObject]
<egg>><microowve>>>>>>>>>>>>>>>>
Reference Answer: [ToggleOhjectOn]] (iaiicoowaeee,Tooooeotoeee
<microwave>, [PickupOh-
ject] <egg>>, [Navigation] <sink>> [PutObject] <egg.> <sink>>>
Model Response: [PickupObject] <egg>> [Navigation] <sink>> [PutObject] <egg><<ink>>
Your Analysis and Judgment: The model's response omits erucial steps for heating the egg, assuming it is already heated without evidence from prior actions. correctly performs the transport placement the egg, using appropriate actions and objects. However, by neglecting the heating process essential to the task description, the response is incomplete. My Final Rating: [[3]]. **End of Examplc**
With above description and example, analyze and judge below case:
Task Description: task description
Allowed Actions: allowed_actions
Visible Objects: image2
Already Executed Action-Object Sequences: already_.xxecuted.tte
Reference Answer: target
Model Response: model response
Your Analysis and Judgment:
47

---

Published as a conference paper at ICLR 2025
G BENCHMARK POOL DETAILS
Image2Text: MMMU (Yue al., 2024), MMBench (Liu et al., 2023b), SEED-Bench (Li et al., 2023b), SEED-BBnch 2 (Li et al., 2024b), ChartQA (Masry et al., 2022), A-OKVQA (Schwenk et al., 2022), HallusionBench (Guan al., 2024), Math Vista (Lu et al., 2023), GQA (Hudson & Manning, 2019), MM-Vet (Yu et al., 2023b), ScienceQA (Saikh et al., 2022), DocVQA (Mathew al., 2021), POPE (Li al., 2023e), InfographicVOA (Mathew et al., 2022), Q-Bench (Wu al., 2023), VisWiz (Gurari et al., 2018), and TextVQA (Singh et al., 2019)
Vixeo2Text: Activity QA (Yu et al.. 2019), HowToQA (Li et al., 2020), STAR (Wu al., 2024), TVQA (Lei et al., 2018). TGIF-QA (Jang et al., 2017). EgoSchema (Mangalam et al., 2023), SUTD- TrafficQA (Xu al., 2021). NexIQA (Xiao et al., 2021), PororoQA (Kim et al., 2017). IVQA (Liu et al., 2018), WildQA (Castro et al., 2022), Perception-Test (Patraucean et al., 2024), MSVD-QA (Xu al., 2017), and Social-IQ-2.0 (Zadeh et al., 2019)
Audio2Text: Clotho-AQA (Lipping et al., 2022), DAQA (Fayek & Johnson, 2020), and CLEAR (Lin et al., 2021)
H MODEL DETAILS
Image2Text: Claude 3.5 Sonnet (Anthropic, 2024a), GPT-4o (OpenAI, 2024a), GPT-4V (Achiam et al., 2023), Qwen2-VL-72B (Wang et al., 2024), Gemini 1.5 Pro (Reid et al., 2024), Llama 3.2 90B (Meta, 2024b). Intern VL2-26B (Chen et al., 2023b), Claude 3 Opus (Anthropic. 2024b), Qwen-VL- (Liu et al., 2024b), Claude 3 Sonnet (Anthropic, 2024b), al., 2024), Reka Flash (Ormazabal
MAX (Bai et al., 2023), Reka Core (Ormazabal (Chen al., 2023b), Qwen- VL-PUUS (Bai et al., 2023). Claude 3 Haiku (Anthropic, 2024b), Gemini 1.0 Pro (Anil et al., 2023), IiternlM-XCCoposee et al., 2024), Omnil.MMM-229 (Yu et al., 2023a), DeepSeek-VL--7B-CC 6B (Young et al., 2024), InfiMM-ZZephrr- (Team, 2024c), MiniCPM-V (Yao al., 2024), Marco- VL, LLaV (Liu al., 2024b), SVIT (Zhao et al., 2023), mPLUG-OWL2 (Ye al., 2024), SPHINX (Lin et al., 2023b), InstructBLIP-S--XX (Dai et al., 2023), InstructBLIP-TS-XL (Dai al., 2023), BLIP-2 FLAN-T5-XXL (Li et al., 2023c), BLIP-2 FLAN-T5-XL (Li et al., 2023c), Adept Fuyu-Heavy (Team, 2024a), LLaMA-Adapteer--75 (Gao et al., 2023), Otter (Li et al., 2023a), MiniGPT4-Vicuna-I1 (Zhu et al., 2023)
al., 2024), IntemWL-Chat-Vi
(Dong et al., 2024), Yi-VL-34B (Young
(Lu et al., 2024a), Yi-VL-
Video2Text: Claude 3.5 Sonnet (Anthropic, 2024a), GPT-4o (OpenAI, 2024a), Gemini 1.5 Pro (Reid et al., 2024), GPT-4V (Achiam et al., 2023), Owen2-VLL-72B (Wang et al., 2024), Gemini 1.5 Flash (Reid et al., 2024), LLaVA-One V (Li et al., 2024a), Qwen2-vL-77 (Wang al., 2024), LLaVA-Next-Vii Next-Videoo33BB (Zhang et al., 2024a), Claude 3 Haiku (Anthropic, 2024b), (Zhang et al., 2024a), Reka-edge (Ormazabal et al., 2024), LLaMA-VID (Li al., 2023d), VideoLLaVA (Lin al., 2023a), Mideo-ChatGPT (Maaz al., 2023), mPLUG-video (Li et al., 2022)
Audio2Text: Gemini 1.5 Pro (Reid et al., 2024), Gemini 1.5 Flash (Reid et al., 2024). Qwen2- Audio-7B- Milio--B--nssruct (Chu et al., 2024), Qwen2-Audio-70 (Chu et al., 2024), SALMONN-13H (Tang et al., 2023), Qwen-Audio (Chu et al., 2023), Owen-Audio-Chen (Chu et al., 2023), SALMONN-7B (Tang et al., 2023), Pengi (Deshmukh et al., 2023)
Text21mage: Flux (BlackForestLabs,, 2024), DALL-E HD (Betker et al., 2023), PixArtAlpha (Chen et al., 2023a), PlayGround V2.5 (Li et al., 2024c), PlayGround V2 (Li et al., 2024c), 5 (Rombach et al., 2022), SD3 (Esser et al.), SDXL (Podell et al., 2023), Stable Cascade (Pernias et al., 2023)
Text2Video: ModelScope (Wang et al., 2023a), ZeroScope V2, Cog VideoX-BB (Yang et al., 2024), HotShot-XL (Mullan et al., 2023), LaVie (Wang al., 2023b), Show-1 (Zhang et al., 2023), VideoCrafter2 (Chen et al., 2024)
Text2Audio: AudioLDM 2 (Liu al., 2024a), Make-An-Audio 2 (Huang et 2023), Stable Audio (Evans et al., 2024). Tango 2 (Majumder et al., 2024), ConsistencyTTs (Bai et al., 2024), AudioGen (Kreuk al., 2022), Magnet (Ziv al., 2024)
48

---

Published as a conference paper at ICLR 2025
Text2Action: GPT-4-Turbo (Achiam et al., 2023). Gemini 1.5 Pro (Reid et al., 2024), Mistral-Large 2 (Team, 2024b), GPT-4o (OpenAI, 2024a), Reka Core (Ormazabal et al., 2024), Claude 3.5 Sonnet (Anthropic, 2024a), 1emma--2-9-Instteee (Team et al., 2024), Reka Flash (Ormazabal al., 2024), Claude Haiku (Anthropic, 2024b), Mistral-Medium (Jiang et al., 2023), Gemini 1.5 Flash (Reid al., 2024), Qwen-2-72BBInstteee (Wang al., 2024), Mistral-Small (Jiang et al., 2023), GPT-4o-Mini (OpenAL, 2024b), Yi-1.5-34B-CC (Young et al., 2024),
(Meta, 2024a),
(Meta, 2024a), GPT-3.5-Turbe (Achiam et al., 2023), Qwen-2-7B- Instruct (Wang et al., 2024), Yi-1.5-9B ----hhh (Young et al., 2024), Reka Edge (Ormazabal et al., 2024)
Image2Action: GPT-4V (Achiam et al., 2023), Claude 3.5 Sonnet (Anthropic, 2024a), GPT-4o (OpenAI, 2024a), Claude 3 Opus (Anthropic, 2024b), Claude Sonnet (Anthropic, 2024b), Claude 3 Haiku (Anthropic, 2024b), Gemini 1.5 Pro (Reid et al., 2024), Intern Ve-Caat--aaaVii (Chen et al., 2023b), Qwen-VLL-MAX VL-MAX (Bai et al., 2023), Qwen- VI.-PIUS (Bai al., 2023), InfiMM- ZepMy-- (Team, 2024c), DeepSeck-VL-7B-CHK -htt (Lu et al., 2024a), MiniCPM-V (Yao et al., 2024), Yi-VL- 34B (Young et al., 2024), Qwen2-VL-72B (Wang et al., 2024), IntemLM-XCooposer22- (Dong et al., 2024), LLaVA-1.6-13B (Liu et al., 2024b), ALLaVA-166-344 (Liu et al., 2024b)
Table 2: Correlations between Model Judges and Human Preference Elo
GPT
Claude
Gemini
Avg.
1st turn
0.82
0.68
0.78
0.83
2nd turn
0.67
0.56
0.6
0.58
Avg.
0.75
0.8
0.83
0.78
49

---

Published as a conference paper at ICLR 2025
Geometry Problems
Diagnosis Food Web Physical Sciences
Food Anirnal identification Object Comparison Companison Appearance Description
Image Quality Blur Analysis Naise Issue Exposure issue Emotion Detection
Algebre Questions
Physies Calculations
Engineering Design
Biology
Leaf Identification
Image Captioning
Numerical Comparisar
Image Description Seasonal Identification
Identification Environment Description
Object Analysis Scene Description
Graph Interpretation
Lunar Phases Experiments/ Temperature WeeperttureWWether
Image Details Image Analysis PythorvCode Requests
Arithmetic Language Translation
Image Composition
Emotion/Maod
Demagraphics Ecanomic Data Internet Statistics Health Event information
Details Dates License Plates Screen Content
Object: Lecation Vehicle identification
Clothing/Altire
Relationship
Food Description
Expression/Emaation
People Actions Quantity Count
Art/Style Actions/Objects
Geography Highlights
Capital Cities Landmarkss Brands/Companies Miscellaneou Questions Sports Trivia
Image Identification information Location Inquiry Movie introduction Cooking Instructions
Artwork Titles Person Identification Location Identification Type Translation Verification
Social Media
Capitals/Population Historical/Economiic
Directions/Location
Figure 35: Query topic summarization for the Image2Textt queries in Figure The plot aggregates all queries and divides them into 16 regions. From each region, 100 queries are uniformly sampled and analyzed by GPT-4 for topic summarization.
50

---

Published as a conference paper at ICLR 2025
Occlusion Game Objects Identical Objects Transparent Hidden Object Location Faod/Dish Activity
Food Preparation
Accident Location
Geametric Shapes
Road Conditions
Accident Type Road Condition Incident Detai is Contributing Factors
Letter Matching
Vehicle Types
Object Containers and Covers
Traffic Analysis Environmental Details
Interaction Animal Behaviae
Actions Objects Ingredients Animals People
Count Peocle Identify Actions Time Stamps Sports Descriptions
Sequence of Events Main Activity Location Description of Action Purpose/Objectivs
Person Actions
Questions Reactions
People Interaction Performanzes Activitles Descriptions
Color Identification
Clothing Objects Locations
Critical Actions
Person Location
Scenes/Events Predictions
Human Interaction
Count Information
Reactions
Character Interactions
Show Dialogue Character Reactions
Emotional Tone Relationships Reactions Demeanor Opinions
character Director
Baby Actions Relationships Locations Emotions
Feelings/Reacton Next action Visual quality
Object Descriptions Event Questions
Figure 36: Query topic summarization for the Video2Text queries in Figure 9. The plot aggregates all queries and divides them into 16 regions. From each region, 100 queries are uniformly sampled and analyzed by GPT-4 for topic summarization.
51

---

Published as a conference paper at ICLR 2025
Location Environment Nearby Buildings
Weather Conditions
Water Sounds
Sound Characteristics
Sound Position
Scene Pasition Sequence Sounds Count of Specific Notes
Sound Loudness Sound Brightness Instrument Note
Plano Lyrics Product
Objects and Actions Miscell aneous
Activity Inquiry Work Progress Location Query Speech Content Misceillaneous
Speaking Detertion Speech Content Ambient Sounds
Animal Sounds Vehicle Saunds Instruments Natural Environment Sound Characteristics
Brightness Louciness
Sound Count Sequence
Description Audio Captions
Volume Change
Audio Interpretation Human Speech Analysis Emotional Response Podcast Proferences Miscellaneous
Duration Sound Loudness
Conversations Contextual Inquiries
Sound Sequences Sound Identification
Sound Event Sequence
Human Reactions Comparative Sounds
Sound Comparison Specific Sound Inquiry
Invitations Calls Requests Statements
Phone Ringing Court/Legal Experiences
Human Tyyping Emergency Vehicles
Frequency Comparison
Loudness Comparison
Phone Ringinn
Equal Occurrences
Questions Credit Card WaminG
Crowds Rioting
Crowd vs. Other Noises
Vehicles Passing
Shapiro Query
Figure 37: Query topic summarization for the Audio2Text queries in Figure 9. The plot aggregates all queries and divides them into 16 regions. From each region, 100 queries are uniformly sampled and analyzed by GPT-4 for topic summarization.
52

---

Published as a conference paper at ICLR 2025
Dictionary Words News Articles
Planning Order Menagement Domain and Web Data Trading Service and Installation
Directions Internet Secunity Medical Referral
Data Deletion
Rollback Data
Comics Recommendations
Clone Command
ation Requests
Cookies Map Dueries
Audia Options
GitHub Repository
Settings Navigation
Subscristions Educational Health and Safety User Awareness Reporting and Evaluation
Notifications Adjust Settings Administer Medication Instructions Mai Maintenance Tasks
Computer Operations
commencce Subreddit Posts User Reviews
Garning Configuration Navigation Miscellaneous
Political Advocacy
Stacking
Seasonal Activities
Delivery Investigation Relationships Safety
Exercise Safety Practices
Object Actions Game Combet
Call Action Daily Updates
Games/Sparts Travel Tips
Logistics Planning
Personal Requests
Gameplay Mechanics
Weapons
Instructions Scientific Procedures
Model Engines Archimedes Experiment Mechanics
Winning Conditions
Dress Code
Fun Stealing Basement Beers
Demans Assistance
Animal Care
Cleaning
General Rules
Guests
Handling Materials
Ovals
Figure 38: Query topic summarization for the Text2Action queries in Figure 9. The plot aggregates all queries and divides them into 16 regions. From each region, 100 queries are uniformly sampled and analyzed by GPT-4 for topic summarization.
53

---

Published as a conference paper at ICLR 2025
Fuel Tank Drawing Wiring Drilling Clamping
Image Editing Drawing Photo Enhancements File Operations Scan Adjustments
Medical Procedures Assembbly Instructions
Navigato/MMove Adjust Settings Delete/Remove
Safety Instructions Technical Process
Gardening Hausehold thores Baby care placement Health Instructions
Gun Manipulation
Zoom/Enlarge Analyze Image Game Instructions Printing/ legt Prints VR/3D viewing
Clicking Scrolling Tapping Typing Viewing
Laser Calibration
Medication Label.
Arch Bricks Measuring Techniques
Cooking Potato Moving Objects Labeling Kitchen Counters Safety Storing Iterns
Measuring Construction
Image Identification
Action and Interaction Creative Arts Missions Quests Costumes and Props
User Authentication Game Instructions Map/Navigation Poster Printing
Reglster/Login Change Username Create/Post Content
Messaging
Photography Travel Dining Shopping Assistance
Directions Landmarks Photography Safety Alerts Miscellancous
Upvote Requests Mavigation/DDriving Information Retrieval
Subscription Requests
Comment Requests Price Inquiries Viewing Requests
CalendariAlarms instructions/Actions
Navigation Requests
Figure 39: Query topic summarization for the Image2Action queries in Figure 9. The plot aggregates all queries and divides them into 16 regions. From each region, 100 queries are uniformly sampled and analyzed by GPT-4 for topic summarization.
54

---

Published as a conference paper at ICLR 2025
0.25
0.15
0.10
0.08
0.05
0.08
0.02
0.02
0.00
SEED (multiplo ahoice)
GQA (fre--frrm)
MMMU (multiple choice)
DacVQA (aee-A((ree-fomm) (freeffom))
ToxVOA (free- form]
VieWit (free- (free-fom) fom)
Inforaphicvck (free-form)
(rmultittt choice)
Q- B-Bunch (mmitiiu (muttple-chhooo shole)
ADKYOA (free- (free-form)
MMBancc (muutipl chatoe)
HallusionBandh (muttipl ohalos)
CharrOA for--form)
MaihVists (mulitpt holeee
(free- (feevat(freo-oon form)
HNMU (free 4orm)
MathVata (free- ro))
NH+-alt
Figure 40: Image2Text benchmark pool distribution on benchmark level.
0.28
0.15
0.10
0.05
0.04
3.04
0.08
0.91
3.01
3.01
3.01
0.00
SEED (multiplo-- cholce)
GQA (frae-fom) fomm)
VinWir (fred form]
DocKOA form)
SEED2 (muttiple (muttipp----ccooco e-chhicccee
ToxtQA ToreeA(frre-tn form)
InfogrophicVon (free farm)
O-Bensh (multimie mamttiiie-chtlld chttle
MMBBench (multppo- -
(fres-form)
MM4-Vet (fres-form)
cholce)
Mathvists (multipi sholce)
Holtalannenchh inulipple choice)
MMMU (frees form)
MathViita (fress form)
MMMU (multple-cholle
ChamQA C (fress
Figure 41: Image2Text Hard benchmark pool distribution on benchmark level.
55

---

Published as a conference paper at ICLR 2025
0.20
0.15
0.14
0.10
0.10
0.10
0.05
0.08
0.08
0.00
0.00
ActivitNeet-Q QA (free- form)
TVQA (muilippe choice)
MSVD- QA (free- (tres-fom) form)
NovCAA freetfettttxteeeeeee form)
HearQA-choice inuttipls- cholle)
TOIF TOIF-QA frres: form)
TOIF- TOIF-QA imuitippe chaice)
EjoSchema (mwrtsppts tnultppts-choica) choice)
IVDA ffrep form)
Sociat-QQ22 2.0 imultiple cholee)
SUTD- SUTD-Trattin TraffteQA imuldppl loultpple-choice choice))
Perception Text (muttiple choice)
STAR enultipte imuttipt--choice choice)
How NoW (muttppl-chhne
Figure 42: Video2Text benchmark pool distribution on benchmark level.
0.25
0.25
0.20
0.16
0.14
0.12
0.10
0.08
0.05
0.07
0.06
0.06
0.05
0.03
0.01
0.00
0.00
ActwityNtt-- QA (free-form) form]
TVGA imuttiple chaic)) choice)
HowToQA tnuitiple chnine)
MontA MontgA-frestrtiooo (free form)
NeaOA-choica truttiple chattl choice)
EgoSchemaa fnultiple choics)
IVAA (ellffee-fomm foml
Social-10- 2.0 imultiple chaice)
TOIF QA (free form)
MSVD QA frree form)
TOIF OA (muttiple choiae)
SUTD- SUTD- Trafififf (multiple muuitpleahacaaa choice)
STAR (muttiple choic)
Percaption- Text fm fmultiple choice)
Figure 43: Video2Text Hard benchmark pool distribution on benchmark level.
56

---

Published as a conference paper at ICLR 2025
0.58
0.5
0.4
0.30
0.3
0.2
0.1
0.01
0.0
Clatho-AOA (tree- (tree-fom) fomm
DADA DAQA (free-fom) fom)
CLEAR form)
Figure 44: Audio2Text benchmark pool distribution on benchmark level.
0.58
0.5
0.5
0.4
0.3
0.2
0.1
0.01
0.0
Clothe-AOA (free-fom)
DAQA (free- (frme-fom) form)
CLEAR form)
Figure 45: Audio2Text- Hard benchmark pool distribution on benchmark level.
57

---

Published as a conference paper at ICLR 2025
1003 1055
1065
1037
The First Turn
1027 1032
1032
1022
1020
1005
996
968
DALLEE HD
Flus
Por PorArcAlona
PlayGroudd V2.5
PlayGround V2
Stable Cascade
SD
SOXL
HotShot--R -XL
CogVideek- SB
Lavie
VideoCrafter2
ModeetScopp
Show-1
ZeroScope
AudioLDK 2
Make--nn An-Audio 2
Stable Audio
Comsistecyy
Tango 2
AudloGen
Magnet
1058
1025
1036
Second Turn
1042
10321001
1022
1018
1008
1004
1026
1019
1012
1007
008
998
390
985
956
918
912
Piux
DALL- ALL-E HD
PlayGround V2.5
PlyGround V2
SD3
Stable Carcado
SD1.5
SDKL
HotShat-it Xt.
LaVie
CogViduoX: SB
ModelScope
ZeroScope V2
VidvoCrafte Crafte22
Show
Audiot.N DM
Tengo 2
Stable Audio
Make- Make-An Audio
ConsustencyTi
AudooGen
Magnet
Text2Image
Texi2Video
Tex12Audio
Figure 46: The turn-level scores of MMG tasks. MMG tasks designed to be two-turn inter- leaved ones, where the first turn a generation task and the second turn is an editing task based on the content generated in the first turn and the history instruction.
58


Reviews:
[{"review_id": "MipmBzeNYR", "reviewer": "Reviewer_xVxi", "summary": "The paper introduces MixEval-X, a large benchmark which can be used for evaluating large multimodal models in large number of multimodal tasks, ranging from understanding to generation. The paper also shows an extensive evaluation of several existing approaches to the considered tasks, providing a reference for future works. It also introduced a carefully designed pipeline for the evaluation of different models.", "questions": "- The discussion about dynamism and contamination is not very clear (L153). Could the authors clarify this?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The evaluation provided in the paper is extensive, both considering the number of tasks and models evaluated. This paper can really become a reference for many future works.\n- The tasks considered are handling very different modalities, which can foster the adoption of this dataset from different AI sub-communities.\n- The user studies are carefully designed and involved a high number of annotators.", "weaknesses": "The paper is very solid, with few weak points. It presents several contributions. This can make it somewhat challenging to read due to the frequent cross-references throughout the document. However, I see this as a minor issue, largely due to the richness of the content.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ty3otMmrLr", "reviewer": "Reviewer_U4Qy", "summary": "This paper focuses on introducing reliable evaluation methods to guide the development of multimodal large language models (LLMs). The authors extend MixEval to cover a wider range of tasks and incorporate additional modalities by proposing a multimodal benchmark mixture and adaptation-rectification pipelines to better reflect real-world task distributions. Furthermore, the paper benchmarks multimodal LLMs on their capabilities across text-to-X, X-to-text, and agent tasks, where X represents modalities such as images, videos, and more.", "questions": "1. Ensuring Accuracy of Web Queries: How is the accuracy of information retrieved from web queries verified or ensured?\n\n2. Handling Temporal Shifts: Given that accurate information can change over time (e.g., updates in data or events), how does the system handle these temporal shifts to maintain reliability?\n\n3. Preventing Data Leakage: How does the benchmarking process ensure that there is no data leakage, especially considering that LLMs may have been pretrained on data sourced from the web?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This approach is simple yet effective for creating a benchmark that encompasses a wide range of tasks.\n\n2. The benchmark has been shown to align closely with the distribution of real-world datasets.", "weaknesses": "1. Missing Any-to-Any Models: Several relevant and popular any-to-any models have been omitted from the benchmark, such as:\n\n    CoDi: Any-to-Any Generation via Composable Diffusion\n\n    NExT-GPT: Any-to-Any Multimodal LLM\n\n    Including these models would provide a more comprehensive evaluation.\n\n2. Discussion of Limitations: Although not mandatory, a discussion on the limitations of the study would be valuable. It could provide context for the results and suggest areas for future research.\n\n3. Clarity of Benchmark Creation: The process of creating the benchmark is very difficult to follow. Including a diagram or visual representation would greatly aid understanding and improve the overall readability of the paper. Additionally, the method for generating the hard dataset is not well explained, and providing more detail would improve comprehension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "giWvDodksT", "reviewer": "Reviewer_Edzp", "summary": "This paper introduces MixEval-X, a new any-to-any capability benchmark for assessing performance of frontier multimodal models across a diverse set of tasks and input-output modality combinations. The paper proposes to use the framework introduced in MixEval to perform benchmark mixtures from pre-existing benchmarks to construct a well-curated benchmark that is more generic and follows real-world task distributions.", "questions": "- Where are the actual web-queries being mined from? Would it possible to add some details around this mining strategy as well as add a few of these web-queries to give a feel of what the real-world task distribution looks like?\n\n- For the non-hard version of the benchmarks, the best models already perform quite well (76.9%, 74.2%, 62.7%). Does this mean that this version of the benchmark is quite easy for current frontier models already, and has a lower saturation ceiling? Could the authors please comment on this.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- This is a very comprehensive piece of work that is very well motivated and provides a significant step in unifying multimodal frontier model evaluation going forward.\n\n- There is a substantial amount of empirical evaluations and analyses conducted on the benchmark and its constituent sub-benchmarks to showcase its real-world utility.\n\n- In particular, the combination of figs 36-39 + fig 9 is particularly striking. It can clearly help model developers assess which benchmarks are probing which particular capabilities to guide model development.", "weaknesses": "- A key limitation of the work is the lack of any discussion surrounding model evaluation costs. I believe it is important to provide details on how much it costs to evaluate any frontier models on the benchmark, i.e. both the cost of running models / APIs on the benchmark tasks but also the cost of using LLMs as judges.\n\n- It is not clear which set of sub-benchmarks use automated LLMs-as-judges, humans as annotators and which others use pre-existing ground-truth information for evaluating models. It might well be that each sub-benchmark uses a combination of these, but I believe it would be useful to augment Tab 1 with this information on what set of ground-truths and evaluation strategies each sub-benchmark uses (i.e. LLM-as-judge, preferences, ground-truth etc).\n\n- The challenge of test-set contamination: From the construction of the benchmark, it is not made abundantly clear how the proposed benchmark alleviates the problem of test-set contamination---as I understand, it is merely a combination of pre-existing benchmark samples in a principled manner by aligning with the real-world task distribution? If this is the case, it does indeed ensure a more robust distribution in the evaluation splits as well as better alignment to real-world usage, but it does not actually alleviate the concern of test-set contamination. A broad discussion point on this would be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fnp7ZIqaNz", "reviewer": "Reviewer_fCLM", "summary": "MixEval-X extends MixEval to support multiple modalities beyond text, comprising eight new subsets (excluding the original MixEval) categorized into three types: multi-modal understanding (MMU), multi-modal generation (MMG), and agent tasks.\n\n- **Benchmark Creation**: The creation process differs for each type. For MMU tasks (Text2Text, Image2Text, Audio2Text), a benchmark mixture pipeline aligns web queries with similar tasks from existing benchmarks, similar to MixEval. For MMG and agent tasks, an adaptation-rectification pipeline standardizes real-world queries, though the exact adaptation-rectification process requires further clarification.\n- **Evaluation**: Grading methods are customized for each task type. MMU and agent tasks are evaluated with LM-based parsers, with smaller models for MMU and Frontier LMs for agent tasks. MMG tasks, harder to automate, rely on crowd-sourced evaluation.\n- **Intuition & Interesting Claim**: MixEval-X retains MixEval's core philosophy: simulating web-distributed questions as a proxy for real-world questions, forming a compressed benchmark for efficient evaluation. Image2Text outputs in MixEval-X show a high correlation with Vision Arena, supporting this approach. A key finding indicates that Judge models in multimodal-generation tasks often diverge from human preferences.\n\nOverall, this work offers a comprehensive evaluation of models across diverse multimodal tasks.", "questions": "Q1. **Automatic Benchmark Creation & Grading for Agent Tasks**\n- a) Creation: Could you clarify if there are any specific checks and balances in place to ensure the action set was complete and the context made sense, especially if these tasks were primarily generated by models?\n- b) Grading: Since the grading appears to be automated, was there any human oversight to detect and correct potential errors in this pipeline or checks to indicate what fraction of inputs/model behaviour intermediate was erroneous/undefined and not caught by the grader? Ensuring clarity here would be valuable, as automated grading might lack nuanced understanding.\n\nQ2. **Scalability and Applicability of Automatic Evaluation**\n- The paper claims scalability and automated evaluation, but these seem to primarily apply to MMU tasks. For MMG tasks, wouldn’t repeated sampling from new prompts or models require costly human annotation? It seems to me that the scalability only extends to certain tasks, does this align with the broader goals and spirit of the paper?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **[Important] Comprehensive Analysis & Leaderboard**: MixEval-X offers extensive benchmarks and detailed leaderboards to rerank existing models and organizations. I appreciated the thorough, high-standard benchmarks applied across modalities and communities.\n- **[Important] Clarity**: The paper effectively motivates the setting for readers unfamiliar with the topic. It is easy to understand, with clearly stated contributions, proposed methods that address the defined problem, and well-presented and interpreted results. However, some sections may feel dense due to space constraints in the 10-page limit.", "weaknesses": "I have a few concerns asked in Questions section. However, note that I could not find major weaknesses specific to this work, which did not apply for the general MixEval framework.\n\nOverall, the method is a compelling and substantial extension of MixEval, with sound intuition and strong performance demonstrated through extensive comparisons. I would be pleased to see this paper presented at ICLR, but note that I am not a domain expert -- looking forward to seeing the other reviews and discussions with the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)