ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks
Authors: Yan Yang, Dongxu Li, Haoning Wu, Bei Chen, Liu Liu, Liyuan Pan, Junnan Li
Accepted at: acl25

ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks

Yan Yang1  Dongxu Li1 ✉
Haoning Wu2  Bei Chen  Liu Liu3  Liyuan Pan4 ✉  Junnan Li5
1ANU  2NTU  3KooMap, Huawei
4BITSZ & School of CSAT, BIT  5Salesforce AI Research
dongxuli1005@gmail.com  liyuan.pan@bit.edu.cn
Project Page: https://yan98.github.io/ProBench/

<table>
  <tr>
    <th>Coding: Screenshots and UI Elements</th>
    <th>Knowledge: Document and Text-based Images</th>
    <th>Science: Medical Images</th>
  </tr>
  <tr>
    <td><b>Query:</b> i want you to write a Rshiny code in rstudio to generate above visualization. Can you do that?</td>
    <td><b>Query:</b> Explain this framework to me in detail and in chronological order. I am an aspiring consultant and I need to know this. Also give me potential issues and solutions that will come up through this.</td>
    <td><b>Query:</b> The image above represents a H&E stain of a skeletal muscle biopsy from a young boy who came into the clinic reporting muscle weakness. You are his doctor. Does the boy have Duchenne muscular dystrophy? Explain. Your answer should include an analysis of the biopsy (you can use arrows to point to various features) and lead to a full diagnosis of the muscle that indicate diseased or healthy conditions.</td>
  </tr>
  <tr>
    <td>Task sub-field: Code Generation<br>Image field: Interactive Tools<br>Keywords: Multiple complex visual elements; no domain knowledge.</td>
    <td>Task sub-field: Human and Culture<br>Image sub-field: Diagrams<br>Keywords: Profitability framework; structured diagram; moderate reasoning.</td>
    <td>Task sub-field: Life Science/Medical<br>Image sub-field: Pathology Slides<br>Keywords: Medical diagnosis; pathological analysis; fiber size variation; signs of necrosis and infiltration; specialized knowledge.</td>
  </tr>
</table>

Figure 1: Examples of ProBench with varying lengths. We show the task and image fields in the header of each sample. Due to space limitations, more diverse and longer samples are provided in the supplementary material.

Abstract

Solving expert-level multimodal tasks is a key milestone in general intelligence. As the capabilities of multimodal large language models (MLLMs) continue to evolve, evaluation of frontier multimodal intelligence becomes necessary yet challenging. In this work, we introduce ProBench, a benchmark of open-ended user queries encapsulating professional expertise and advanced reasoning. ProBench consists of 4,000 high-quality samples independently collected from professionals based on their productivity demands. It spans across 10 fields and 56 sub-fields, including science, arts, humanities, coding, mathematics, and creative writing. Experimentally, we evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal that although the best open-source models rival the proprietary ones, they all face significant challenges in visual perception, textual understanding, domain knowledge, and advanced reasoning.

1 Introduction

Solving expert-level multimodal tasks with multimodal large language models (MLLMs) represents an important milestone toward achieving human-level general intelligence. However, these tasks require MLLMs to possess strong user query understanding, domain-specific knowledge, and advanced reasoning abilities. Ensuring their reliability before deployment necessitates rigorous evaluation. To address it, we introduce ProBench, a challenging and automatic evaluation benchmark leveraging MLLM-as-a-Judge. ProBench consists of 4,000 queries from professional users, covering diverse productivity demands to assess MLLM capabilities in open-ended scenarios (Fig. 1).

One common benchmark to evaluating MLLM performance with expert knowledge is MMMU (Yue et al., 2024a). While effective for automatic evaluation using predefined answers, these closed-ended visual question answering benchmarks fail to capture MLLM capabilities in open-ended user interactions. Specifically, they do not adequately assess MLLM ability to follow user instructions or align with human preferences, both of which are fundamental for real-world applications (Lu et al., 2024; Luo et al., 2024; Chen et al., 2024b). Similar limitations apply to other benchmarks, such as MMMU-pro (Yue et al., 2024b), MMBench (Liu et al., 2025), and others (Lu et al., 2023; Masry et al., 2022; Singh et al., 2019; Wu et al., 2024).

Alternatively, MLLM-as-a-Judge is employed to automatically evaluate model performance in open-ended scenarios. However, existing benchmarks fail to rigorously assess MLLMs on expert-level professional tasks. Some (Chen et al., 2024b) are artificially constructed by a small group of experts, limiting their ability to reflect real-world user interactions. The remaining benchmarks (Luo et al., 2024; Lu et al., 2024), such as WildVision, are
Figure 2: Comparison with WildVision (Lu et al., 2024) on challenge levels of (a) text, (b) image, and (c) reasoning for user instruction queries. To ensure a fair comparison, we follow WildVision by selecting the top 500 highest-quality queries from the single-round conversations.

mostly set in general chat environments and require much less domain knowledge to solve.

To fill this gap, in this paper, we aim to design an open-ended benchmark that requires expert-level knowledge for multimodal tasks. Our ProBench is created from high-quality interactions within 100K real-world, professionally crowdsourced multimodal conversations for productivity scenarios. Specifically, samples are collected by encouraging users to ask questions related to their daily professional work, which usually require significant expert-level knowledge. This distinction sets our benchmark apart from prior works like WildVision (Lu et al., 2024) (Fig. 2). For a comprehensive evaluation, ProBench includes three tracks: single-round, multi-round, and multi-linguistic conversations. They respectively span 10 task fields and 56 sub-fields, support 17 languages, and support conversations with up to 13 conversation turns. An overview of ProBench is presented in Fig. 3.

Leveraging MLLM-as-a-Judge (e.g., gpt-40), we assess 24 leading MLLMs on ProBench. Our evaluation reveals several key limitations in state-of-the-art MLLMs: i) current MLLMs struggle in visual perception, textual understanding, domain knowledge, and advanced reasoning, suffering from tasks like mathematics and planning; ii) multi-linguistic understanding and long-context reasoning during multi-round interaction remain challenging for most existing MLLMs. Our main contributions are summarized as follows:

• we introduce ProBench, an open-ended multimodal benchmark tailored for professional work scenarios requiring expert-level knowledge, featuring 4,000 samples across 10 task fields over 56 sub-fields. The benchmark also features multi-round conversations up to 13 turns and multi-linguistic tracks in 17 languages;
• we design an automatic pairwise evaluation pipeline using MLLM-as-a-Judge, achieving 79.9% agreement with human experts. The evaluation is robust to different comparison baseline and judge model choices. We also provide a distilled version of Llama-vision to support cost-effective local evaluations;
• we conduct comprehensive evaluations using 24 leading MLLMs, showing that ProBench presents significant challenges for existing MLLMs, in visual perception, advanced reasoning, and domain knowledge. This signifies the need for more advanced multimodal models for high-value practical scenarios.

Figure 3: ProBench overview. Distributions of (a) task fields on the single-round track, (b) languages on the multi-linguistic track, and (c) conversation rounds on the multi-round tracks.

2 ProBench

Preliminary. The ProBench dynamically ranks MLLMs by employing the ELO rating system, implemented through statistical modeling based on direct pairwise model comparisons. In the following, we provide an overview. For further details, please refer to (Elo, 1966; Hunter, 2004). Given \( N \) MLLMs, an online ELO rating system compares model \( i \) with rating \( r_i \) and model \( j \) with rating \( r_j \) using the probability \( P(y_{i,j} = 1) \). Here, \( y_{i,j} \) denotes the binary outcome, where \( y_{i,j} = 1 \) indicates that model \( i \) wins, and \( y_{i,j} = 0 \) indicates that model \( j \) wins. The probability is calculated by

\[
P(y_{i,j} = 1) = \frac{1}{1 + 10^{(r_i - r_j)/\alpha}},
\]

where \( \alpha \) is a hyperparameter that serves as a scaling factor, typically set to \( \alpha = 400 \). The ELO rating is
Figure 4: Framework of ProBench. Starting with 100K crowdsourced conversations, we identify high-quality user queries to curate single-round, multi-linguistic, and multi-round tracks. Using MLLM-as-a-Judge, we benchmark and rank 24 state-of-the-art MLLMs with ELO ratings. To ensure fairness, the ELO ratings are de-biased to remove confounder effects (e.g., MLLM response formats), resulting in the final ProBench leaderboard. Icons in the figure are sourced from (Freepik et al., 2025).

dynamically updated after each model comparison. Taking model \( i \) as an example, the rating is updated according to the following rule:

\[
r_i^{\text{upt}} = r_i + K \times (s_{i,j} - P(y_{i,j} = 1)) .
\]

Similarly, \( K \) is a constant determining the magnitude of rating adjustments, commonly set to \( K = 32 \). The term \( s_{i,j} \) is a scalar representing the actual outcome: 0 for a loss, 0.5 for a tie, and 1 for a win. This updating rule encourages that a higher-rated model gains fewer points for a win, and loses more points for a defeat, while a lower-rated model experiences the opposite effect.

However, when using MLLM-as-a-Judge, the comparison results can be sensitive to model presentation order and confounded by response style variations (Li et al., 2024c). To address these challenges, the ProBench incorporates the Bradley-Terry model (Hunter, 2004) as an additional layer atop the ELO system. For \( N \) MLLMs and \( M \) pairwise comparisons, each round \( 1 \leq m \leq M \) compares model \( i \) and model \( j \). We have \( \mathbf{X}_m^{\text{win}} \in \mathbb{R}^N \) to indicate which model is presented first\footnote{This bias can be easily mitigated by evaluating twice while swapping the comparison order.}, while \( \mathbf{X}_m^{\text{sty}} \in \mathbb{R}^S \) captures \( S \) stylistic differences between the outputs of models \( i \) and \( j \) (\emph{e.g.}, word counts, and use of markdown). The Bradley-Terry model then refines the rating of model \( i \) as

\[
r_i^{\text{ref}} = C + K \times \hat{\beta}_i ,
\]
\[
\hat{\beta}, \hat{\gamma} = \arg\min_{\beta, \gamma} \sum_{m,i,j} \ell_{\text{bce}}(\beta^\top \mathbf{X}_m^{\text{win}} + \gamma^\top \mathbf{X}_m^{\text{sty}}, s_{i,j}) ,
\]

where \( \ell_{\text{bce}}(\cdot, \cdot) \) is the binary cross-entropy loss, \( C \) is a baseline rating constant, \( \beta \in \mathbb{R}^N \) and \( \gamma \in \mathbb{R}^S \) are respectively known as the model strength and style coefficients, and \( \hat{\beta}_i \) is a scaler indicating strength of model \( i \). This refinement known as style control in the literature (Li et al.) compensates for stylistic biases, ensuring a fair model performance evaluation.

**Overview.** Our paper aims to establish a comprehensive and challenging benchmark for evaluating MLLMs. The resulting ProBench is built on two primary components: i) curating high-quality conversations from a crowdsourced dataset, categorized into single-round, multi-linguistic, and multi-round tracks; ii) employing MLLM-as-a-Judge to compare and rank MLLMs. In total, 3000, 500, and 500 conversations are selected for the single-round, multi-linguistic, and multi-round tracks, respectively, from an initial pool of 100K crowdsourced user-MLLM conversations. An overview is presented in Fig. 4.

2.1 Benchmark establishment

The benchmark is curated based on three guiding principles: i) diversity, selected user instruction queries target to avoid redundancies while extensively covering MLLM-based tasks; ii) MLLM-driven, the chosen queries of conversations are tailored to evaluate the unique capabilities of MLLMs in the multimodal domain; iii) coherence, the benchmark enables targeted evaluations for specific MLLM tasks, rather than providing undifferentiated evaluations. We first describe the common steps involved in curating the three tracks, followed by a discussion of the track-specific methodologies.

**Common step.** We filter out short user instruction queries that contain excessive stop words, and apply MinHash-based text deduplication (Lee et al.,
2021) to retain a pool of non-redundant queries. To address potential redundancy or irrelevance between the instructions and images within a user query, we perform image-instruction deduplication. This step removes queries that can be sufficiently answered using only the textual instructions, leveraging an MLLM-based filter.

Single-round track. A language detector is employed to filter out non-English user instruction queries. Starting with a pool of MLLM task and sub-task fields derived from (Chen et al., 2024b), we use an MLLM-based annotator to assign user instruction queries to existing fields or propose new ones where necessary. Additionally, the annotator assesses the challenge level of each query. To ensure diversity, domain balancing is performed, and overrepresented task fields are downsampled, resulting in 3000 user instruction queries.

Multi-linguistic track. User instruction queries are categorized by their languages, excluding all English-based conversations. Based on frequency, the queries are grouped into Portuguese (PT), French (FR), Spanish (ES), German (DE), and an “Other” category (e.g., Chinese, Vietnamese, and more). An MLLM-based annotator is then used to assess the challenges of the queries, with the 100 most difficult queries retained for each group.

Multi-round track. Similar to the single-round track, we focus on user instruction queries in English for this track. Multi-round conversations are required to feature interconnected queries across rounds, demonstrating a progressive nature. To achieve this, we identify the reasoning challenges and interdependencies between queries within the conversations, applying an MLLM annotator. Ultimately, the 100 most challenging independent queries and 400 interconnected multi-round user instruction queries are preserved.

Detailed prompts used for the above steps are provided in the supplementary material. With the ProBench, we are readily to assess and rank the MLLMs.

2.2 MLLM-as-a-Judge and ranking

We evaluate MLLM performance in addressing user instruction queries using a 5-point Likert scale (Likert, 1932), by conducting pairwise comparisons against a baseline model (e.g., GPT-4o). While evaluations by domain-specific human experts are considered as the gold standard, they are resource-intensive, time-consuming, and challenging to scale for large-scale benchmarks. As an alternative, we employ MLLM-as-a-Judge as an approximation of human expertise (Li et al., 2024c; Zheng et al., 2023; Chen et al., 2024a). The MLLM-as-a-Judge is guided by the following principles.

• Correctness: ensures the accuracy of information, absence of factual errors, and alignments with known and visual knowledge. (For the multi-linguistic track, response language consistency is emphasized).
• Helpfulness: provides clear, practical, and actionable guidance to address the user instruction query.
• Relevance: focuses on the prompt requirements, avoiding extraneous or tangential information.
• Conciseness: avoids unnecessary verbosity while maintaining clarity and direct language.
• Completeness: covers all essential aspects of the user instruction query, providing sufficient information to address it.

Details of the prompts used to guide MLLM-as-a-Judge are provided in the supplementary material. Subsequently, we apply the ELO rating system, as described in the preliminary section, to compute the de-biased ratings of each MLLM. These ratings are used for leaderboard comparisons, ensuring a fair and consistent evaluation across models.

3 Experiment

3.1 Experimental setup

Implementation detail. All MLLMs are benchmarked using the v1lm (Kwon et al., 2023) and Hugging Face (Wolf, 2019) codebases, with greedy sampling employed for response generation. For MLLMs with limited context lengths (e.g., a 4096 token context in Molmo-7B-D-0924), sliding window generation is applied to handle longer inputs. Our MLLM judge utilizes gpt-4o-2024-08-06 with greedy sampling for consistent and reproducible evaluation. For pairwise comparisons in Elo rating calculations, we set gpt-4o-2024-05-13 as the baseline, evaluate each model twice by swapping the presentation order for each user query, and de-bias the ELO ratings by following the methodology of (Li et al., 2024c).

MLLM. We evaluate 24 leading MLLMs: gpt-4o-mini-2024-07-18 (Hurst et al., 2024), gpt-4o-2024-08-06 (Hurst et al., 2024), gpt-4o-2024-05-13 (Hurst et al., 2024), claude-3-5-sonnet-20241022 (Anthropic, 2024), gemini-1.5-pro-002 (Team et al., 2023), gemini-1.5-flash-002 (Team et al., 2023),
Table 1: Comparisons of state-of-the-art MLLMs on the single-round track are presented using the following abbreviations: Sci. (Science), Cd. (Coding), CW. (Creative Writing), IE. (Information Extraction), Perc. (Perception), Knowl. (Knowledge), Arts (Arts), Plan. (Planning), Math (Mathematics), and Mt. (Metrics). We provide ELO ratings for each task, followed by an overview that includes the average number of output tokens (#Token), 95% confidence interval (95% CI), win rate (WR), and overall ELO rating. The MLLMs are sorted by the overall ELO rating in each group of model size.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="10">Task-Specific ELO Ratings</th>
    <th rowspan="2">#Token</th>
    <th rowspan="2">95% CI</th>
    <th rowspan="2">WR</th>
    <th rowspan="2">Elo</th>
  </tr>
  <tr>
    <th>Sci.</th>
    <th>Cd.</th>
    <th>CW.</th>
    <th>IE.</th>
    <th>Perc.</th>
    <th>Knowl.</th>
    <th>Arts.</th>
    <th>Plan.</th>
    <th>Math.</th>
    <th>Mt.</th>
  </tr>
  <tr><td colspan="14"><b>Proprietary MLLMs</b></td></tr>
  <tr><td>claude-3-5-sonnet-20241022</td><td>1228</td><td>1252</td><td>1259</td><td>1211</td><td>1213</td><td>1272</td><td>1236</td><td>1192</td><td>1197</td><td>1251</td><td>405</td><td>(-7, 8)</td><td>65.84</td><td>1228</td></tr>
  <tr><td>gemini-1.5-pro-002</td><td>1151</td><td>1145</td><td>1105</td><td>1100</td><td>1110</td><td>1067</td><td>1107</td><td>1095</td><td>1134</td><td>1147</td><td>501</td><td>(-8, 10)</td><td>50.58</td><td>1118</td></tr>
  <tr><td>gpt-4o-2024-05-13</td><td>1114</td><td>1114</td><td>1114</td><td>1114</td><td>1114</td><td>1114</td><td>1114</td><td>1114</td><td>1114</td><td>1114</td><td>491</td><td>(-0, 0)</td><td>50.00</td><td>1114</td></tr>
  <tr><td>gpt-4o-2024-05-18</td><td>1049</td><td>1074</td><td>1065</td><td>1094</td><td>1096</td><td>1101</td><td>1130</td><td>1102</td><td>1037</td><td>1159</td><td>526</td><td>(-4, 10)</td><td>47.12</td><td>1094</td></tr>
  <tr><td>gpt-4o-2024-08-06</td><td>1096</td><td>1112</td><td>1050</td><td>1097</td><td>995</td><td>1080</td><td>1032</td><td>1058</td><td>1175</td><td>1015</td><td>374</td><td>(-7, -7)</td><td>44.98</td><td>1079</td></tr>
  <tr><td>gemini-1.5-flash-002</td><td>1025</td><td>877</td><td>1092</td><td>1007</td><td>1022</td><td>1011</td><td>993</td><td>946</td><td>1035</td><td>1087</td><td>493</td><td>(-8, 9)</td><td>35.33</td><td>1009</td></tr>
  <tr><td colspan="14"><b>7B+ Open-source MLLMs</b></td></tr>
  <tr><td>Pixtral-Large-Instruct-2411</td><td>1248</td><td>1230</td><td>1194</td><td>1280</td><td>1242</td><td>1242</td><td>1250</td><td>1245</td><td>1221</td><td>1175</td><td>1266</td><td>715</td><td>(-8, 8)</td><td>65.97</td><td>1229</td></tr>
  <tr><td>InternVL2_5-78B</td><td>728</td><td>1083</td><td>1018</td><td>1051</td><td>1091</td><td>1031</td><td>1084</td><td>1042</td><td>1073</td><td>1065</td><td>1023</td><td>558</td><td>(-7, 10)</td><td>42.85</td><td>1064</td></tr>
  <tr><td>Qwen2-VL-72B-Instruct</td><td>728</td><td>1009</td><td>914</td><td>965</td><td>991</td><td>986</td><td>960</td><td>962</td><td>921</td><td>998</td><td>970</td><td>557</td><td>(-9, 9)</td><td>31.37</td><td>978</td></tr>
  <tr><td>Molmo-7B-0924</td><td>728</td><td>828</td><td>733</td><td>953</td><td>859</td><td>903</td><td>881</td><td>862</td><td>817</td><td>871</td><td>852</td><td>301</td><td>(-12, 8)</td><td>18.46</td><td>856</td></tr>
  <tr><td>NVLM-D-72B</td><td>728</td><td>780</td><td>877</td><td>991</td><td>810</td><td>849</td><td>835</td><td>767</td><td>881</td><td>838</td><td>725</td><td>561</td><td>(-10, 10)</td><td>16.63</td><td>834</td></tr>
  <tr><td>Llama-3.2-90B-Vision-Instruct</td><td>908</td><td>830</td><td>751</td><td>624</td><td>754</td><td>806</td><td>842</td><td>626</td><td>769</td><td>940</td><td>662</td><td>448</td><td>(-11, 10)</td><td>12.89</td><td>782</td></tr>
  <tr><td>llava-onevision-qwen2-72b-ov</td><td>728</td><td>696</td><td>735</td><td>762</td><td>726</td><td>767</td><td>689</td><td>663</td><td>679</td><td>853</td><td>620</td><td>360</td><td>(-11, 12)</td><td>10.09</td><td>734</td></tr>
  <tr><td colspan="14"><b>10B+ Open-source MLLMs</b></td></tr>
  <tr><td>Pixtral-12B-2409</td><td>12B</td><td>1028</td><td>965</td><td>1099</td><td>1031</td><td>1024</td><td>1057</td><td>1047</td><td>1083</td><td>996</td><td>1063</td><td>659</td><td>(-5, 8)</td><td>39.1</td><td>1037</td></tr>
  <tr><td>Aria-Chat</td><td>3.9/25.3B</td><td>990</td><td>982</td><td>985</td><td>937</td><td>998</td><td>1034</td><td>1019</td><td>974</td><td>973</td><td>1016</td><td>675</td><td>(-7, 8)</td><td>32.88</td><td>990</td></tr>
  <tr><td>InternVL2_5-38B</td><td>38B</td><td>1000</td><td>979</td><td>1028</td><td>987</td><td>1021</td><td>904</td><td>932</td><td>1041</td><td>1026</td><td>933</td><td>521</td><td>(-9, 9)</td><td>32.5</td><td>987</td></tr>
  <tr><td>InternVL2_5-26B</td><td>26B</td><td>890</td><td>816</td><td>1008</td><td>894</td><td>944</td><td>876</td><td>864</td><td>984</td><td>896</td><td>490</td><td>(-10, 8)</td><td>22.59</td><td>900</td></tr>
  <tr><td>Llama-3.2-11B-Vision-Instruct</td><td>11B</td><td>671</td><td>541</td><td>681</td><td>702</td><td>766</td><td>761</td><td>624</td><td>524</td><td>744</td><td>614</td><td>531</td><td>(-13, 16)</td><td>7.93</td><td>688</td></tr>
  <tr><td colspan="14"><b>7B+ Open-source MLLMs</b></td></tr>
  <tr><td>InternVL2_5-8B</td><td>8B</td><td>824</td><td>806</td><td>983</td><td>880</td><td>914</td><td>840</td><td>915</td><td>895</td><td>835</td><td>868</td><td>644</td><td>(-11, 8)</td><td>20.45</td><td>878</td></tr>
  <tr><td>Qwen2-VL-7B-Instruct</td><td>7B</td><td>803</td><td>689</td><td>827</td><td>877</td><td>861</td><td>816</td><td>736</td><td>680</td><td>858</td><td>833</td><td>787</td><td>(-9, 10)</td><td>15.40</td><td>818</td></tr>
  <tr><td>MiniCPM-V-2_6</td><td>6B</td><td>644</td><td>599</td><td>671</td><td>659</td><td>812</td><td>676</td><td>673</td><td>667</td><td>656</td><td>681</td><td>646</td><td>(-12, 10)</td><td>7.97</td><td>689</td></tr>
  <tr><td>llava-onevision-qwen2-7b-ov</td><td>7B</td><td>607</td><td>570</td><td>807</td><td>683</td><td>809</td><td>681</td><td>715</td><td>608</td><td>732</td><td>724</td><td>578</td><td>(-13, 10)</td><td>7.93</td><td>688</td></tr>
  <tr><td>Molmo-7B-D-0924</td><td>7B</td><td>536</td><td>304</td><td>720</td><td>631</td><td>638</td><td>655</td><td>681</td><td>531</td><td>614</td><td>603</td><td>310</td><td>(-14, 12)</td><td>5.41</td><td>617</td></tr>
  <tr><td>Molmo-7B-G-0924</td><td>7B</td><td>457</td><td>154</td><td>653</td><td>483</td><td>681</td><td>599</td><td>606</td><td>580</td><td>428</td><td>528</td><td>296</td><td>(-18, 19)</td><td>5.54</td><td>540</td></tr>
</table>

Aria-Chat (Li et al., 2024b), InternVL2_5-8B (Wang et al., 2024b), InternVL2_5-26B (Wang et al., 2024b), InternVL2_5-38B (Wang et al., 2024b), InternVL2_5-78B (Wang et al., 2024b), Pixtral-12B-2409 (Agrawal et al., 2024), Pixtral-Large-Instruct-2411 (Agrawal et al., 2024), Qwen2-VL-7B-Instruct (Wang et al., 2024a), Qwen2-VL-72B-Instruct (Wang et al., 2024a), MiniCPM-V-2_6 (Yao et al., 2024), Llama-3.2-11B-Vision-Instruct (Dubey et al., 2024), Llama-3.2-90B-Vision-Instruct (Dubey et al., 2024), Molmo-7B-O-0924 (Deitke et al., 2024), Molmo-7B-D-0924 (Deitke et al., 2024), NVLM-D-72B (Dai et al., 2024), llava-onevision-qwen2-7b-ov (Li et al., 2024a), and llava-onevision-qwen2-72b-ov (Li et al., 2024a).

3.2 Experimental result

Tab. 1 and Tab. 2 present the evaluation results. Our key observations are summarized into the following five folds: i) best open-source models rival the best proprietary MLLMs. claude-3-5-sonnet-20241022 and Pixtral-Large-Instruct-2411 respectively belonging to proprietary and open-source MLLMs consistently achieve leading ELO scores across all three tracks. Both models significantly outperform the baseline gpt-4o-2024-05-13; ii) training recipe outweighs model size. While scaling parameters can generally enhance performance, the performance of MLLMs can be size-agnostic, with greater emphasis placed on the training recipe (i.e., optimization strategy and training data quality). For example, Pixtral with 12B parameters and Aria-Chat with 3.9B activated parameters (out of a total of 25.3B) per token consistently demonstrate first-tier performance; iii) reasoning tasks remain the hardest. On the single-round track, most MLLMs generally perform well on writing-based tasks (e.g., creative writing). However, their performance on logic-intensive tasks is notably poor, similar to findings in prior LLM studies (Ahn et al., 2024; Quan et al., 2025). The two tasks separately exhibit the lowest Spearman correlation with overall ELO ratings and receive the lowest scores among task fields. Similarly, among all open-source models, performance also suffers significantly in planning tasks, which have the lowest average score (excluding coding); iv) multi-linguistic tasks challenge MLLMs. MLLMs face significant challenges in multi-linguistic tasks, with 11 out of 24 MLLMs showing an overall ELO decrease compared to their performance on the single-round track. Notably,
Table 2: Comparisons of state-of-the-art MLLMs on the multi-linguistic and multi-round tracks. We provide an overview that shows the average number of output tokens (#Token), 95% confidence interval (95% CI), win rate (WR), and overall ELO rating for each of the track. Refer to our supplementary material for comparison details on different languages and rounds. The MLLMs are sorted by the overall ELO rating on the multi-linguistic track in each group of model size.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Overview on multi-linguistic track</th>
    <th colspan="4">Overview on multi-round track</th>
  </tr>
  <tr>
    <th>#Token</th>
    <th>95% CI</th>
    <th>WR</th>
    <th>Elo</th>
    <th>#Token</th>
    <th>95% CI</th>
    <th>WR</th>
    <th>Elo</th>
  </tr>
  <tr>
    <td colspan="9"><b>Proprietary MLLMs</b></td>
  </tr>
  <tr>
    <td>claude-3-5-sonnet-20241022</td>
    <td>485</td>
    <td>(-21, 29)</td>
    <td>74.58</td>
    <td>1301</td>
    <td>1477</td>
    <td>(-20, 18)</td>
    <td>70.82</td>
    <td>1268</td>
  </tr>
  <tr>
    <td>gpt-4o-2024-05-13</td>
    <td>585</td>
    <td>(0, 0)</td>
    <td>50.00</td>
    <td>1114</td>
    <td>1563</td>
    <td>(0, 0)</td>
    <td>50.00</td>
    <td>1114</td>
  </tr>
  <tr>
    <td>gemin-1.5-pro-002</td>
    <td>629</td>
    <td>(-20, 20)</td>
    <td>59.11</td>
    <td>1178</td>
    <td>1425</td>
    <td>(-26, 19)</td>
    <td>53.88</td>
    <td>1141</td>
  </tr>
  <tr>
    <td>gpt-4o-2024-08-06</td>
    <td>480</td>
    <td>(-17, 26)</td>
    <td>60.35</td>
    <td>1187</td>
    <td>1052</td>
    <td>(-22, 18)</td>
    <td>45.41</td>
    <td>1082</td>
  </tr>
  <tr>
    <td>gpt-4o-mini-2024-07-18</td>
    <td>657</td>
    <td>(-21, 16)</td>
    <td>45.84</td>
    <td>1085</td>
    <td>1749</td>
    <td>(-17, 24)</td>
    <td>55.16</td>
    <td>1130</td>
  </tr>
  <tr>
    <td>gemin-1.5-flash-002</td>
    <td>567</td>
    <td>(-25, 19)</td>
    <td>28.47</td>
    <td>954</td>
    <td>1388</td>
    <td>(-16, 19)</td>
    <td>38.14</td>
    <td>1030</td>
  </tr>
  <tr>
    <td colspan="9"><b>7B+ Open-source MLLMs</b></td>
  </tr>
  <tr>
    <td>Pixtral-Large-Instruct-2411</td>
    <td>124B</td>
    <td>966</td>
    <td>(-23, 22)</td>
    <td>73.81</td>
    <td>1294</td>
    <td>2593</td>
    <td>(-23, 19)</td>
    <td>69.73</td>
    <td>1259</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B-Instruct</td>
    <td>7B</td>
    <td>841</td>
    <td>(-18, 21)</td>
    <td>47.56</td>
    <td>1097</td>
    <td>1608</td>
    <td>(-21, 19)</td>
    <td>32.24</td>
    <td>985</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B8</td>
    <td>7B</td>
    <td>841</td>
    <td>(-14, 20)</td>
    <td>47.71</td>
    <td>1063</td>
    <td>2015</td>
    <td>(-21, 15)</td>
    <td>44.84</td>
    <td>1078</td>
  </tr>
  <tr>
    <td>NYLM-1D-72B</td>
    <td>72B</td>
    <td>729</td>
    <td>(-97, -73)</td>
    <td>21.99</td>
    <td>894</td>
    <td>1371</td>
    <td>(-35, 33)</td>
    <td>8.49</td>
    <td>704</td>
  </tr>
  <tr>
    <td>Llama-3-2.90B-Vision-Instruct</td>
    <td>90B</td>
    <td>968</td>
    <td>(-20, 21)</td>
    <td>20.92</td>
    <td>883</td>
    <td>1350</td>
    <td>(-36, 24)</td>
    <td>9.88</td>
    <td>730</td>
  </tr>
  <tr>
    <td>Molmo-72B-0924</td>
    <td>72B</td>
    <td>426</td>
    <td>(-27, 19)</td>
    <td>18.90</td>
    <td>861</td>
    <td>967</td>
    <td>(-28, 25)</td>
    <td>18.64</td>
    <td>858</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-7b-ov</td>
    <td>72B</td>
    <td>534</td>
    <td>(-27, 24)</td>
    <td>11.95</td>
    <td>767</td>
    <td>1176</td>
    <td>(-31, 26)</td>
    <td>10.30</td>
    <td>738</td>
  </tr>
  <tr>
    <td colspan="9"><b>10B+ Open-source MLLMs</b></td>
  </tr>
  <tr>
    <td>InternVL2_5-38B</td>
    <td>38B</td>
    <td>868</td>
    <td>(-20, 18)</td>
    <td>43.98</td>
    <td>1072</td>
    <td>1734</td>
    <td>(-18, 21)</td>
    <td>34.68</td>
    <td>1004</td>
  </tr>
  <tr>
    <td>Pixtral-12B-2409</td>
    <td>12B</td>
    <td>1199</td>
    <td>(-14, 22)</td>
    <td>35.73</td>
    <td>1012</td>
    <td>2264</td>
    <td>(-19, 20)</td>
    <td>40.48</td>
    <td>1047</td>
  </tr>
  <tr>
    <td>Aria-Chat</td>
    <td>3.9/25.3B</td>
    <td>1014</td>
    <td>(-23, 17)</td>
    <td>35.33</td>
    <td>1009</td>
    <td>2321</td>
    <td>(-27, 12)</td>
    <td>23.92</td>
    <td>913</td>
  </tr>
  <tr>
    <td>InternVL2_5-26B</td>
    <td>26B</td>
    <td>814</td>
    <td>(-28, 19)</td>
    <td>17.70</td>
    <td>847</td>
    <td>554</td>
    <td>(-27, 28)</td>
    <td>15.77</td>
    <td>823</td>
  </tr>
  <tr>
    <td>Llama-3-2.11B-Vision-Instruct</td>
    <td>11B</td>
    <td>2027</td>
    <td>(-29, 21)</td>
    <td>8.40</td>
    <td>699</td>
    <td>2094</td>
    <td>(-38, 32)</td>
    <td>6.03</td>
    <td>637</td>
  </tr>
  <tr>
    <td colspan="9"><b>7B+ Open-source MLLMs</b></td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B-Instruct</td>
    <td>7B</td>
    <td>1216</td>
    <td>(-24, 22)</td>
    <td>12.25</td>
    <td>772</td>
    <td>2004</td>
    <td>(-34, 25)</td>
    <td>9.48</td>
    <td>722</td>
  </tr>
  <tr>
    <td>InternVL2_5-8B</td>
    <td>8B</td>
    <td>1021</td>
    <td>(-22, 20)</td>
    <td>11.95</td>
    <td>767</td>
    <td>1835</td>
    <td>(-25, 22)</td>
    <td>11.77</td>
    <td>764</td>
  </tr>
  <tr>
    <td>MiniCPM-V-2_6</td>
    <td>8B</td>
    <td>890</td>
    <td>(-36, 35)</td>
    <td>4.44</td>
    <td>581</td>
    <td>1861</td>
    <td>(-33, 37)</td>
    <td>5.35</td>
    <td>615</td>
  </tr>
  <tr>
    <td>Molmo-7B-D-0924</td>
    <td>7B</td>
    <td>406</td>
    <td>(-52, 33)</td>
    <td>4.32</td>
    <td>576</td>
    <td>923</td>
    <td>(-34, 26)</td>
    <td>5.04</td>
    <td>604</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-7b-ov</td>
    <td>7B</td>
    <td>686</td>
    <td>(-68, 37)</td>
    <td>3.07</td>
    <td>514</td>
    <td>1743</td>
    <td>(-30, 30)</td>
    <td>6.58</td>
    <td>653</td>
  </tr>
  <tr>
    <td>Molmo-7B-O-0924</td>
    <td>7B</td>
    <td>512</td>
    <td>(-73, 51)</td>
    <td>1.95</td>
    <td>433</td>
    <td>925</td>
    <td>(-49, 37)</td>
    <td>3.43</td>
    <td>534</td>
  </tr>
</table>

llava-onevision-qwen2-7b-ov experienced the most substantial decline; v) <b>multi-round evaluation enhances model performance separability.</b> Multi-round tasks usually demand long-context reasoning across turns, amplifying performance gaps among MLLMs. MLLMs that underperform in single-round tasks exhibit significantly lower ELO scores. This trend is particularly evident in open-source MLLMs with 7B+ and 10B+ parameters (excluding Pixtral-12B-2409).

3.3 Ablation and discussion

<b>Performance declining with difficulty.</b> We evaluate the ELO rating variances of MLLMs by categorizing user queries into easy and hard groups. The results are presented in Fig. 5. Existing MLLMs tend to exhibit a noticeable performance decline compared to the baseline gpt-4o-2024-05-13 as the reasoning challenge level increased from easy to hard, while MLLM with poor performance typically deteriorates further on the harder queries. This observation aligns with human intuition that more challenging tasks inherently provide better separability when evaluating the MLLM performance, highlighting the limitations of most MLLMs in effectively handling complex user queries.

<b>Error analysis.</b> We analyze scenarios in which the state-of-the-art MLLM underperforms relative to the baseline. Fig. 6 (a) illustrates the shortcomings of the MLLM compared to the baseline across five evaluation aspects, highlighting completeness and correctness as the primary issues. Fig. 6 (b) categorizes the error types in the MLLM losses relative to the baseline. Overall, the analysis underscores the need of state-of-the-art MLLM to improve their visual perception, textual understanding, domain knowledge, and reasoning capability.

<b>Robustness of ProBench.</b> We study the setting of our evaluation protocol on the 500 most challenging queries from the single-round track. Specifically, Fig. 7 considers two set of experiments: i) comparisons of using three top-performing MLLM as the judge (i.e., gpt-4o-2024-08-06, claude-3-5-sonnet-20241022, and Pixtral-Large-Instruct-2411); ii) explorations of three baseline models (i.e., gpt-4o-2024-05-13, claude-3-5-sonnet-20241022, and Pixtral-12B-2409) in comparisons, representing different model scales. The results reveal a high degree of agreement within our evaluation process, with an average Spearman correlation coefficient of 0.979 among the different MLLM judges and 0.983 among the baseline models, highlighting our robustness and consistency.
Figure 5: Ablation study of reasoning challenge. We show the ELO ratings of MLLMs on two levels: easy and hard.

<table>
  <tr>
    <th></th>
    <th>Compl.</th>
    <th>Concis.</th>
    <th>Corre.</th>
    <th>Helpf.</th>
    <th>Relev.</th>
    <th>Text.</th>
    <th>Perc.</th>
    <th>Reas.</th>
    <th>Know.</th>
    <th>Reje.</th>
  </tr>
  <tr>
    <td>claude-3.5-somni-20241022</td>
    <td>53.0%</td>
    <td>7.73%</td>
    <td>24.01%</td>
    <td>12.40%</td>
    <td>2.80%</td>
    <td>21.25%</td>
    <td>33.97%</td>
    <td>32.95%</td>
    <td>3.82%</td>
    <td>8.02%</td>
  </tr>
  <tr>
    <td>geminai-1.5-pro-002</td>
    <td>37.91%</td>
    <td>8.28%</td>
    <td>30.28%</td>
    <td>19.36%</td>
    <td>4.17%</td>
    <td>24.89%</td>
    <td>36.50%</td>
    <td>30.80%</td>
    <td>3.91%</td>
    <td>3.91%</td>
  </tr>
  <tr>
    <td>gpt-4o-2024-08-06</td>
    <td>38.0%</td>
    <td>8.8%</td>
    <td>19.96%</td>
    <td>17.2%</td>
    <td>1.64%</td>
    <td>20.9%</td>
    <td>40.85%</td>
    <td>29.71%</td>
    <td>6.24%</td>
    <td>2.97%</td>
  </tr>
  <tr>
    <td>gpt-4o-mini-2024-07-18</td>
    <td>29.88%</td>
    <td>8.82%</td>
    <td>12.18%</td>
    <td>12.18%</td>
    <td>3.81%</td>
    <td>12.8%</td>
    <td>20.96%</td>
    <td>15.34%</td>
    <td>3.84%</td>
    <td>2.00%</td>
  </tr>
  <tr>
    <td>geminai-1.5-flash-002</td>
    <td>38.03%</td>
    <td>6.38%</td>
    <td>27.64%</td>
    <td>23.42%</td>
    <td>4.53%</td>
    <td>28.85%</td>
    <td>25.16%</td>
    <td>29.06%</td>
    <td>6.92%</td>
    <td>10.31%</td>
  </tr>
  <tr>
    <td>Pixtral-Large-Instruct-2411</td>
    <td>21.73%</td>
    <td>8.55%</td>
    <td>57.27%</td>
    <td>9.15%</td>
    <td>4.40%</td>
    <td>20.28%</td>
    <td>39.59%</td>
    <td>36.48%</td>
    <td>3.65%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>InternVL2_5-78B</td>
    <td>46.58%</td>
    <td>5.79%</td>
    <td>28.37%</td>
    <td>16.95%</td>
    <td>2.32%</td>
    <td>23.39%</td>
    <td>37.51%</td>
    <td>34.51%</td>
    <td>4.06%</td>
    <td>0.53%</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B-Instruct</td>
    <td>35.35%</td>
    <td>4.75%</td>
    <td>43.24%</td>
    <td>13.84%</td>
    <td>2.82%</td>
    <td>21.85%</td>
    <td>34.33%</td>
    <td>37.21%</td>
    <td>5.88%</td>
    <td>0.72%</td>
  </tr>
  <tr>
    <td>Molmo-72B-0924</td>
    <td>47.77%</td>
    <td>3.05%</td>
    <td>37.09%</td>
    <td>10.35%</td>
    <td>1.74%</td>
    <td>21.95%</td>
    <td>38.24%</td>
    <td>32.93%</td>
    <td>4.58%</td>
    <td>2.29%</td>
  </tr>
  <tr>
    <td>NVLM-D-72B</td>
    <td>42.97%</td>
    <td>5.19%</td>
    <td>34.16%</td>
    <td>14.11%</td>
    <td>3.57%</td>
    <td>29.86%</td>
    <td>31.35%</td>
    <td>30.88%</td>
    <td>7.19%</td>
    <td>0.62%</td>
  </tr>
  <tr>
    <td>Llama-3.2-11B-Vision-Instruct</td>
    <td>35.71%</td>
    <td>3.44%</td>
    <td>41.82%</td>
    <td>13.42%</td>
    <td>5.27%</td>
    <td>21.46%</td>
    <td>33.97%</td>
    <td>26.48%</td>
    <td>5.32%</td>
    <td>12.88%</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-72b-ov</td>
    <td>38.03%</td>
    <td>3.80%</td>
    <td>38.18%</td>
    <td>13.11%</td>
    <td>7.88%</td>
    <td>24.06%</td>
    <td>32.05%</td>
    <td>32.05%</td>
    <td>4.19%</td>
    <td>4.19%</td>
  </tr>
  <tr>
    <td>Pixtral-12B-2409</td>
    <td>25.85%</td>
    <td>7.16%</td>
    <td>31.48%</td>
    <td>10.90%</td>
    <td>4.65%</td>
    <td>23.39%</td>
    <td>37.51%</td>
    <td>34.51%</td>
    <td>4.06%</td>
    <td>0.53%</td>
  </tr>
  <tr>
    <td>InternVL2_5-38B</td>
    <td>40.97%</td>
    <td>4.49%</td>
    <td>28.93%</td>
    <td>14.81%</td>
    <td>1.80%</td>
    <td>24.28%</td>
    <td>39.48%</td>
    <td>29.14%</td>
    <td>6.10%</td>
    <td>1.00%</td>
  </tr>
  <tr>
    <td>Aria-Chat</td>
    <td>32.22%</td>
    <td>6.21%</td>
    <td>48.15%</td>
    <td>9.77%</td>
    <td>3.66%</td>
    <td>21.08%</td>
    <td>40.46%</td>
    <td>32.57%</td>
    <td>5.60%</td>
    <td>0.28%</td>
  </tr>
  <tr>
    <td>InternVL2_5-26B</td>
    <td>42.54%</td>
    <td>3.49%</td>
    <td>38.94%</td>
    <td>12.70%</td>
    <td>2.33%</td>
    <td>22.86%</td>
    <td>32.91%</td>
    <td>34.92%</td>
    <td>8.15%</td>
    <td>1.16%</td>
  </tr>
  <tr>
    <td>Llama-3.2-11B-Vision-Instruct</td>
    <td>31.17%</td>
    <td>3.85%</td>
    <td>49.06%</td>
    <td>11.71%</td>
    <td>4.22%</td>
    <td>21.33%</td>
    <td>31.95%</td>
    <td>34.69%</td>
    <td>5.66%</td>
    <td>6.37%</td>
  </tr>
  <tr>
    <td>InternVL2_5-38B</td>
    <td>34.01%</td>
    <td>6.33%</td>
    <td>42.66%</td>
    <td>13.04%</td>
    <td>3.97%</td>
    <td>20.86%</td>
    <td>38.12%</td>
    <td>33.53%</td>
    <td>6.99%</td>
    <td>0.50%</td>
  </tr>
  <tr>
    <td>Pixtral-12B-2409</td>
    <td>36.06%</td>
    <td>3.13%</td>
    <td>31.39%</td>
    <td>12.21%</td>
    <td>2.76%</td>
    <td>21.86%</td>
    <td>32.76%</td>
    <td>32.76%</td>
    <td>7.96%</td>
    <td>0.30%</td>
  </tr>
  <tr>
    <td>MiniCPM-V-2_6</td>
    <td>40.48%</td>
    <td>3.32%</td>
    <td>41.28%</td>
    <td>12.60%</td>
    <td>3.32%</td>
    <td>24.20%</td>
    <td>33.11%</td>
    <td>32.34%</td>
    <td>7.15%</td>
    <td>1.19%</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-7b-ov</td>
    <td>28.76%</td>
    <td>5.34%</td>
    <td>44.58%</td>
    <td>16.61%</td>
    <td>4.71%</td>
    <td>30.56%</td>
    <td>31.45%</td>
    <td>27.89%</td>
    <td>5.74%</td>
    <td>4.35%</td>
  </tr>
  <tr>
    <td>Molmo-7B-D-0924</td>
    <td>33.44%</td>
    <td>2.27%</td>
    <td>51.73%</td>
    <td>8.77%</td>
    <td>3.79%</td>
    <td>27.06%</td>
    <td>32.03%</td>
    <td>29.16%</td>
    <td>8.89%</td>
    <td>2.87%</td>
  </tr>
  <tr>
    <td>Molmo-7B-O-0924</td>
    <td>36.37%</td>
    <td>1.54%</td>
    <td>46.92%</td>
    <td>12.30%</td>
    <td>2.86%</td>
    <td>29.76%</td>
    <td>30.85%</td>
    <td>29.86%</td>
    <td>7.94%</td>
    <td>1.59%</td>
  </tr>
</table>

(a) (b) (c)

Figure 6: Error analysis. We study cases where MLLM underperforms compared to the baseline. (a) The distribution of losing cases of the MLLM across five evaluation aspects: completeness (Compl.), conciseness (Concis.), correctness (Corre.), helpfulness (Helpf.), and relevance (Relev.). (b) The distribution of error types in losses of the MLLM, categorized into five types: textual understanding error (Text.), visual perceptual error (Perc.), reasoning error (Reas.), lack of domain knowledge error (Know.), and refusal to answer (Reje.). (c) Color bar of the heatmap.

Judge alignment with human expert. To validate the effectiveness of MLLM-as-a-Judge, human annotators are tasked with rating the comparisons using a 5-point Likert scale. Our evaluation protocol achieves an agreement of 79.9% with human expert, indicating a strong ability of MLLM-as-a-Judge to simulate human preferences accurately. These findings demonstrate the viability of ProBench as an automatic, large-scale, and challenging benchmark for evaluating the assistance capabilities of MLLMs in professional productivity scenarios. By effectively aligning with human judgments, ProBench provides a reliable automatic framework for advancing MLLM development and assessment.

Future work and limitation. Although our ProBench has provided valuable insights into the performance and capabilities of MLLMs, several limitations remain that warrant further exploration.

One key limitation is potential bias in the benchmark tasks, which may not fully capture the diversity of real-world productivity scenarios for MLLMs. Future work could focus on expanding the benchmark to include a broader range of challenging tasks, potentially through the data synthesis (e.g., diffusion models and MLLMs), to improve the diversity. By addressing these challenges, ProBench can continue to evolve as a robust and comprehensive tool for advancing the development and evaluation of MLLMs.

3.4 Distilled local evaluator

Considering the high API cost of using gpt-4o-2024-08-06 as the judge, we fine-tune a local evaluator to enable cost-effective and GPU-friendly evaluations for future MLLMs. We use the widely spread Llama-3.2-11B-Vision-Instruct as our backbone model. The Qwen and Pixtral MLLM fam-
Figure 7: Ablation study of MLLM-as-the-Judge. (a-c) Pairwise comparisons of Elo scores for MLLMs evaluated using different MLLM judges. They are gpt-4o-2024-08-06, claude-3-5-sonnet-20241022 (claude-3-5-sonnet), and Pixtral-Large-Instruct-2411 (Pixtral-Large), respectively. (d-f) Comparison of using gpt-4o-2024-05-13, claude-3-5-sonnet-20241022 (claude-3-5-sonnet), and Pixtral-12B-2409 (Pixtral) as baselines. The red line in each plot indicates the best-fit curve for visualization.

ilies are reserved for testing, with the remaining data allocated for training. Our network is trained to distill both the reasoning and decisions of using gpt-4o-2024-08-06 as the judge. The network achieves an average root mean squared error of 32.58 in Elo ratings.

4 Related work

The evolution of MLLM-as-a-Judge is largely inspired by the concept of LLM-as-a-Judge (Li et al., 2024c; Dubois et al., 2024; Zheng et al., 2023), which aims to automatically measure the alignment between MLLMs and human preferences. While pairwise comparison (Li et al., 2024c; Chen et al., 2024a) is considered as most preferred, it suffers from biases introduced by factors such as the presentation order of MLLM outputs, verbosity, and markdown styles. To mitigate these issues, style control has been proposed (Li et al.), using statistical modeling to de-bias these confounding effects, thereby improving the MLLM judges.

Other approaches, such as few-shot judging, have also been explored, but they face challenges such as reliance on the few-shot example selection and increased evaluation costs (Zheng et al., 2023). Existing MLLM-as-a-Judge leaderboards can be specified to (Luo et al., 2024; Lu et al., 2024; Chen et al., 2024a). However, these often focus on a narrow scope of MLLM capability dimensions (Luo et al., 2024; Lu et al., 2024), or rely on artificially posed evaluations by a limited number of human experts (Chen et al., 2024b), making them inadequate for assessing MLLMs on professional tasks. Consequently, they fail to capture the dynamic nature of real-world human and MLLM interactions for a comprehensive assessment of MLLM capabilities. In contrast, this work introduces a challenging benchmark, ProBench, curated from large-scale crowdsourced datasets reflecting real-world professional productivity scenarios. It features three distinct evaluation tracks: single-round, multi-round, and multi-linguistic conversations, across various task fields, offering a robust framework for evaluating MLLM performance in real-world scenarios.

5 Conclusion

This paper introduces the ProBench, which features single-round, multi-round, and multi-linguistic tracks to enable a comprehensive and challenging assessment of the alignment between MLLMs and human preferences across diverse professional productivity demands. By employing MLLM-as-a-Judge, the benchmark evaluates MLLM pairwisely, achieving 79.9% agreement with human expert judgments, and underscoring its reliability. Through benchmarking 24 leading MLLMs, our results reveal significant shortcomings of existing MLLMs, particularly in visual perception and reasoning. Furthermore, models often struggle with multi-linguistic and multi-round tracks, highlighting the challenges of diverse language requirement and complex interactions. It reveals valuable insights for future MLLM developments. We hope it inspires successors.
References

Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Moncault, Saurabh Garg, Theophile Gervet, et al. 2024. Pixtral 12b. arXiv preprint arXiv:2410.07073.

Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157.

AI Anthropic. 2024. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card, 3.

Dongping Chen, Ruoxi Chen, Shilin Zhang, Yino Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024a. Mlm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. arXiv preprint arXiv:2402.04788.

Jiacheng Chen, Tianhao Liang, Sherman Siu, Zhengqing Wang, Kai Wang, Yubo Wang, Yuan-sheng Ni, Wang Zhu, Ziyian Jiang, Bohan Lyu, et al. 2024b. Mega-bench: Scaling multimodal evaluation to over 500 real-world tasks. arXiv preprint arXiv:2410.10563.

Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nvlm: Open frontier-class multimodal llms. arXiv preprint arXiv:2409.11402.

Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. 2024. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146.

Abhimanyu Dubey, Abhinav Jauhuri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2024. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36.

Arpad E Elo. 1966. The USCF Rating System: Its Development, Theory, and Applications. United States Chess Federation.

Freepik, Eucalypt, Three Musketeers, Dewi Sari, Fantasyou, Jk Icon, and Flat Icons. 2025. Various icons.

David R Hunter. 2004. Mm algorithms for generalized bradley-terry models. The annals of statistics, 32(1):384–406.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os-trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.

Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024a. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.

Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. 2024b. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993.

Tianle Li, Anastasios Angelopoulos, and Wei-Lin Chiang. Does style matter? disentangling style and substance in chatbot arena, august 2024a. URL https://blog.lmarena.ai/blog/2024/style-control.

Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024c. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939.

Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of Psychology.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2025. Mm-bench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun-yuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. arXiv e-prints, pages arXiv–2310.

Yujie Lu, Dongfu Jiang, Wenhui Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. 2024. Wildvision: Evaluating vision-language models in the wild with human preferences. arXiv preprint arXiv:2406.11069.
Ziyang Luo, Haoning Wu, Dongxu Li, Jing Ma, Mohan Kankanahalli, and Junnan Li. 2024. Videoautoarena: An automated arena for evaluating large multimodal models in video analysis through user simulation. arXiv preprint arXiv:2411.13281.

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244.

Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuanchen Ren, Bofei Gao, Yibo Miao, Yunlong Feng, et al. 2025. Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings. arXiv preprint arXiv:2501.01257.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jingou Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. 2024b. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442.

T Wolf. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.

Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024. Longvideobench: A benchmark for long-context interleaved video-language understanding. arXiv preprint arXiv:2407.15754.

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. Minicpm-v: A gpt-4v level mlm on your phone. arXiv preprint arXiv:2408.01800.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024a. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567.

Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botaoyu, Ge Zhang, Huan Sun, et al. 2024b. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Metareview score: 3
Metareview: This paper proposes ProBench, a benchmark for assessing MLLMs across professional productivity tasks. The evaluation samples are sourced from 100K real-world crowdsourced conversations for productivity scenarios and the created benchmark contains 4,000 samples spanning 10 tasks, 56 fields, and 17 languages. After the creation of the benchmark, 24 leading MLLMs are evaluated comprehensively with MLLM-as-a-Judge. An in-depth analysis is conducted to demonstrate the data quality and model ranking in the benchmark.
Summary of reasons to publish: This study is well-motivated and targets real-world problems: Compared with existing benchmarks like MMMU and MMMU-Pro, the evaluation is driven by productivity demands and the benchmark proposed in this study is closer to real-world scenarios. It provides a testbed for MLLMs to evaluate their ability to answer open-ended queries in the professional daily setting.

The dataset is well-curated and validated: Starting from a large pool of crowdsourced conversations (100K), it induces a dataset covering a diverse range of tasks, fields, and languages. I also manually reviewed parts of the samples in the appendix and they are challenging, diverse, and close to real-world scenarios.

A broad range of MLLMs are evaluated: This evaluation covers both open and closed MLLMs and MLLMs of different sizes. A human evaluation is conducted to analyze the alignment between the MLLM judge and human experts, showing a decent agreement between them. In-depth analyses like error and robustness analyses are conducted.