On Path to Multimodal Generalist: General-Level and General-Bench
Authors: Hao Fei
Yuan Zhou
Juncheng Li
Xiangtai Li
Qingshan Xu
Bobo Li
Shengqiong Wu
Yaoting Wang
Junbao Zhou
Jiahao Meng
Qingyu Shi
Zhiyuan Zhou
Liangtao Shi
Minghe Gao
Daoan Zhang
Zhiqi Ge
Siliang Tang
Kaihang Pan
Yaobo Ye
Haobo Yuan
Tao Zhang
Weiming Wu
Tianjie Ju
Zixiang Meng
Shilin Xu
Liyu Jia
Wentao Hu
Meng Luo
Jiebo Luo
Tat-Seng Chua
Shuicheng YAN
Hanwang Zhang
Venue: ICML 2025 oral (2025)
Keywords: Large Language Model
Multimodal Large Language Model
Multimodal Generalist
Evaluation
Benchmark
Primary area: deep_learning->foundation_models

Abstract: The Multimodal Large Language Model (MLLM) is currently experiencing rapid growth, driven by the advanced capabilities of language-based LLMs. 
Unlike their specialist predecessors, existing MLLMs are evolving towards a Multimodal Generalist paradigm. 
Initially limited to understanding multiple modalities, these models have advanced to not only comprehend but also generate across modalities. 
Their capabilities have expanded from coarse-grained to fine-grained multimodal understanding and from supporting singular modalities to accommodating a wide array of or even arbitrary modalities. 
To assess the capabilities of various MLLMs, a diverse array of benchmark test sets has been proposed. 
This leads to a critical question: *Can we simply assume that higher performance across tasks indicates a stronger MLLM capability, bringing us closer to human-level AI?*

We argue that the answer is not as straightforward as it seems. 
In this project, we introduce an evaluation framework to delineate the capabilities and behaviors of current multimodal generalists. 
This framework, named **General-Level**, establishes 5-scale levels of MLLM performance and generality, offering a methodology to compare MLLMs and gauge the progress of existing systems towards more robust multimodal generalists and, ultimately, towards AGI (Artificial General Intelligence). 
Central to our framework is the use of **Synergy** as the evaluative criterion, categorizing capabilities based on whether MLLMs preserve synergy across comprehension and generation, as well as across multimodal interactions.
To evaluate the comprehensive abilities of various generalists, we present a massive multimodal benchmark, **General-Bench**, which encompasses a broader spectrum of skills, modalities, formats, and capabilities, including over 700 tasks and 325,800 instances. 
The evaluation results that involve over 100 existing state-of-the-art MLLMs uncover the capability rankings of generalists, highlighting the challenges in reaching genuine AI. 
We expect this project to pave the way for future research on next-generation multimodal foundation models, providing a robust infrastructure to accelerate the realization of AGI.
Project Page: https://generalist.top/,
Leaderboard: https://generalist.top/leaderboard/,
Benchmark: https://huggingface.co/General-Level/.

On Path to Multimodal Generalist: General-Level and General-Bench
Hao Fei Yuan Zhou
Juncheng Li Xiangtai Li Qingshan Xu Bobo Li
Shengqiong Wu
Yaoting Wang*
Junbao Zhou? Jiahao Meng Qingyu Shi Zhiyuan Zhou
Liangtao Shi Minghe Gao
Daoan Zhang? Zhiqi Ge Siliang Tang²
Kaihang Pan Yaobo Ye Haobo Yuan 2 Tao Zhang
Weiming Wu Tianjie Ju Zixiang Meng Shilin Xu Liyu Jia Wentao Hu Meng Luo
Jiebo Luo? Tat-Seng Chua
Shuicheng Yan 1ll Hanwang Zhang?
Abstract
preserve synergy across comprehension and gen- eration, as well as across multimodal interactions. To evaluate the comprehensive abilitics of vari- ous generalists, we present a massive multimodal benchmark, General-Bence
The Multimodal Large Language Model (MLLM) is currently experiencing rapid growth, driven by the advanced capabilities of language-bused LLMs. Unlike their specialist predecessors, exist- ing MLLMs are evolving towards Multimodal Generalist paradigm. Initially limited to under- standing multiple modalities, these models have advanced to not only comprehend but also gener- across modalities. Their capabilities have ex- panded from coarse-grained to fine- tine-grined mul- timodal understanding and from supporting sin- gular modalities to accommodating wide array of or even arbitrary modalities. To assess the capabilities of various MLLMs, a diverse array of benchmark test sets has been proposed. This leads critical question: Can we simply assume that higher performance across tasks indicates stronger MLLM capability, bringing closer to human-level AI?
which encom- passes broader spectrum of skills, modalities, formats, and capabilities, including over 700 tasks and 325,800 8,0 instances. The evaluation results that involve over 100 existing state-of-the-art MI.LMs uncover the capability rankings of generalists, highlighting the challenges in reaching genuine AI. We expect this project pave the way for fu- ture research next-generation multimodal foun- dation models, providing robust infrastructure accelerate the realization of AGI.
Project Page: https: /ttps::/gente https://generalist.too
Leaderboard: https: fttt::://ggeeeraiss..co leaderboard/
Benchmark: https://hugee httpa::/uugiinnfaae.com General-Level/
We argue that the answer is not as straightfor- ward seems. In this project, we introduce an evaluation framework to delineate the capabilities and behaviors of current multimodal generalists. This framework, named General-Levell es- tablishes 5-scale levels of MILM performance and generality, offering a methodology to com- pare MILMs and gauge the progress of existing systems towards more robust multimodal general- ists and, ultimately, towards AGI (Artificial Gen- eral Intelligence). Central to framework is the use of Synergy as the evaluative criterion, cate- gorizing capabilities based on whether MILMMs
1 Introduction
Large Language Models (LLMs, c.g., ChatGPT (OpenAI, 2022a) and LLaMA (Touvron et al., 2023)) have revolu- tionized the NLP field by serving as generalists addressing vast spectrum of NLP tasks. This breadth of capability has edged humans ever closer to the realization of Artificial General Intelligence (AGI). Yet, human intelligence inher- ently operates across multiple modalities, not solely through language. This observation has spurred the development of multimodal LLMs (Alayrac et al., 2022; Li al., 2023a; Liu al., 2023a: OpenAI, 2022b), i.c., multimodal general- ists, which are rapidly gaining traction and evolving towards AGI. The recent progress in MLLMs is marked by signif- icant advancements. For example, the initial multimodal agents where LLMs serve as mere task schedulers, later have evolved into joint foundation MLLMS (Zhu et al., 2023a; Liu et al., 2023a; Zhang et al., 2023a; OpenAl, 2022b; Wu et al., 2024a; Chen al., 2024a; Sun al., 2024). Also,
'Equal contribution 'NUS 'NTU "ZJU *KAUST 'PKU 'HFUT "UR *WHU "NJU "SITUU "Skywork AI. Correspondence to: Shuicheng Yan < nus.edu.sg>. Hanwang Zhang <han- wangzhang ntu.edu.sg>
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

On Path to Multimodal Generalist: General-Level and General-Bench
Language intelligence supports unidirectionally intelligence" of other modalities
Total synergy across any modalities, functions and tasks for authentic multimodal intelligence
Video
Audio
Video
Audio
Image
Image
Language
Language
(a) Existing intelligent pattern in multimodal generalist
(b) Ideal intelligent pattern in multimodal generalist
Figure 1: The "intelligence" in most existing multimodal generalists (i.e., MLLMs) hinges on language intelligence (i.e., from LLMs) (a), whereas the ideal intelligence mode should be maintaining synergy across modalities and tasks (b).
MLLMs have progressed from understanding only multi- modal signals to both comprehending and generating multi- modal content, even editing capabilities (Wang et al., 2023a; Munasinghe et al., 2023; Zhang et al., 2024a; Fei et al., 2024a). Further, these models have advanced from coarse- grained modal understanding to fine-grained multimodal comprehension, such as pixel-level visual modeling (Ren et al., 2023; Yuan et al., 2023a; Rasheed et al., 2023). More significantly, MILLMs that initially support only singleton non-textual modalities have now facilitated the understand- ing and generation of signals across various modalities, even simultaneously accommodating any modality (Wu et al., 2024a; Zhan et al., 2024; Lu et al., 2024a).
predominantly build on the language intelligence of LLMs to simulate the indirect intelligence of multimodality, which is merely extending language intelligence aid multimodal understanding. While LLMs (e.g., ChatGPT) have already demonstrated such synergy in NLP, reflecting language in- telligence, unfortunately, the vast majority of MLLMs do not really achieve it across modalities and tasks.
In this project, we introduce a sophisticated evaluation framework, General-Level, for more accurately posi- tioning and assessing the capabilities of current MLLM gen- eralists, charting path toward authentic multimodal AGI. Drawing inspiration from the tiered classification mecha- nism in the automotive industry for autonomous vehicles defines five
Accordingly, the community has introduced various bench- marks to evaluate those MLLMs (Wu et al., 2023a; Xia et al., 2024a; Yue et al., 2024; Meng et al., 2024a; Liu et al., 2025; Li al., 2024a; Ying et al., 2024a; Li al., 2024b). The prevailing evaluation mindset might yet be largely outdated, simplistically assuming that superior performance across tasks presents a stronger generalist capability (Xu et al., 2023a; Yu et al., 2023; Fu et al., 2024a; Chen et al., 2024b), and then being closer to AGI. We contend this perspective overly simplifies the implication inherent in real multimodal generalization. Theoretically. it's effortless assemble "super agent" from all singleton state-of-the-ar (SoTA) spe- cialists to achieve the above goal, while such simplistic integration would never suffice to realize genuine AGI. We argue that the key to advancing towards AGI lies in the syn- ergy effect-a capability that enables knowledge learned one modality or task to generalize and enhance mastery in other modalities or tasks, fostering mutual improvement across different modalities and tasks through interconnected learning.' As illustrated in Figure 1, most current MLLMs
(Yurtsever et al., 2020), General-Ieve principal levels of model performance and generality. Cen- tral to the framework the synergy ability as the evaluative criterion, categorizing capabilities based on whether gener- alists preserve synergy in and across multimodal compre- hension and generation, well as cross-modal interactions. From the lowest to the highest level, the scope of synergy ability required progressively escalates from single tasks or modalities to total synergy. As a generalist strives to advance to a higher level, must demonstrate significant enhancements in its synergy capabilities, during which the difficulty of progression is also inherently increasing.
To effectively evaluate within the General -Leee frame- work, suitable benchmark is essential. While there are nu- merous MLLM evaluation benchmarks, e.g., LVLM-eHub (Xu et al., 2023a), MME (Fu et al., 2024a). MMMU (Yue et al., 2024), SEED-Bench (Li et al., 2024a), MMT-Bench (Ying et al., 2024a), and MEGA-Bench (Chen al., 2024b). they might have certain limitations that render them inad- equate for needs. Firstly, existing benchmarks often convert all tasks into a uniform multiple-choice QA format (Fu et al., 2024a; Ying et al., 2024a), simplifying the evalua-
'Synergy, essence, can be understood a form of general- ization ability.
2

---

On Path to Multimodal Generalist: General-Level and General-Bench
tion process but consequently restricting assessments only the models' multimodal comprehension capabilities. How- ever, a true multimodal generalist should support not only comprehension, but also possess capabilities in multimodal generation, editing, and beyond. Second, the majority of current benchmarks (Wu et al., 2023a; Liu et al., 2025; Li et al., 2024a) predominantly focus on the image modality and overlook other crucial modalities such as video, audio, even 3D and beyond, which are vital for robust multimodal generalist. Third, these benchmarks are typically limited to coarse-grained multimodal understanding (Xu et al., 2023a; Yu et al., 2023; Fu et al., 2024a) and fail to adequately assess finer-grie tner-grained ones, which actually lag far behind the current advancements in MLLMs, i.e., supporting pixel-level im- age understanding and generation (Fei et al., 2024a; Zhang et al., 2024a). In response these challenges, we pro- pose General-Bench,
MLLMS represent an important advancement of unifica- tion to handle all modalities and tasks with one foundation model, i..., multimodal generalists. Naturally, empowering multimodal generalist with strong multimodal intelligence capabilities is an essential pathway toward realizing AGI.
Technically, the vast majority of existing MLLMs have frameworks that are anchored by an LLM to serve as the core for reasoning and decision-making By integrating various modules of different modalities or tasks (typi- cally existing specialists, e.g., CLIP (Radford et al., 2021) and Stable Diffusion (Rombach et al., 2022)), MLLMs are facilitated with the comprehension and even generation of di- verse modalities. Representative MLLMs include Blip2 (Li et al., 2023a), LLAVA (I.iu et al., 2023a), MiniGPT-4 (Zhu et al., 2023a), Flamingo (Alayrac et al., 2022), and NExT- GPT (Wu et al., 2024a), among others. However, such an architectural setup merely simulates 'pseudo" multimodal intelligence, as it still fundamentally relies on the language intelligence of LLMs without genuine non-language modal- ity intelligence. As emphasized earlier, a capable generalist must possess synergy capabilities across all modalities and tasks, akin how an LLM (c.g.. CharGPT) generalizes well unseen NLP tasks, despite not being exposed to all tasks during its training. While these current multimodal generalists can deliver strong performances on multimodal benchmarks, sometimes even on par with SoTA specialists, they do not fundamentally achieve true synergy.
massive multimodal evaluation benchmark, spanning from various modalities (e.g., image, video, audio, 3D, language, and beyond) in diverse native formats, covering a wide range of tasks that thoroughly assess the full capabilities of a multimodal generalist.
Our evaluation of over 100 existing top-performing LLM/MLLM systems has uncovered critical insights into their capabilities and rankings as multimodal generalists. The most notable finding is that most MLLMs lack the cross- task or cross-modal synergy ability required for higher-level classifications, with even advanced models like GPT-4V and GPT-4o not achieving top ranks. This highlights a consider- able gap in achieving the goals of multimodal generalists. Also, the majority of existing MILMS manage only a few basic multimodal tasks and skills, which negatively affects their scoring. Most critically, no model has yet demon- strated the ability to enhance language intelligence through non-language modalities, underscoring the substantial chal- lenges in the pursuit of genuine AGI.
Consequently, this paper positions synergy as the central cri- terion for evaluating multimodal generalists on their journey toward AGI. Current evaluation methods (Li et al., 2024b) for MLLMs still adhere to the traditional approach used for specialists, simply comparing the MLLM performance on multimodal tasks, assuming that higher scores indicate greater strength and closer proximity to AGI. Going beyond that, we propose a new evaluation framework-no -ot only do we compare whether models support various modalities and tasks and their performance, but also rank them based on the synergy capabilities of multimodal generalists. Meanwhile, we significantly expand the scope of current MLLM benchmark datasets in terms of modality and task coverages, as well as task formats, contributing to the most comprehensive benchmark dataset to date the community.
Contributions: 1) We introduce a tiered classification sys- tem called General-Level 1 for multimodal generalists, establishing rigorous norm that can guide future MLLM research. 2) We contribute new evaluation benchmark (General - Bench) that provides the most comprehensive coverage of modalities and tasks available to date.
2 Background and Related Work
3 General-Level: A 5-Level Taxonomy of Multimodal Generalists
More and more tend to recognize that LLMs have unlocked the potential of language intelligence, bringing unprece- dented hope to achieve AGI. Essentially, an LLM serves as a generalist capable of tackling nearly all downstream NLP tasks. LLMS have subsequently evolved in an effort to extend this intelligence across various other modalities, ie.. MLLMs (Bai et al., 2023; Zhang et al., 2023b; Jin et al., 2023; Li et al., 2024c; Fei et al., 2024b;c). Unlike the past 'smaller' specialists (Van Den Oord et al., 2016; Rad- ford et al., 2021; Rombach et al., 2022; Liu et al., 2023b),
3.1
Preliminary
3.1.1
OBSERVATIONS AND PRINCIPLES
Observation-1: Multimodal Comprehension vs. Simul- taneous Multimodal Comprehension and Generation. Initially, MILMS are capable only of interpreting multi- modal signals, meaning their responses are limited to textual
3

---

On Path to Multimodal Generalist: General-Level and General-Bence
Generalists Level
Generalists Level
Generalists Level
Modalities
Tasks//kklls
Tasks/Skills
Comprehension
Generation
synergy
Synergy Tasks/Skills
Synergy across Comprehension
Synergy Modillties
Figure 2: specific illustration on synergy effect.
outputs based on user-provided multimodal inputs. How- over, an MLLM that only offers multimodal comprehension operates at the most basic and rudimentary level. More ad- vanced MLLMs have since emerged, equipped with not only multimodal comprehension but also the ability to generate and even edit content across various modalities. It is widely believed that the more advanced multimodal generalist is, the more should encompass advanced functionalities, encompassing both comprehension and generation.
tools and modules (often specialists) to handle specific mul- timodal tasks. This setup referred to as a multimodal agent. Subsequently, attention shifted towards building joint MLLMs, where the LLM is tightly integrated with other modules, such multimodal understanding components (front-end) and multimodal generation components (back- end), through shared embedding space. This setup allows for joint training, where the entire system, including all pa- rameters, can be updated end-to-end. The complexity of AGI requires deeper integration and generalization across tasks and modalities.
Observation-2: Covering Broader Modalities. Being a multimodal generalist requires the ability to extensively sup- port and handle a wide range of modal data, including, but limited to, text, images, videos, audio, and even 3D, The extent of modal support is indicative of the breadth of AI system's 's capabilities. Initially, MILMs could manage only singleton non-ling non-linguistic modality, c.g., images, videos, or audio signals. To date, these models have evolved to simul- tancously support multiple non- lon-linguistic modalities such as combining images with videos, videos with audio, and even any modality in the current most advanced cases.
3. SYNERGY AS CORE TO GENERALISTS
We argue that determining whether a multimodal generalist is stronger cannot be simplistically equated with achieving higher scores on benchmark or/and supporting as many multimodal tasks as possible compared to other models common practice in current MLLM benchmarking and eval- uation. A simple countercxample can illustrate this point: it could be comparatively easier to construct a 'super agent' by integrating all SoTA specialists for various multimodal tasks into single system. Such agent could achieve top- level performance across all tasks (on par with the strongest individual specialist models) while supporting a wide range of multimodal functionalities. However, such agents can be far from the multimodal generalist we expect as a pathway to AGI. Such a type of agent lacks inherent multimodal intelligence and capabilities, it relies on an ensemble of specialized systems rather than embodying true, native multimodal generalization.
Observation-3: Supporting Various Tasks and Paradigms. To qualify must be capable of handling broad range of tasks with different definitions and requirements. The greater the variety of tasks supported, the stronger the generalist's s overall versatility. For example, early visual MI.LMs could only manage coarse-grained image understanding, but recent advancements have enabled them achieve fine-graing gra-ggri pixel-level multimodal comprehension, such pixel-level image/video grounding and editing. This advancement necessitates that the model's decoding components should be versatile enough to generate outputs in various task formats, not merely restricted to text.
multimodal generalist, it
Instead, the ideal multimodal generalist (and ultimately AGI) we envision should be a multimodal counterpart of an all-capable OpenAI ChatGPT series. Such a model would not only surpass SoTA specialists task-wise performance across various tasks and modalities but also exhibit ex- ceptional cross-task, cros-comppeheesioon-geneeee cross-modality generalization capabilities. In other words,
Observation-4: Multimodal Agent vs. Multimodal Foun- dation Model. Initially, researchers approach multimodal tasks by using LLMs as task schedulers, where an LLM orchestrates the execution of tasks by invoking external
and

---

On Path to Multimodal Generalist: General-Level and General-Bence
Table 1: General-Level framework toward classifying multimodal generalists into FIVE levels based on the synergy abilities models preserve. We denote the number of tasks within the Comprehension group by M; the number within the Generation group by N; and the number of NLP tasks by T.
Level
Definition
Scoring
Example
Level-1:
Various current models, cach fine-tuned on a specific task dataset specific modalities, are task-specific players (i.e., SoTA special- ists). This includes various learning tasks, such as linguistic/visual recognition, classifi- cation, generation, segmentation, grounding. inpainting. and more.
For each task in the benchmark (6-th tusk), the current SoTA specialist's score is recorded as:
CLIP (Li et al. 2022), FLUX
Specialists
(Labs,
2023),
of oota
PastSpeech2
(Ren et al., 2021),
Upgrading Condition: Supporting as many tasks and functionalities as possible
Level-2:
Models are task-unified players. c.g. MI..Ms, capahle supporting different modalities and tasks. Such MLLMs can integrate various models through existing en- coding and decoding technologies to achieve aggregation and unification of various modalities and tasks (such comprehension and generation tasks).
The average score between Comprehension and Generation tasks (i.e., across all tasks) represents the score at this level. A model that can score non-zero on the data is considered capable of supporting that task. The more supported tasks and the higher the scores, the higher its overall score:
Unified-io-2 (Lu
al., 2024a),
Generalists of Unified Comprchension and/or
AnyGPT (Zhan
et al., 2024),
Generation
NEXT-GPT (Wu
et al.. 2024a),
SEED-LLuMA (Ge ct al., 2023),
M
N
S2
G
GPT-4V (Ope- nÅI, 2022b). ...
M
N
1=1
J=1
Upgrading Condition: Generalists achieving stronger synergy and cross many tasks possible
Level-3:
Models are task-unified players, and syn- engy is in Comprehension and/or Generation. MLLMS enhance several tasks' performance beyond corresponding SoTA scores through joint learning actoss multiple tusks due to the synergy effect.
Assign a mask weight of 0 or to each task; mask=l only if the corresponding score (of or
GPT-40 (OpenAl, 2022b), Gemini-
af) exceeds the SoTA specialist's score, other-
1.5 (Team et al.,
Generalists with syn- ergy in Comprehension
2024a). Claude-
wise mask=0. Then, calculate the average score
3.5 (Team. 2024),
and/or Generation
between Se and 8g. The more tasks to surpass the SoTA specialist, the higher the Ss:
DeepSeek VL (Lu et al., 2024b). 20244b
Sg
(Sg + Sc)) where
LLaVA-One- Vision (1.i et al., 2024d), Qwen2-
Sc
M F=1
1
Σ
of
0
of of gote otherwise
VL (Wang et al., 2024a),
In-
VLm5 (Chen et al. 2024e), Phi-3.5-Vision (Abdin et al., 2024).
1 Sc = N
Jo
if of 2 Lota otherwise
Upgrading Condition: Generalists in unified comprehension and generation capability with synergy in between
Level-4:
Models are task-unified players, and synergy
Calculate the harmonic mean between Compre- hension and Generation scores. The stronger synergy a model has between Comprehension and Generation tasks, the higher the score:
Mini-Gemini (Li al., 2024c), Vit (Fei
across Comprehension Generation.
et al., 2024a), Emu2-37B (Sun
Generalists with symergy across Comprehension and Generation
25c
S4 Sc + SG
et al., 2024),
Upgrading Condition: Generalists achieving cross-modal synergy with abductive reasoning ability
Level-5:
Models are task-unified players, preserving the synergy effect across Comprehension, Generation, and Language. In other words, the model not only achieves cross-mydality synergy between Comprehension and Gener- ation groups but also further realizes synergy with language. The Language intelligence can enhance multimodal intelligence and vice versa; understanding multimodal information can also aid in understanding language.
Calculate the model's average score exceeding SoTA NLP specialists on NLP benchmark data; normalize it to a [0.1] weight, and multiply by the score from level-4 as the level-S score:
None found yet (Let's y wait for multimodal Chat- GPT moment!)
Generalists with total synergy across Compre- hension, Generation and Language
S. = S4 x wt. where
SL
WL
where Spoll
T
[ok
if 2 Gsta
S₁
T
U
otherwise
the knowledge learned from certain tasks, skills, and modal- ities should be transferable to other tasks, skills, and modal- ities diees-xttrapolating the understanding to effectively engage with other tasks and modalities, and vise versa, creating a synergistic effect where the combined result exceeds the sum of individual contributions, achieving a 1+1>2 effect. ChatGPT on the language side can be good example: it out-
performs SoTA specialists in unseen tasks without having undergone specific training for those tasks. This generaliz- ability is what we claim as the synergy effect.
3.2 Defining Levels Centered on Synergy
Based on the above principles, we introduce a 5-level taxonomy of multimodal generalists, General-ievel.
5

---

On Path to Multimodal Generalist: General-Level and General-Bench
Editig
3D QA
Legoub-ts
Commonsena Rearoning
Reasortng
Hlesoringg
emetional Facil Synthesia
touiltysse pive skite?
Abccnott offece
Spatial Spatial Relation Detection
Toxt-o de Unimationn
Key wordd Estracioo
Nammed Ently Recognition
Text Classificatio
Sintax Parsing
3D Scane Understannio ding Texture Mappang
Teriual Enailment
T on
Tee- Spe-p
Exant Extraction
Paamod Lipry
the
Reooghloo
weten
Intent Detection
Quesitio Anawering
3D Object Recognition
Depth Estimation
Reading Comp Comprehension
VQA
Audio QA
Gen
Visel Geounding
Comp
Music Generaton
Language
Image Clas sificatiooon
Text: to Speech
Gen
3D
Caption
Depth Estimtion
Object Counting
Audio Capton
Audio
Audio QA
Camp
General
Comprehension
OCR
Bench
Audio Caption
Image
Image QA
Speech Recegnition
Genera Generaaion ation
Video
Imagee:to to-text Retrieval
Anomaly deteccioo
Tert-bosed bosed Imagee Editing
Video Style Transfer
Objfect Detection
Video Super Resolution
Visual Grounding
Vicno Retrioaal
Image to Video Synthesia
Video Inpainting
Text Video Synthesis
Video Editing
Compreleension
Generation
Text-to image Generation
Semantic Segmentation
Face Recognition
Sketzh-t Imagee
Stetch image Generate anneration
Video Fran Frame Prediction
Preson Re- Aeeiddeniiiiian
Video Mom Retrieval
Video Captioning
Video ob
Video
Text based image Editing
Image Impainting
Super Resolution Synthesis
Layout-to ayout-tto-h imagemaageee Synthesis
Semarlic Seguen dion
Trocing
cessifictton
Video Geheriion
Image Impainting
Vidtoo Temporai Grnund
DCR
Moningion
Figure 3: Overview of General-Bennee which covers 145 skills for more than 700 tasks with over 325,800 samples under comprehension and generation categories in various modalities. Appendix B.6 gives holistic hierarchical taxonomies.
General-Lever framework evaluates generalists based on the levels and strengths of the synergy they preserve. Specifi- cally, we define three levels and scopes of synergy, ranked from low to high: 'task-task', "comprehensionngeneratiio and poodality-modaligy as illustrated in Figure 2. Achiev- ing these levels of synergy becomes progressively more challenging, corresponding higher degrees of general in- telligence. Assume have benchmark of various modal- ities and tasks, where modalities into the Comprehension group and the Gener- ation group, as well as the language (i.e., NLP) group, as illustrated in Figure Now, we can define the scoring specification of General-Leve
We assume that current MLLMs have already demonstrated synergy mode from language non-language modalities. Then the remaining mission is to confirm the existence of synergy in the reverse direction, from non-language to language modalities. Therefore, for level 3-mmsurring total synergy-we do not measure the generality across all modalities and tasks. Instead, we assess whether model can improve NLP task performance to exceed that of NLP SoTA specialists
categorize tasks under these
4 General-Bench: A Holistic Benchmark for Multimodal Generalists
as in Table 1.
We introduce Genera -Bench, benchmark the outlined criteria and serve the standard dataset for evaluation framework. Appendix $B shows more details about the benchmark data.
When caleulating scores using the corresponding formula, normalize all task metrics to 100-point scale. While most task evaluation scores typically range from 0- 1-00 such as Fl and Accuracy, certain metrics, e.g., FID, MAE, and PSNR, yet yield scores outside this usual range. Thus, we design some mapping functions to standardize performance scores. Our framework also incorporates the principle of diminishing scores: an MLLM (i.e., multimodal generalist) achieve scores at multiple levels, but it is classified at its highest level, where it achieves a non-zero score.
The data compiled for General -Bechh is visualized in Figure visualizes the General-Bence highlights of task/modality support. Overall, the current version of the dataset includes those most common modalities (inner ring). and except for NLP tasks, all modalities distinguish be- tween comprehension and generation tasks (middle ring). General Bench particularly places a strong emphasis
6

---

On Path to Multimodal Generalist: General-Level and General-Bench
Table 2: Performance of multimodal generalists on various image comprehension skills. Skill full names and specific tasks are listed in Appendix. The full performance records of more generalists are shown in Appendix & C.
Image
Comprehension Skill (Ave within each #L-C Group)
Task Completion
Level Image
#1
#2
#4
#5
#7
#8
#9
#10
Model
#11
#12
#13
#14
#15
#16
#17
#18
#19
#20
#Supported
#Win-over- Specialist
Level-2
Lovel-3
Level-4
#21
#22
#23
#24
#25
#26
#27
#28
#29
A30
Task
N31
8132
#33
R33
#35
6136
N37
438
N39
(140
51.27
53.32
42.04
22.30
39.02
22.42
46.02
15.67
51.20
28.01
SoTA Specialist
36.40
70.00
50.40
15
65.97
43.78
58.90
16.60
78.00
63.73
87.84
50.48
58.66
19.90
53.55
72.25
64.10
34.51
95.70
35.90
/
39.80
57.20
54.60
63.27
29.60
87.10
98.00
39.60
36.42
82.02
69.42
58.64
39.54
0.00
66.18
36.08
61.
0.00
16.90
20.88
GPT-4V
71.90
0.00
37.12
0.00
50.30
51.04
63.52
16.06
72.20
0.00
70.90
0.00
51.60 11.60
0.00
72.51
0.00
0..00
0.00
97.98
0.00
177 (65.1%) 1%)
105 (38.6%)
18.16
12.85
0.00
40.05
0.00
90.40
0.00
31.64
89.10
22.22
22.54
18.08
84.84
73.87
63.42
43.23
0.00
71.56
39.65
68.
0:00
67.
23.24
GPT-40
81.30
0.00
39.61
0.00
48.63
71,23
61.54
15.12
93.00
0.00
79.38
0.00
55.25
0.00
77.53
0.00
0.00
0.00
98.79
0.00
177 (65.1%)
112 (41.2%)
19.67
14,51
0.00
44.30
0.00
90.40
0.00
33.47
91.20
35.56
24.80
21.12
87.88
72.33
23.41
39.39
0.00
62.38
34,30
0.00
59.20
23.79
Gemini-1.5 1,5- Pro
84,57
0,00
31.55
0.00
60.87
60.86
40.10
15,20
86,40
0.00
0.00
0.00
58.09
0.00
76.72
0.00
0,00
0.00
96.76
0.00
177 (65.1%)
101 (37.1%)
19:67
12.66
0.00
36.41
0.00
98.00
0.00
388 45
92.00
30.37
22.18
21.20
83.23
67.00
25.79
37.85
0.00
59.45
29.91
63.61
0.00
56.50
22.19
Gemini-1.5- F,5-alhh
80.63
0.00
28.97
0.00
55.22
56.91
32.92
16.57
$2.60
0.00
0.00
0.00
54.57
0,00
73,57
0.00
0.00
0,00
93.42
0.00
177 (65.1%)
94 (34.6%)
18.54
10.85
0.00
28.53
0.00
0.00
29.97
20.64
18.22
80.40
65.38
57.69
39.95
0.00
63.35
34.50
63.43
0.00
45.62
20.44
Cluude-3.5-Opun
70.39
0,00
41.19
0.00
54.75
60.21
58,15
13.87
77.80
0,00
66.57
0.00
51.23
0,00
73.04
0.00
0.00
0.00
94.65
0.00
178 (65.4%)
93 (34.2%)
19:00
11.08
0.00
38.28
0.00
91.38
0.00
0.00
87.31
23.87
28.71
25.75
84.65
53.76
7.31
36.62
0.00
22.22
41.89
0.00
21.20
12.83
Emu2-32B
56.33
0.00
29.43
0.00
45
39 47
21.45
12.20
64.20
0.00
0.00
0.00
44
0.00
54.59
5 5.28
0.00
0.00
70.34
0.00
178 (65.4%)
52 (19.1%)
30.90
5.18
1.25
17.73
0.00
72.80
0.00
0.00
73.40
31.72
14.09
18.73
56.97
55.32
3.44
34.16
0.00
42.61
42.04
51.34
0.00
0.00
24.35
Phi-3.5-Vision
0.00
0.00
41.00
21.77
0.00
0.00
52.13
11.89
0.00
0.00
179 (65.8%
85 (31.3%
16.46
9.39
0.00
Instruct
67.56
32.32
$1.51
23,70
90,10
0.00
0.00
57.68
0.00
$2.02
19.31
0.00
83.40
0.00
15.02
80.00
3.98
23.06
25.41
71.31
66.98
5.74
35
0.00
56.58
40.50
48.79
0.00
43.18
25.32
Qwen2-VL-72B
0.00
0.00
45.66
29.44
0.00
0.00
59.87
10.89
0.00
0.00
177 (65.1%)
99 (36.4%)
19.41
12.34
0.00
81.86
38.59
58.99
16.17
97.43
0.00
0.00
72.47
0.00
92.41
4.33
0.00
77.64
0.00
16.83
79.34
11.65
29.62
32.22
62.83
46.68
0.00
31.85
0.00
40.59
13.48
35.10
0.00
7.20
9.09
SEED-LLaMA-I3B
0.00
0.00
25.42
8.00
0.00
0.00
33.60
4.76
0.00
0.00
174 (64.0%)
39 (14.3%)
26.81
3.49
0.00
38.53
22.52
32.67
24.96
32.20
0.00
0,00
47.48
0,00
66.19
0.00
0.00
71.60
0.00
0.80
69.80
14.43
13.13
10.19
51.72
53.54
0.00
33.85
0.00
49.78
27.69
50.71
0.00
6.00
9.41
DeepSeek-VI..e V1.-B
53.53
0.00
19.30
0.00
42.69
35.35
21.59
33.01
5.80
0.00
0.00
0.00
40.1
0.00
51.56 36
7,80
0.00
0.00
50.71
0.00
180 (66.2%)
64 (23.5%)
13.13
5.75
0.00
20.44
0.00
0.00
16.83
42.60
9.44
11.97
65.05
59.96
4.86
24.
0.00
38.08 08
35,39
57.54
0.00
7.76
12.46
In2emVL2,,588B
30.13
0.00
28.37
0.00
26.68
46.05
17.74
16.95
D.00
7.82
0.00
48.81
0.00
54,99
8.06
0.00
0.00
74.49
0.00
183 (67.3%)
71 (26.19))
25.20
13.09
0.00
18.18
0.00
99.60
0.00
10.57
85,90
33.52
9.71
16.91
57.17
47.6
3.90
51.58
2.30
35.66
4.81
39.78
0.000
13.30
13.81
Vitron-V V1
50.00
0.00
66.60
28. 14
39.47
22.28
23.52
8.19
0.00
44.96
82.72
25.
0.00
52.11
22.24
71.89
14.63
64.20
0.00
252 (92.6%)
62 (22.8%)
30.13
7.65
4.59
19.07
36.70
51.38
55.85
4,70
69.26
15.34
19.12
24.48
59.07
50.47
1.90
32.31
0.00
42.52
11,84
50
0.00
3.80
22.11
MoE-LLAVV-
Phi2-2.7B-4e-384
51.13
0.00
20.92
0.00
33.98
26.99
35.40 50
19.87
0.00
80
0.00
0.00
41.79
0.00
$2.00
8.62
0.00
0.00
52.73
0.00
180 (66.2%)
55 (20.2%)
12.55
5.47
0.00
15.69
0.00
50.40
0.00
13.71
84.05
8.70
12.57
15.95
51,52
52.53
0.00
26.00
0.00
36.72
12.35
44.03
0.00
0.60
20.88
mPLUG-OW12-
LLaMA2-76
51.60
0.00
23.60
0.00
41.66
29.01
27.08
18.67
86.80
0.00
0.00
0.00
31.75
0.00
51.67
9.39
0.00
0.00
42.51
0.00
177 (65.1%)
45 (16.5%)
12.21
4.60
0.00
15.27
0.00
60.20
0.00
9.00
80.10
8.88
12.14
17.48
70.10
on the diversity its evaluation data, covering a wide range of fields and scenarios assess different aspects of model capabilities. First, the dataset spans variety of domains and disciplines, incorporating 28 major areas within both
the physical sciences (e.g., Physies, Math, Geometry, Biol- ogy) and the social sciences (e.g., Humanities, Linguistics. History, Social). The evaluation of a generalist's skills and capabilities is categorized into universal modality-invarian
7

---

On Path to Multimodal Generalist: General-Level and General-Bende
Table 3: Performance of part of multimodal generalists on image generation skills.
Image Generation Skill (Avg within each WI-G Group)
Taask Completion
Level Score on Image
Model
#1
#2
#3
#4
#5
#6
#7
#8
#Supported #Winning Level-2
Level-3
Level-4
89
#10
#11
#12
#13
#14
#15
Task
Specialist
So Specialist
18.70
45.40
33.77
16.30
4.86
24.00
99.29
15.06
/
/
53.16
16.47
25.33
43.93
20.35
67.44
36.11
SEED-LLaMM-14B
127.10
30.18
87.90
0.00
37.10
14,58
175.33
7.51
127.42
0.00
51.82
98.33
62.60
0.00
0.00
35 (77.8%)
0 (0.0%)
26.81
3,49
0.00
Emu2-32B
93.52
0.00
34.85
8,53
101.80
81.95
0.00
0.00
34 (75.6%)
(4.4%)
30.90
5.18
1.25
40.51
118.55
15.43
154.26
0.00
57.09
58.17
AnyGPT
158.21
0.00
40.47
10,30
117.21
115.91
0.00
0.00
36 (80.0%)
(0.0%)
23.10
1.29
0.00
28.88
108.06
14.91
193.39
0.00
53.02
64.21
LaVIT-V2 (7B)
79.79
0.00
31.35
11.87
149.78
59.23
0.00
0.00
36 (80.0%)
(0.0%)
29.50
3.71
0.00
46.40
89.78
15.79
161.54
0.00
50.18
51.68
VEXT-GGPEV1L5
49.71
0.00
6.00
3.91
75.71
41.20
0.00
47.30
41 (91.1%)
(0.0%)
18.69
3.24
0.00
28.19
86.45
6.53
53.42
12.45
38.98
72.72
Vitron-VI
19.78
0.00
21.17
7.45
32.15
35.33
86.53
23.47
42 (93.3%)
3 (6.7%)
30.13
7.65
4.59
37.88
24.89
17:95
31.04
0.00
48.30
58.87
abilities and modality-specifff skills. The modality-invarian abilities comprehensively include 12 categories, such as content recognition, commonsense knowledge, reasoning ability, causality discrimination, affective analysis, creativ- ity, and innovation, etc. For modality-specific skills, we explicitly detail the main capabilities under both compre- hension and generation for each modality, which correspond to the meta-tasks (skills) of our dataset.
5.2 Main Evaluation Results
We note that all the generalists run the evaluation on our General-Bence data set under a zero-shot setting. The overall results of part of the models on image comprehen- sion and generation are presented in Table 2 and Table 3, respectively; Due to space limitations, main results in Appendix SC.2. video results are shown in Table 11; audio results are shown in Table 12; 3D results are shown in Table 13; The results of all generalists on NLP tasks are shown in Table 14. The complete perform- ing scores of all MILMs across all tasks and datasets are presented in Appendix &C. Overall, we have the following observations.
move the rest
5 Experiments
In this section, we conduct comprehensive evaluation General-Bend from which we gain observations and jump to some conclusions.
Observation-1: Lack of task support. From these results, the first observation is that the vast majority of MLLMs exhibit a lack of support for a wide range of tasks in our benchmarks. Even models like OpenAl's GPT-4V and GPT- 4o, which achieve top rankings on many existing MLLM benchmarks and leaderboards (Li et al., 2023b; Liu et al., 2024a), fail to demonstrate satisfactory task support on benchmark. Specifically, GPT-4V and GPT-4o support only 177 out of 271 image comprehension tasks (65.1%). Among open-source models, Intern achieves a task sup- port rate of 71% for image comprehension tasks, outper- forming GPT-4V and GPT-4o. For other modalities- such as video, audio, and 3D-the task-supporting rates are much less. Only Vitron-VI supports over 90% of image tasks, and Sa2VA-8B achieves 72.2% supporting rate in the video com- prehension group. This highlights a pervasive issue: current MLLMs require significant improvements in their architec- tural design to support as many tasks as possible.
5.1 Experimental Settings
For different models, we consistently follow the settings provided in their respective GitHub repositories, including model parameters and hyperparameters. We do not per- form additional pre-training or fine-tuning. Fach task and dataset comes with a predefined instruction prompt text. During evaluation, we use the same default prompt across all MLLMs to ensure fairness. The inference time varies across models. Smaller models complete evaluations within few minutes, while larger models require significantly more time. On pure text-based NLP tasks, model inference is highly efficient; however, on video tasks, models demand more memory and have slower inference speeds. Our open- source codebase supports multi-GPU distributed inference, effectively accelerating the evaluation process. Also, we organize personnel into multiple groups run models in parallel, further optimizing efficiency. For each task, we provide predefined evaluation scripts. Once the model gen- crates outputs, the scripts are used to evaluate performance systematically.
Observation-2: Few generalists surpass the SoTA special- ist. Also, we can notice that there are few models capable of surpassing the SoTA generalist. Overall, the tasks and skills that various MLLMs can surpass the SoTA specialists
8

---

On Path to Multimodal Generalist: General-Level and General-Bench
are quite few. As seen, closed-sourced models (e.g., GPT- 4V, GPT-4o, Gemini-1.5, and Claude-3.5) have the highest winning rate, with over 30% The best open-soureed Qwen2- VL-72B achieves rate of 36.4% image comprehension by surpassing SoTA specialists. In other modalities such as video, audio, 3D, and language, the chances to surpass SoTA specialists are much lower. If an MLLM cannot out- perform the SoTA specialist, it implies that the foundational conditions of cross- tooss-task/abilityy synergy for these MILMS to become multimodal generalists are not met.
specialists in NLP. Furthermore, the performance gap be- tween MLLMS and SoT specialists in NLP tasks is larger than the gap observed in other modalities. While certain rel- evant research suggests that models, such as Vicuna, Qwen2, and LLaMA, trained with multimodal data (e.g., images) can also improve NLP tasks, such improvement has not yet enabled models to outperform SoTA NLP specialists on core language tasks. Our large-scale evaluation shows they still fall short of outperfomning fine-tuned language specialists. We hypothesize that existing MLLMs, despite utilizing language-centered LLMS as their core, have sig- nificantly weakened their language capabilities due to an excessive focus on training and fine-tuning on non-language modalities. This trade-off not only undermines their lan- guage understanding but also fails leverage multimodal information to enhance language-related tasks.
Observation-3: Focus more on content comprehension than supporting generation. For instance, GPT-4V and GPT-4o achieve better results than the SoTA specialist in certain skills within image comprehension tasks, and this improvement significantly more pronounced than that of other models. However, GPT-4V and GPT-4o are limited to image comprehension tasks and provide zero support for image generation tasks. It is thus evident that GPT-4V and GPT-40 are not well-rounded multimodal generalists. This trend becomes even more evident in other modalities. A significantly higher number of MILLMs support multimodal understanding compared to those supporting generation. Furthermore, the rate at which MILMS surpass SoTA spe- cialists in multimodal understanding benchmarks is much higher than in multimodal generation benchmarks. We em- phasize that this imbalance reflects a critical limitation in the capability building of current multimodal generalists.
5.3 More Analyses and Discussions
Due to space limitations of the main article, we show much more experimental results as well as the analyses and dis- cussions in Appendix, including:
The main results of the multimodal generalists on other modalities, in SC.2;
The complete results of models on specific tasks, from $C.6 to SC.10;
The General-Level scores of leaderboards, in $C.3;
. Capability of tasks and modalities break in $C.4;
Insufficient support for all modalities. We also found that many MLLMs are unable to support all modalities simultaneously. Moreover, the vast majority of existing MLLMs are predominantly focused on understand- ing or generating image-baser modalities. In contrast, much less attention has been devoted to video, audio, and 3D modalities (attention: image > video > 3D > audio), with relatively few multimodal generalists addressing these areas. Most MLLMs, including the strongest ones, primarily han- dle image and language tasks, offering little to no support for other modalities. The completeness of support across various modalities and functionalities is insufficient for ex- isting MLLMs to qualify as true multimodal generalists. We emphasize that to be considered a multimodal generalist, a model must be capable of understanding and generating signals from as many modalities as possible simultaneously.
Observation-4:
Analysis and discussion synergy, in $C.5;
. Future work of this project, in $D.
6 Conclusion
Inspired by the concept of capability levels defined in the au- tonomous driving industry, we propose General lLevel, framework that evaluates and categorizes the capabilities of existing MLLMs through 5-level hierarchical rating mechanism based on their ability to maintain synergy across comprehension, generation, and multimodal interactions. General-Level provides a structured methodology to assess MLLMS across diverse tasks, modalities, and syner- gies. particularly in comprehension and generation. To sup- port this evaluation, we further present General-Ben large-scale multimodal benchmark that spans a wide spec- trum of tasks (702), modalities (language, image. video, audio, 3D, etc.), domains (29), and original formats with 325,800 instances. By benchmarking over 100 popular L.LMs/MLLMM we uncover the current capability limita- tions and provide clear ranking of generalist performance. We hope the General-Leved and Gene Benerall-Bnch in this study will propel the community to develop next- generation multimodal foundation models to achieve more sophisticated, general- purpose multimodal intelligence.
Observation-5: Multimodality does NOT really enhance language. The ideal multimodal generalists should enable mutual enhancement across modalities. Unfortunately, our experimental results (as shown in Table 14) reveal that none of the current MLLMs provide any improvements in NLP tasks. Although various MILLMs achieve certain scores on NLP tasks, none of them surpass the performance of SoTA
It would thus more rational to claim the current OpenAl GPT-4V/4o series as partial generalists, or visual generalists.
9

---

On Path to Multimodal Generalist: General-Level and General-Bench
Impact Statement
cording to their respective terms of use, and no proprietary content is redistributed without permission.
This work adheres to a rigorous ethical framework to en- sure the responsible development, evaluation, and deploy- ment of multimodal generalists. Below, we elaborate on the key ethical considerations. These measures ensure that General-Bence serves as responsible and inclusive benchmark, contributing to the sustainable and equitable development of multimodal AI systems.
Misuse Potential. We are aware of the potential risks asso- ciated with misuse of multimodal intelligence technologies, such as applications in surveillance or the manipulation of public opinion. To mitigate such risks, we have developed guidelines to encourage ethical use. These guidelines em- phasize the importance of transparency, accountability, and consent in any application or further development of the technologies evaluated in this work.
The benchmark and eval- uation process ensure strict compliance with privacy regu- lations. All tasks and datasets used in General-Bence are carefully curated to exclude personally identifiable infor- mation (PII). To safeguard privacy, any data derived from public sources is anonymized, and sensitive content is fil- tered out. Our procedures align with relevant data protection standards, such as GDPR and CCPA, emphasizing our com- mitment to ethical research practices.
Privacy and Data Protection.
Accessibility and Inclusivity. In alignment with our commitment to fostering inclusivity in the Al research community, all code, tasks, and datasets related to General-Bence are openly available. This ensures that researchers from diverse backgrounds and varying resource levels can equally contribute to, and benefit from, advance- ments in multimodal generalist research.
Data Collection. The dataset for General BBechh is built using publicly available resources or through collabo- rations with contributors who explicitly consented to their data being included. Data collection protocols are designed to prioritize ethical sourcing, ensuring that contributors un- derstand their rights, including the ability to withdraw their data at any time. This ensures transparency and fairness throughout the dataset construction process.
Evaluations and Performance. We clarify that the per- formances of all models reported in this paper-including both specialists and generalists- are influenced by the spe- cific testing environment. This includes factors such as the size and content of the dataset, as well as the parameters used in the reproduced code. As we continuously update the dataset, the evaluation results presented in this paper may differ from those obtained in future versions. We em- phasize that such differences are considered reasonable and expected deviations, and should not raise any concerns. Our leaderboard is open and under active development, and we warmly welcome participation from external practitioners.
Annotator Compensation. Human annotators play a crucial role in ensuring the high quality of the General-Bench dataset. We engage well-trained an- notators, including postgraduate students and crowdsoure- ing professionals, and provide fair compensation for their work. Annotators are either volunteered to contribute, or paid based on the estimated time required to complete spe- cific tasks. All are signed to give their best efforts in data annotation and model implementation to ensure the work quality.
References
OpenAl. Introducing chatgpt. 2022a.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap- tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, 1ttp/2222.1999111 2023.
Bias and Fairness. Recognizing the potential biases in Al systems, we take active measures analyze and mitigate biases related to gender, ethnicity, language, and other so- ciocultural factors within the dataset and evaluation tasks. Diverse and representative data collection practices are em- ployed across multiple modalities and languages. While we acknowledge that complete cradication of bias is challeng- ing, we strive to identify and address biases as the bench- mark evolves.
Jean- Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for sho- learning. In Proceedings of the NeurIPS, 2022.
Intellectual Property Protection. All datasets and tasks included in General-Bent respect intellectual property rights. Data collected from external sources is fully repur- posed and modified, and is used under proper licensing agreements, ensuring compliance with intellectual property laws. Open-sourced models are strictly used according to their licenses. Models evaluated via APIs are handled ac-
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training
10

---

On Path to Multimodal Generalist: General-Level and General-Bench
with frozen image encoders and large language models. In Proceedings of the ICMI., pages 19740-19742, 2023a.
Hao Fei, Shengqiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision Ilm for understanding, generating, segmenting, editing. 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. CoRR, abs/2304.08485, 2023a.
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. arXiv preprint arXiv:2312.02228, 2023.
OpenAI. Gpt-4 technical report. 2022b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. CoRR, abs/2304.10592 2023a.
Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xin- jie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. arXiv preprint arXiv:2312.10033 2023a.
Dong Zhang. Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Em- powering large language models with intrinsic cross- modal conversational abilities. CoRR, (bs/2305.11000) 2023a.
Hanoona Rasheed. Muhammad Maaz. Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. arXiv preprint arXiv:2311.03350 2023.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat- Seng Chua. Next-gpt: Any-to-any multimodal Ilm. In Proceedings of the International Conference Machine Learning, 2024a.
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal Ilm with discrete sequence modeling. arXiv preprint arXiv:2402.0 12226, 2024.
Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26428-26438, 2024a.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Anirud- dha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and ac- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26439 26455, 2024a.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiy- ing Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pages 14398-14409 2024.
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai al. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint 2rXiv:2309.14181. 2023a.
Zhanyu Wang, Longyue Wang, Zhen Zhao, Minghao Wu, Chenyang Lyu, Huayang Li, Deng Cai, Luping Zhou, Shuming Shi, and Zhaopeng Tu. Gpt4video: A uni- fied multimodal large language model for Instruction- followed understanding and safety-aware generation. arXiv preprint ar arXiv:2311.16511 2023a.
Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang, Wenhao Zheng, Zhaorun Chen, Chenhang Cui, Mingyu Ding, Linjie Li, et al. Mmie: Massive multimodal interleaved comprehension benchmark for large vision- language models. arXiv preprint arXiv:2410.1013 2024a.
Shehan Munasinghe, Rusiru Thushara, Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Mubarak Shah, and Fahad Khan. Pg-video-llava: Pixel ground- ing large langu-la models. arXiv preprint arXiv:2311.13435, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reason- ing benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9567, 2024.
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. arXiv preprint arXiv:2406. 19389, 2024a.

---

On Path to Multimodal Generalist: General-Level and General-Bench
Fanqing Meng. Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. Mmiu: Multimodal multi-image understand- ing for evaluating large mssionllanuaae models. arXiv preprint 2rXiv:2408.027//6 2024a.
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Com- mon practices and emerging technologies. IEEE access, 8:58443-5846, 2020.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi- modal model an all-around player? In European Con- ference on Computer Vision, pages 216-233. Springer, 2025.
Pan Zhang, Xiaoyi Dong, Bin Wang. Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, et al. Internim- xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309. 151 2023b.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13299-13308, 2024a.
Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiao- qiang Lei, al. Unified language- pretraining with dynamic discrete visual tokenization. arXiv preprint arXiv:2309 200,0,99,, 2023.
Kaining Ying. Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehen- sive multimodal benchmark for evaluating large vision- language models towards multitask agi. arXiv preprint arXiv:2404. 16006, 2024a.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 18814, 2024c.
Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, Ying Tai, Wankou Yang, Yabiao Wang, and Chengjie Wang. A survey on benchmarks of multimodal large lan- guage models. arXiv preprint arXiv:2408.0866: 2024b.
Hao Fei, Yuan Yao, Zhuosheng Zhang, Fuxiao Liu, Ao Zhang. and Tat-Seng Chua. From multimodal I1m to human-level ai: Modality, instruction, reasoning, ef- ficiency and beyond. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024): Tutorial Summaries, pages 1-8, 2024b.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-chub: A comprehensive evalua- tion benchmark for large vision-language models. arXiv preprint X32223366009665, 2023a.
Hao Fei, Shengqiong Wu, Meishan Zhang, Min Zhang, Tat-Seng Chua, and Shuicheng Yan. Enhancing video- language representations with structural spatio-temporal alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024c.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for inte- grated capabilities. arXiv preprint arXiv:2308.02444 024900, 2023.
Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbren- ner, Andrew Senior, Koray Kavukcuoglu, et al. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 1609039999 12, 2016.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li. Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. CORR, abs/2306. 133944, 2024a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. In International conference on ma- chine learning, pages 8748-8763. PMLR, 2021.
Jaacheng Chen, Tianhao Liang. Sherman Siu, Zhengqing Wang, Kai Wang, Yubo Wang, Yuansheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, et al. Mega-bench: Scaling multimodal evaluation to over 500 real-world tasks. arXiv preprint X1v: 10563, 2024b.
12

---

On Path to Multimodal Generalist: General-Level and General-Bench
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foun- dation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185- 24198, 2024c.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint 1rXiv22303005499 2023b.
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone, arXiv preprint 2xXi:2244.114219 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision- -anguu-l understanding and generation. In Proceedings of the ICML, pages 12888-12900 2022.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge. and Ying Shan. Seed-bench: Benchmarking multimodal Ilms with generative comprehension. CoRR, abs/2307.16125 2023b.
Black Forest Labs. Flux. https: //g//gttuuuu com/ Flack-forest-Lue aaa--aabbssfus 2023.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high- quality end-to-end text to speech. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mm- bench: Is your modti-modal model an all-around player? Computer Vision ECCV 2024 18th European Con- ference, Milan, Italy, September 29-October 4, 2024, Pro- ceedings, Part VI. pages 233, 2024a.
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. arXiv preprint 2100022310.01218, 2023.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talk- ing, drawing and editing with visual foundation models. CoRR, abs/2303.04671, 2023b.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprini arXiv:2403.05532 2024a.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solv- ing Al tasks with chatgpt and its friends in huggingface. CoRR, abs/23303.17588 2023.
Anthropic Team. The claude model family: Opus, sonnet, haiku. preprint, 2024. URL https: //api. www.steeesceeetteeeeoe.. ootppooccccccoo..oooooooooooo 268232499.
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.05437, 2023c.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhu- oshu Li, Hao Yang, et al. Deepseek-vl: towards real- world lasion-language understanding. arXiv preprint arXiv:2403.05555, 2024b.
Kaihang Pan, Siliang Tang, Juncheng Li, Zhaoyu Fan, Wei Chow, Shuicheng Yan, Tat-Seng Chua, Yueting Zhuang, and Hanwang Zhang. Auto-encoding morph-token for multimodal 11m. arXiv preprint arXiv:2405.0192. 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03336,, 2024d.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reason- ing via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507-2521 2022.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- llsoun-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2/11, 2024a.
13

---

On Path to Multimodal Generalist: General-Level and General-Bench
Jingkun Ma, Runzhe Zhan, Derek F Wong, Yang Li, Di Sun, Hou Pong Chan, and Lidia S Chao. Visaidmath: Bench- marking visual-aided mathematical reasoning. arXiv preprint arXiv:2410. 2024.
Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shao- hui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, En- hong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi- modal Ilms in video analysis. CoRR, abs/2405.21075. 2024b.
Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Kr- ishna, Graham Neubig, and Deva Ramanan. Natural- bench: Evaluating vision-language models on natural adversarial samples. arXiv preprint arXiv:2410.1466, 2024e.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. MLVU: A comprehensive bench- mark for multi-task long video understanding. CoRR, 2024a.
Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F. Chen. Audiobench: A universal benchmark for audio large language models. CoRR, abs/2406. 16020, 2024b.
Kunchang Li, Yali Wang, Yinan Hc, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Lou, Limin Wang. and Yu Qiao. Mvbench: A comprehensive multi- modal video understanding benchmark. In IEEF/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 22195-22206 2024f.
Ruibin Yuan, Yinghao Ma. Yizhi Li, Ge Zhang, Xingran Chen, Hanzhi Yin, Le Zhuo, Yiqi Liu, Jiawen Huang, Zeyue Tian, Binyue Deng. Ningzhi Wang, Chenghua Lin, Emmanouil Benetos, Anton Ragni, Norbert Gyenge, Roger B. Dannenberg, Wenhu Chen, Gus Xia, Wei Xue, Si Liu, Shi Wang, Ruibo Liu, Yike Guo, and Jie Fu. MARBLE: music audio representation benchmark for universal evaluation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023b.
Xiuyuan Chen, Yuan Lin, Yuchen Zhang, and Weiran Huang. Autoeval-video: An automatic benchmark for assess- ing large vision language models in open-ended video question answering. In Computer Vision - ECCV 2024- 18th European Conference, Milan, Italy, September 29- October 4, 2024, Proceedings, Part LXVIII, pages 179- 195, 2024d.
Ziqi Huang. Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprehensive benchmark suite for video gen- erative models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 21807-21818, 2024.
S. Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Du- raiswami, Sreyan Ghosh, and Dinesh Manocha. MMAU: A massive multi audio understanding and reasoning benchmark. CoRR, abs/2410. 19166, 2024.
Yuze He, Yushi Bai, Matthieu Lin, Wang Zhao, Yubin Hu, Jenny Sheng, Ran Yi, Juanzi Li, and Yong-Jin Liu. benchh: Benchmarking current progress in text-to-3d generation. CoRR, 10.02277.. 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for mul- timodal large language models. CoRR, abs/2306.13321 13394, 2023.
Junjie Zhang, Tianci Hu, Xiaoshui Huang, Yongshun Gong, and Dan Zeng. 3dbench: A scalable 3d benchmark and instruction -tniingttnngggn dataset. In Proceedings of the Thirty- Third International Joint Conference on Artificial Intelli- gence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pages 1706-1714, 2024b.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng. Siyuan Huang, Yu Qiao, and Ping Luo. LvIm-chub: A comprehensive evalua- tion benchmark for large vision-language models. CoRR, abs/2306.00925,, 2023b.
Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video- bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. CoRR, abs/2311.16103,, 2023.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for inte- grated capabilities. In Forty-first International Confer- ence on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang
14

---

On Path to Multimodal Generalist: General-Level and General-Bench
Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal lmss. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 13084-13094, 2024.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun
Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang. Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-L I: A fully open, vision-centric exploration of multimodal 1lms. CoRR, abs/2406. 16860, 2024a.
Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang, Wenhao Zheng. Zhaorun Chen, Chenhang Cui, Mingyu Ding. Linjie Li, Lijuan Wang, and Huaxiu Yao. MMIE: massive multimodal interleaved comprehension benchmark for large vision-languae models. CoRR, abs/2410.10139, 2024b.
Fanqing Meng. Jin Wang. Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. MMIU; multi- modal multi-image understanding for evaluating large vision-language models. CoRR, abs/2408.02718. 2024b.
Yusu Qian, Hanrong Ye, Jean-Philippe Fauconnier, Peter Grasch, Yinfei Yang, and Zhe Gan. Mia-bench: Towards better instruction following evaluation of multimodal llms. CoRR, aws//200.01555550 2024.
Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang. Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang. Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards mul- titask AGI. In Forty-firsi International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024b.
Yifan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang. Junfei Wu, Feng Li, Kun Wang, Qing- song Wen, Zhang Zhang, Liang Wang, Rong Jin, and Tie- niu Tan. Mme-realworld: Could your multimodal LLM challenge high-resolution real-world scenarios that are difficult for humans? CoRR, abs/2408. 132557 2024c.
Jiacheng Chen, Tianhao Liang, Sherman Siu, Zhengqing Wang, Kai Wang. Yubo Wang. Yuansheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, Dongfu Jiang. Xuan He, Yuan Liu, Hexiang Hu, Xiang Yue, and Wenhu Chen. Mega- bench: Scaling multimodal evaluation over 500 real- world tasks. CoRR, 011/2410.10663 2024e.
Wentao Ge, Shunian Chen, Guiming Hardy Chen, Junying Chen, Zhihong Chen, Nuo Chen, Wenya Xie, Shuo Yan, Chenghao Zhu, Ziyue Lin, Song Dingjie, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jianquan Li, Xiang Wan, and Benyou Wang. Milm-bench: Evaluating multimodal Iims with per-sample criteria, 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint S.t1::2408.001188. 2024b.
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang. Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, and Weisi Lin. Q-bench: A bench- mark for general- general-purpose purpose foundation models on low-level vision. In The Twelfth International Conference on Learn- ing Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024b.
Ben Wang and Aran Komatsuzaki. 6 Billion Parameter Autoregressive Language Model. https://gi com / kingoflllz/ mesh-transformer: jax, May 2021.
GPT-J-6B: A
Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenx- uan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang- Hui Liu, Pan Lu, Chunyuan Li. Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, and Muhao Chen. Muirbench: A comprehensive benchmark for robust multi-image understanding. CoRR, 04.22406094111 2024c.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-1 130b to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
Dingjie Song. Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Bench- marking milms in long context. CoRR, 18532 2024.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15111 2024.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench-2: Benchmarking multimodal large language models. CoRR, 1. 2023c.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403. 17299,, 2024.
15

---

On Path to Multimodal Generalist: General-Level and General-Bench
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang. Dong Yan, et al. Baichuan 2: Open large-scale language models. XrXiv preprint arXiv:2309. 10305, 2023.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 13040-13051, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chathot impressing gpt-4 with 90%* chatgpt quality. See https:/vvicuaa Imsys. org (accessed 14 April 2023), 2(3),,, 2023.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-I: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024b.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mer- ouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-4OB: an open large language model with state-of of-therrrt- art performance. 2023.
Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Ling- peng Kong, et al. Detgpt: Detect what you need via reasoning. arXiv preprint arXiv:2305.14167, 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023d.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024a.
Ao Zhang, Liming Zhao, Chen-War Xie, Yun Zheng, Wei Ji, and Tat-Seng Chua. Next-chat: An Imm for chat, detec- tion and segmentation. arXiv preprint arXiv:231 1,.41,9,499. 2023c.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint ar arXiv:2403.04665, 2024.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of- interest. arXiv preprint arXiv:2307.03660 2023d.
Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, al. Mini-internvl: flexible-transe pocket multi-modal model with 5% parameters and 90% performance. Visual Intelligence, 2(1):1-17, 2024.
Hanoona Rasheed, Muhammad Maaz. Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming- Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 13009-13018, 2024.
Xiaoyi Dong, Pan Zhang. Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. InternIm- xcomposer2: Mastering free-form text-image composi- tion and comprehension in vision-language large model. arXiv preprint arXiv:2401. 2024.
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Buptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024.
Bin Lin, Zhenyu Tang. Yang Ye, Jiaxi Cui. Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe- llava: Mixture of experts for large vasion-language mod- els. arXiv preprint arXiv:2401. 15947, 2024.
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam. Honglu Zhou, Viraj Prabhu, Yu- tong Dai, Michael S Ryoo, et al. xgen-mm (blip-3); A family of open large multimodal models. arXiv preprint arXiv:2408.060 08882, 2024.
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modks models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26763-26773 2024g.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o:
16

---

On Path to Multimodal Generalist: General-Level and General-Bench
One single transformer to unify multimodal understand- ing and generation. arXiv preprint arXiv:2408. /2528, 2024.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen- audio: Advancing universal audio understanding via uni- fied large-scale audio- language models. arXiv preprint arXi:2231 2023.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmen- tation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10733 2024.
2024.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xix- uan Song. et al. Cogvlm: Visual expert for pretrained lan- guage models, arXiv preprint arXiv:2311.0307, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lec. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024b.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370- 387. Springer, 2025.
Haobo Yuan, Xiangtai Li, Tao Zhang. Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuse Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv, 2025.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General lan- guage model pretraining with autoregressive blank in- filling. arXiv preprint arXiv:2103. 10360, 2021.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Ad- vances in Neural Information Processing Systems, 36: 20482-20494, 2023.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision- language models: insights and future directions. In Work- shop on Responsibly Building the Nexi Generation of Multimodal Foundational Models, 2024.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiang- miao Pang, and Dahua Lin. Pointlim: Empowering large language models to understand point clouds. In Euro- pean Conference on Computer Vision, pages 131-147. Springer, 2025.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.000 40356699, 2024a.
Ziyu Zhu, Xiaojian Ma. Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IFEE//VE International Conference on Computer Vision pages 2911-2921, 2023b.
Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jian- chao Tan, Quzhe Huang, Bin Chen, Chenyi Lei, An Liu, et al. Unified language-vision pretraining in 1lm with dynamic discrete visual tokenization. arxiv 2024, arXiv preprint arXiv:2309.04660
Zixiang Zhou, Yu Wan, and Baoyuan Wang. Avatargpt: All-in-one framework for motion understanding planning generation and beyond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1357-1366, 2024b.
Boyang Zheng, Jinjin Gu, Shijun Li, and Chao Dong. Lm4lv: A frozen large language model for low-level vision tasks, arXiv preprint 24X1:22405.15744 2024.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign lan- guage. Advances in Neural Information Processing Sys- tems, 36, 2024b.
Yikang Zhou, Tao Zhang, Shilin Xu, Shihao Chen, Qianyu Zhou, Yunhai Tong, Shunping Ji, Jiangning Zhang. Xi- angtai Li, and Lu Qi. Are they the same? exploring visual correspondence shortcomings of multimodal 1lms, 2025.
Yaqi Zhang. Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: motion generators. arXiv preprint arXiv:2306.10900. 2023e.
Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi- lims to 1000 images efficiently via a hybrid architecture. arXiv preprint arv9 200.99889..0 2024d.
Finetuned llms are general-purpose
17

---

On Path to Multimodal Generalist: General-Level and General-Bench
Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Ki- Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. GAMA: A large audio-language model with advanced audio under- standing and complex reasoning abilities. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Prucessing, pages 6288-631 2024.
Xichen Pan, Sarya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, et al. Transfer between modalities with metaqueries. arXiv preprint arXiv:2504.06256 2025.
Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huamning Wang. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems, 36:18090-18108, 2023.
Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng. Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, et al. Wavllm: Towards robust and adaptive speech large language model. arXiv preprint arXiv:2404.00656, 2024b.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large lan- guage models. arXiv preprint arXiv:2310. 13229 2023.
Rongji Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang. Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Zhou Zhao, and Shinji Watanabe. Audiogpt: Understanding and gen- erating speech, music, sound, and talking head. CoRR, abs/2304. 12995, 2023.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model instuuciion-follot them all. CoRR, abs/2305. 16355, 2023.
Jaaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruc- tion tuning, arXiv preprint arXiv:2309.03900 2023.
Xinyu Wang, Bohan Zhuang, and Qi Wu. Modaverse: Effi ciently transforming modalities with llms. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26606-26616, 2024e.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. CoRR, abs/2205.15868. 2022.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuo- han Li. Dacheng Li, Eric Xing, et al. Judging llm-as-a- judge with mt-bench and chatbot arena. Advances Neu- ral Information Processing Systems, 36:46595 46623, 2023.
18

---

On Path to Multimodal Generalist: General-Level and General-Bench
On Path to Multimodal Generalist: General-Level and General-Bench
Full-version Paper: https://arriv..org/as//2555 04620
Project Page: bteehttsss/geeeraiistt..oo
Leaderboard: http:://gaeeaass..tt//aaaebbbbbe
Benchmark: http:sss//onggnnaacc.cc/eeeeeeee
Is your MLLM a well- generalist?
Level 5: Generalists with total synergy across
Comprehension. Generation and Language
0.11
Vicin- 1.18
GininG 1,50
Level 4: Generalists with synergy across Comprehension and Generation
Emar 37B 7B
3,65
2,67
2 4.31
4.81
VL 26- !
Sa2VA 268 6
Level 3: Generalists with synergy Comprehension Generation
GPT4O
Qwam2- VZ-78 Dwen2-V2--2B
GPT 4096
Inter InterV V.255.B
SazVA
Mins--Gnmiii Gemiini
Qwena- Paema-VE--7- VL-72B
LLaVA LLaVA-OOe- One--in----immnnee Vinian-72B 72B
Jat
7.73
8.41
10.80
11.78
1221
ImageBinn--LLA 4 12:40
12.25
NEXT-GPT-VLS 13.02
Unified-ion- XXL 13.70
13.04
Level 2: Generalists
Unified Comprehension and Generation
Emau Emm22-37B 37B
Vntern 5-B.5BB
VidAgent
PandHGPT-1SB
Vibron VI
Moda Verse- Th.0
AmyGPT
CLIP
FLUX
Hunyaan Video
DINO
FideoFlow
SINY
SAM2
IDGS
3DETR
Naay
PastSpoech??
Wove
Thcatron Dogapaacc
Level 1: Specialists
Figure 4: Leaderboard of multimodal generalists over Seneral-Level (only top performifngg ones shown here).
Table of Contents
A Extension on General-Level Scoring
21
A.I Scoring Specification
21
A.2 Scoring Relaxation
21
A.3 Properties of General-Level.
22
19

---

On Path to Multimodal Generalist: General-Level and Seneral-Bencl
A.4 Receipt Leveling Upper in General-Level
25
Extension on General-Bench Dataset
25
B.1 Data Construction
26
B.1.1
Design Criterion
26
B.1.2
Construction Process
26
B.2
Evaluation and Splitring.
27
B.3
Leaderboard Re-Scoping
27
B.4 Evaluation Metrics
27
B.5 Data Format
33
B.6 Data Taxonomy and Hierarchy
35
B.7 Data Distributions
40
B.8 Extended Data Insights
41
B.9 Comparisons with Existing Benchmarks
42
Extension on Experimental Results
45
C.1 Multimodal Specialist and Generalist Systems
45
C.2 Full Main Evaluation Results
C33 Level and Leaderboard of Multimodal Generalists
................................
C.4 Capability BreakDown
C.5 Analysis and Discussion on Synergy
56
C.6 Results of Image-related Tasks
59
C.7 Results of Video-related Tasks
81
C.8 Results of Audio-related Tasks
89
C.9 Results of 3DD-rllted Tasks
91
C.10 Results of NLP Tasks
93
D Discussions and Future Investigation
115
Author Contribution
116
20

---

On Path to Multimodal Generalist: General-Level and General-Bene
Generation Task Group
Ec specific task
NLP Task
Group
Comprehension Task Group
Language
Image
Video
udio
3D
Figure 5: We categorize tasks of various modalities into Comprehension group, Generation group and NI.P group. Each colored stylish symbol represents a specific task of certain modality.
Extension on General-Level Scoring
A.1 Scoring Specification
Assume we have a benchmark of various modalities and tasks, where we can categorize tasks under these modalities into the Comprehension group and the Generation group, as well as the language (i.e., NI.P) group, as illustrated in Figure Then, we define the scoring specification of General- -Leeel in Table 1.
Also, except for Level-1 1 and Level-5, when calculating S2, S3, and S4, consider reasonable approach when handling different modalities. First, we calculate the specific score component St. of generalist in the i-th modality (assuming there are N modalities in total) for the score Sg. This modality-specin component can accurately reflect the model's Level-k capability in the i-th modality. Next, by decomposing each score into its components across different modalities, we sum the components of each modality with equal weights to obtain the overall score for each level.
N S = Σ
S
The advantage of this method is that it reduces the bias introduced by the number of tasks in different modalities. For example, in our benchmark, image-related tasks (especially comprehension-type tasks) are overwhelmingly more numerous compared to other modalities, such as audio tasks. Therefore, two generalists with similar capability levels, say one for image tasks and the other for audio tasks, would have a higher S₂ score for the senge-generalist over the audio-generalist, due to the larger number of image tasks. This discrepancy is unrealistic and contrary to our core idea for evaluating multimodal generalists. To eliminate the bias caused by the number of tasks within each modality, we propose the above calculation method, which treats the capabilities of different modalities equally. Meanwhile, this method also prioritizes generalists that can support more modalities, For instance, model that supports more modalities will certainly have higher overall score compared to generalist that supports only one modality.
This scoring method ensures that an MILM climbs to higher levels, its scores progressively decrease, which should indicate the increasing difficulty of advancing levels. Climbing from level n to level + requires specific capabilities, ie., demonstrating sufficient synergy capability associated with that level, which we highlight as critical factors Table 1. Within the same level, to achieve a higher score, a model must: 1) support as many tasks and modalities as possible, and simultancously 2) achieve the highest possible performance on individual tasks.
A.2 Scoring Relaxation
central aspect of our General--eeee framework lies in how synergy effects are computed. According to the standard understanding of the 'synergy' concept, e.g., the performance of generalist model on joint modeling of tasks and B (e.g.. P, B)) should exceed its performance when modeling task A alone (e.g., A))y))) or task alone (e.g., Po ()). However, adopting this approach poses significant challenge that hinders the measurement of synergy: there is no feasible way to establish two independent distributions, Po(yl A) and (y(y) B), and joint distribution Po A, B). This limitation arises because a given generalist model has already undergone extensive pre-training and fine-tuning, where tasks A and B have likely been jointly modeled. or B independently in order to derive these distributions. Otherwise, such an approach would result in excessive redundant computation and inference on the benchmark data.
impractical to retrain such a generalist to isolate the learning and modeling of tasks
21

---

On Path to Multimodal Generalist: General-Level and General-Bench
To simplify and relax the evaluation of synergy, introduce key assumption in the scoring algorithm:
Theoretically, we posit that the stronger model's synergy capability, the more likely it is to surpass the task perfor- mance of SoTA specialists when synergy is effectively employed. Then, we can simplify the synergy measurement as: a generalist outperforms a SoTA specialist in specific task, consider it evidence of a synergy effect, i.e., leveraging the knowledge learned from other tasks or modalities to enhance its performance the targeted task.
By making this assumption, we avoid the need for direct pairwise measurements between 'lask-task', 'comprehension-
generation', or moodalitymoddllity,, which would otherwise require complex and computationally intensive algorithms.
A.3 Properties of Leneral-Level
The General-Level framework possesses several important attributes that play a critical role in supporting the hierarchi- cal classification and ranking of MLLMs. These properties are also well-grounded in mathematical theory.
Property-1: Independence from Peer Generalists In our scoring framework, the scores of any generalist depend solely on the dataset and the reference scores of SoTA specialists, without relying on the scores of other tested generalists. These two components are entirely independent. The dataset defines the specific tasks, while the specialists provide baseline reference scores used for the calculation of the experimental generalists' scores. This property ensures that the evaluation of generalists is free from interdependence, maintaining objectivity and fairness among all systems participating in the ranking.
Monotonicity Across Levels Generally, if a generalist is rated at the highest level-k, it is expected to achieve scores at all levels from 2 to k. We further expect that as the level increases, the corresponding scores for the generalist will decrease, i.e., Sk- > Sg. This is reasonable and realistic requirement, as higher levels impose stricter demands on the generalist's capabilities, naturally leading lower scores for the same model. Below, provide proof that the scoring algorithm of General-Lewe
Property-2:
framework mathematically guarantees the strictly monotonic score decline across levels.
The proof for S3 S S2
S₂ (Sa + Sc)
2
M
of
of 2 oora
N
2ota otherwise
if
2
M
0
otherwise
N 1=1
0
1=1
1
0
2
M
N
5=1
S2
The proof for S4 < 83
Suppose:
M
So M
Σ
o if 2 ssta otherwise
2=1
1 Se N
o 2 asota otherwise
0
According to Cauchy-Schwarz Inequality, let's represent
Sc + So
2 2
SS Sc + Sa
2
Expanding this,
(Sc + Sa)
2ScSa
4
Se + SG
22

---

On Path to Multimodal Generalist: General-Level and General-Bende
Multiplying both sides by 4(Sc + Sa),
(Sc + Sa)* SSc + Sc)
Simplifying further
S6 + S/ 2 Sc + Sa)
This factorizes
(Sc (Sc + Sc) 2 0.
Finally, we have
SScSc
S4 Sc + SG
(Sc -oo)
S3.
The proof for S₃ 84
We have
St WL Swai
where
Oz if o 2 otherwise
St. =
which means,
UL 11
Then
Ss S4 S4 wt, S4
S4 * (wi 1) 2 0
Thus,
S - S4 5
Property-3:
Encouraging Rich and Balanced Multimodal Task Support.
More Task, The Better. A good multimodal evaluation system should not only reward models for achieving higher scores on individual tasks and surpassing SoTA specialists but also incentivize a trend where multimodal generalists support as many diverse multimodal tasks as possible. This is a reasonable expectation, as an ideal multimodal generalist should inherently support a broader range of modalities and tasks. The scoring algorithm of our General-Level framework aligns with this objective. For instance, in the of level-2 scoring:
M+N Σ o,
S M+N
model that achieves nonzero scores across a greater number of modalities and tasks will naturally obtain higher average score, thereby ranking higher within the same level.
More Balance, The Better. Moreover, our scoring algorithm also promotes models that achieve more balanced performance across tasks. For example, in the case of level-4 scoring, consider the following scenarios:
1) Model achieves SoTA specialist performance on tasks in the comprehension category but only Y tasks (where
X > Y) in the generation category.
2) Model B achieves SoTA specialist performance on X tasks both the comprehension and generation categories.
23

---

On Path to Multimodal Generalist: General-Level and Ceneral-Benchs
According to the properties of the harmonic mean inequality, S < S..
The proof for S Sf when X >> Y in level-4
Extreme Assumptions:
For Model A. the X tasks in the comprehension group have scores of of = 1, and the Y tasks in the generation group have while all other scores are O.
scores of og
For Model B, both comprehension and generation groups have X tasks with scores of o = 1 and o = 1. while all other
scores are 0.
Model-A Scores:
Model A, the comprehension and generation scores are:
X M
96
N
The overall score for Model
SA
S4
2
2XY
S
88
S4
+
XN
YM
Model-B Scores: For Model B, both comprehension and generation groups have X tasks with scores of 1, so:
0 = M
N
overall score for Model
88 28
2
X2 XN + XM
S
SB 5
X +
Comparison:
We need to compare:
2XY XN + +YM
X2 +
and
Given X >> Y. it follows that:
2XY
X²
XN +YM
XN+ XM
Thus, St S.
Through the above mathematical analysis, we have proven that under the same task distribution, the uneven generation score distribution of Model A results in its level-4 score being lower than that of Model B. This ensures that models with more balanced performance across comprehension and generation are ranked higher.
Property-4: Dynamic Update Benchmarking and Specialists Finally, observe important point: the more tasks included in the benchmark used to evaluate models, the more accurate and objective the resulting evaluations and conclusions. This requirement for the evaluation benchmark to have dynamic properties aligns well with real-world needs. In practice, new tasks, data, and even new modalities are constantly being introduced, and generalist should be capable of covering these newly added tasks and functionalities. Accordingly, in our evaluation system, we allow the benchmark to evolve dynamically, such as by adding new tasks under various modalities and categories. Once new tasks are added, we update the scores and rankings of all tested generalists reflect the expanded benchmark.
On the other hand, also allow updates to the SoTA specialist models timely for each task, scoring at higher levels is anchored to the performance of the SoTA models. This is a reasonable act, as specialists are continually being developed and improved. Once baseline specialist advances, generalists must also improve to remain competitive, or risk being surpassed. framework, the scores corresponding to SoTA specialists are subject to periodic updates. Also,
Thus, Ceneral-Leva we dynamically and regularly update the scoring and ranking of all generalists to ensure the evaluation remains accurate and reflective of the current state of the field.
24

---

On Path to Multimodal Generalist: General-Level and General-Bench
A.4 Receipt to Leveling Upper in General-LLevl
Here we provide guideline to help better understand how to achieve higher levels in General Level framework.
Supporting as many tasks and functionalities as possible. Transitioning from specialists to gen- eralists requires making the system compatible with various task modeling paradigms, i.e., supporting diverse modality types and input formats, as well as handling a wide range of model types and output formats (whether for comprehension and/or generation). Currently, the most popular and widely adopted practice to use an LLM as the backbone/imtelligenee medium, integrating various specialists to build generalists. There are two primary implementation strategies.
First, agent-based generalists (Wu et al., 2023b; Shen et al., 2023). In this approach, the LLM acts as a task scheduler and dispatcher, facilitating message passing through hard integration (explicit text). This is essentially a pipeline architecture. However, since gradient propagation across the entire system is not feasible, this method is prone to error propagation. The performance upper bound of generalists built with this approach is equivalent to the SoTA specialists for all supported tasks, primarily due to the lack of features, information sharing, and limited task collaboration.
Second, end-to-end generalists (Liu et al., 2023c; Li et al., 2023a; Zhu al., 2023a). In this type, the entire system is constructed as a continuous joint model, allowing for full-stack updates via gradient propagation. The most common architecture in this category uses an LLM as the backbone, achieving soft integration of various encoders and decoders through input tokenization and feature embedding, combined with overall fine-tuning.
Level-2 + Level-3: Generalists achieving as stronger synergy and cross as many tasks as possible. To advance from vanilla generalist Level-3, the system must demonstrate cross-task synergy capabilities, enabling at least two tasks (regardless of whether both involve comprehension, generation, or one involves comprehension while the other involves generation) to share features and achieve mutual performance improvements. The most direct method to realize cross-task synergy is through multi-task joint training. Specifically, during joint learning, the system must ensure it can maintain sessessaarddeeesisteenttt common features while preserving each task's 's specific features without degradation, e.g., Vitron (Fei et al., 2024a). Moreover, the model must support synergy across as many tasks as possible and ensure that the synergy effect significant enough to achieve higher evaluations at Level-3.
Generalists in unified comprehension and generation capability with synergy in between. To advance to Level-4, generalists must first achieve unified comprehension and generation capabilities, regardless of whether they support a single modality (non-NLP) or multiple modalities. At the same time, the system must meet the requirement that its capabilities in comprehension and generation synergize and enhance one another. Generally speaking, compared to acquiring comprehension capabilities, obtaining generation capabilities the technical level is relatively more challenging. For instance, the visual comprehension abilities of most visual LLMs tend to be significantly stronger than their visual generation capabilities. If a generalist can score at Level-4, it indicates that the system not only possesses strong comprehen- sion capabilities but also maintains these capabilities while further learning and training its generation abilities. To achieve this, Morph-Tooee Token (Pan et al., 2024) introduces a disentangling visual reconstruction loss for generation learning to avoid interference with the comprehension learning loss.
Level=3-:Level-4:
Level-4-LLvel-5: Generalists achieving cross-modal synergy with abductive reasoning ability. Achieving Level-5 represents the ultimate goal for generalists, where features, knowledge, and even intelligence learned from tasks in certain modalities can (to varying degrees) transfer to tasks in other supported modalities. Currently, most multimodal generalists are limited by architectural developments, primarily enabling language intelligence to support intelligence in other modalities (as illustrated in Figure 1). However, to truly achieve Level-5, synergy must exist across all modalities. For instance, in the current MLLM community. this would require MLLMs to enhance performance on NLP tasks as well, while most of the MILMS perform unsatisfactorily in NLP tasks. From technical perspective, generalists must be capable of abductive reasoning, i.e., the ability to infer and generalize across everything. Also, they need to ensure modality-agnostie context consistency during reasoning.
B Extension on Gencral-Bench Dataset
This part provides an extension to our General Mateeral--Beee -Bench dataset.
25

---

On Path to Multimodal Generalist: General-Level and General-
Scope Definition
2. Task List Curation
Data Collection
Data Cleaning
5 Inspection Validattion
?S?
DO
Establishing detaset scope, incluting modalities, meto tasks, and prediction tion paradigms.
Seanching sources (Seogle, 6itHub, Kaggle, ArXiv, etc.)to condinm well-defined task list.
Collecting dana by sourcing from existiin benchmarks ks (Case A) and manual or creation (Cose B).
F Filtering low qualityy irrelevant datee instunces. and formetting datesets uniformll.
Group Cross validation by arnotaters and finel verification by Team leaders.
Figure An illustration of the data construction pipeline of General-Bence
B.1 Data Construction
B.1.1 DESIGN CRITERION
As previously noted, the current benchmarks that rank MLLMs based solely on their performance have significant limitations, which hinder the encouragement of MLLMs to evolve toward becoming more capable multimodal generalists. Primarily, nearly all existing benchmarks focus on evaluating MILMMs capabilities in visual modalities, particularly images, while significantly neglecting tasks in other modalities such as video, audio, 3D, etc. Moreover, they often assume that MLLMs already possess satisfied NLP capabilities, thus omitting evaluations in language.
Secondly, these benchmarks tend to simply convert free-form predictions into fixed QA format of pre-defined essenil-essentiallly compromise that reflects the current limitations of MLLM capabilties--alloinng many tasks that MILMs cannot produce in specific formats to still be executed. We believe that genuine multimodal generalist should support tasks in their original formats. Furthermore, most benchmarks only assess MLLMs' understanding of visual information; however, multimodal generalist should inherently possess a wide range of capabilities beyond mere comprehension, such as generation, editing, etc. Therefore, we expect to construct a benchmark that possesses these characteristics:
Covering as broad a range of tasks, skills and modalities as possible. Encompassing both comprehension and generation of tasks.
Including rich diversity of tasks across various scenarios and domains.
Preserving the original lask-prediction formats.
Timely maintaining and expanding the dataset dynamically.
B.1.2 CONSTRUCTION PROCESS
The construction of our General cB dataset follows a structured 5-step process to ensure both comprehensiveness and quality. Figure presents the data construction pipeline.
Step-1: Defining Scope and Range. We begin by conducting a series of panel discussions to establish the scope of the dataset. This involves determining the modalities to include, identifying the core general skills (meta-tasks), and specifying the prediction paradigms to address. These discussions help outline comprehensive framework for the dataset, ensuring that it accommodates diverse tasks and capabilities required for evaluating multimodal generalists.
Step-2: Curating Task List. Based on the defined scope, we curate comprehensive task list by systematically searching various sources, including Google, GitHub, Kaggle, ArXiv, and PaperWithCode, etc. For each task, we specify its input- output targets, select appropriate evaluation metrics, and also identify SoTA specialists as reference points. This step ensures that each task is well-defined and aligned with existing SoTA practices.
Step-3:
Collecting Data. Next, we start collecting the data instances. The data collection process is divided into two cases for handling two different scenarios:
Case A: If the data could be sourced from existing benchmark datasstss (only from their test sets), modifications are made to enhance diversity. We will show all the data sources of our benchmark in the following subsections. For textual data, rephrasing is done using ChatGPT. For non-textual modalities such images, videos, and audio, semantically equivalent replacements are identified through retrieval or direct recording from relevant databases or websites.
Case B: For tasks without available datasets or insufficient enough numbers of samples, we manually create instances. This involves crafting input-output pairs according to the task definition, running existing models to generate predictions, and performing manual verification and correction of the results.
We ensure that each task includes (at least) 500 data samples. Also, we ensure that all tasks faithfully retain their original
26

---

On Path to Multimodal Generalist: General-Level and General-Bench
input-output prediction structure format, i.e., not reformatted into QA-based multiple-choice questions.
Step-4: Data Filtering and Cleaning. After collecting datasets for all modalities and tasks, we proceed with data filtering and cleaning. First, we filter out low-quality instances, including those that do not align well with the task's 's evaluation purpose, lack target modality information, or fail to meet the defined prediction paradigms. For tasks where the number of instances is insufficient, we restart the data annotation process to supplement the required quantity. Afterward, we organize all data into a unified storage format according to the designed specifications. For example, textual data is standardized into ISON files with consistent naming conventions applied to all files.
Step-5: Data Inspection and Validation. Finally, we conduct a rigorous inspection and validation process to guarantee data quality and consistency. Annotators work in groups of three, independently reviewing the same instance. An instance accepted only if all three annotators reach consensus. Finally, team leaders or supervisors conduct an additional round of verification to ensure the dataset meets the highest standards of consistency and accuracy.
B.2 Evaluation and Splitting
As each task follows the original format, our evaluation metrics vary in rich task types. For instance, we evaluate X-to-text generation tasks using GLEU/RUGG/CIIDE scores, image segmentation tasks with mloU for generating masks, and image generation tasks using FID, etc. Also, we design some mapping functions to standardize performance scores. In Appendix $B.4 we present the evaluation metrics as well as the mapping tricks in detail.
For most of the tasks, we maintain around 500 testing instances each. Considering that not all practitioners in the community may be interested in participating in the leaderboard for example, some may simply wish to use our dataset for their research or publications- we propose dividing the test set for each task into a closed set and an open set. The closed set reserved for leaderboard evaluations: only the input data is released, and users are required to submit their model's predicted outputs for centralized assessment. In contrast, the open provides full access both inputs and corresponding outputs, enabling practitioners to explore and utilize the data more freely. Each task's 's test set is split into closed and open subsets with a ratio of 2:3.
B.3 Leaderboard Re-Scoping
Given the large scale of our dataset, it would be highly costly for practitioners to run the entire dataset under our proposed General-Level evaluation protocol. Moreover, it's realized that most existing multimodal generalists (e.g., MLLMs) have not yet reached the level of capability required to cover a wide range of modalities and tasks, as envisioned in our framework. As a result, many current models may find it difficult to fully demonstrate their potential on our leaderboard. To improve usability and encourage broader participation, we further propose graded structure for the leaderboard by dividing its scope into four levels of increasing difficulty:
Scope-A: Full-spectrum leaderboard covering all modalities and tasks, designed for highly capable, general-purpose multimodal models. This scope has one leaderboard encompassing all levels in General-Level, making it the most challenging track. We further derive a full version and a quick version leaderboard for casier participation.
Scope-B: Modality-specffic leaderboards, each focusing on a single modality or partially joint modality, and designed for modality-wise generalists. This scope maintains 4 separate leaderboards, one per modality (except for language). Scope-C: Leaderboards focused on either comprehension or generation within single modality. This scope includes 8 leaderboards: 2 4 for comprehenssion/geeeratio across multimodal tasks, with a lower entry barrier for participation. Scope-D: Finer-gra graineg skill-level teass-cluttersseecif leaderboards within each modality, tailored for partial generalists. This scope includes a large number of specific leaderboards, offering the lowest difficulty for participation.
Figure 7 illustrates this design. Each leaderboard scope reflects a different level of difficulty, allowing practitioners to flexibly choose which leaderboard to participate in based on the capabilities of their models and the amount of resources they are willing to invest.
B.4 Evaluation Metrics
Metric List. Since all tasks in General-Bence retain their original task definitions without altering the output or prediction format, our evaluation methods vary according to the nature of different tasks and data. Table 4 summarizes the evaluation metrics and methods used across all tasks.
27

---

On Path to Multimodal Generalist: General-Level and General-Beneh"
Leaderboard Scope-A: A: Full speccruum Hero
Full-spectrum leaderboard covering all modalities and tasks, for highly capable, general-purrpee purpose multimodal models.
Leaderboard Scope- B: cedaality-speciff Unified Hero
Modality-specific leaderboards focusing single modality (or partially joint modality) for modality-wise generalists.
Leaderboard Scope Comprehension//Generttii Hero
Leaderboards of comprehension or generation under one single modality.
Leaderboard Scope-D: -pill-specific Hero
Finer-grained, skill (llustrr--pecifii leaderboards under each modality, for partial generalists.
# Boards:
Hard:
Figure 7: We reorganize General-Bencl into 4 scopes, categorized by the level of participation difficulty for practitioners.
Table 4: Overview of all the evaluation metrics in General Searra--Beedde -ennh. means the higher the better performance, and vice
for 4-
#
Metric
Range
Calculation
Representative Tasks
General
Accf
[0.1] Accuracy is defined as the ratio correctly classified instances to the total number of instances.
Classification
Macro-Acef
[0.1] Macro-Ace evaluates well model performs on average across all classes, Relation Prediction regardless of class imbalance.
EM-Acet
[0.1] Exact Match Accuracy evaluates the percentage predictions are exactly the QA, machine translation, summarization
same their corresponding references.
APt
[0,1] AP, Average Precision, is a metric used to evaluate the performance of object detection
Anomaly Detection
tasks, reflecting the overall precision-recall trade-off across multiple thresholds.
1
[0.1] mAP, Mean Average Precision, is the mean of Average Precision values across all 2D/3D Detection
queries instances:
6 FIf
[0,1] Fl score is the harmonic mcan of Precision and Recall.
QA
Micro-FIt
Micro-F1 score the harmonic of the averr-ed precision and recall.
Classification
8 AUCt
[0.1]
AUC is used in binary classification tasks and measures the area under the ROC curve. It represents the model's ability to distinguish between classes.
Image Generation
Ranking-related
9 R@Kt
[0,1| R @k measures the Recall rate at the top k results in tasks like image retrieval, where Image Scene Graph Pars-
the true positive must appear within the top k predicted results.
ing
10 AP@Kt
[0.1] AP@k is the Average Precision calculated at an IoU threshold of k (k;1). This metric Object Detection
is typically used when higher overlap between retrieved items and ground truth items
is required.
mAP@kt
[0.1] mAP@k refers the mean Average Precision where the Intersection Union Object Detection (loU) threshold is set to k l))
EM@11
10,11 Exact Match evaluates the proportion instances for which the model's top 3D Question Answering
prediction exuctly matches the correct unswer.
ANLSS
[0.1] ANLS, Average Normalized Levenshtein Similarity, measures how well a model OCR ranks items in list based their relevance to query.
Regression-related
14 4
[0,cc) MAE, Mean Absolute Error, measures the average of the absolute differences between Object Counting
the predicted values and the actual values. It's typically used in regression tasks.
RMS 4
[0,cc) RMS, Root Mean Square, metric regression
measures the square Image Depth Estimation
root of the average squared differences between the predicted values and true values.
MSE
[0,cc) MSE, Mean Squared Error, commonly used regression tasks and measures the Object Matting
average squared differences between predicted values and actual values. RMSE, Root Mean Squared Error.
RMSE
[0.oc)
Time Series Prediction
Text Generation-related
28

---

On Path to Multimodal Generalist: General-Level and General-Benel
# Metric
Range
Calculation
Representative Tasks
BLEU-I1
[0.1]
BLEU-I, Bilingual Evaluation Understudy (1-grum), calculates the precision
Text Generation
unigrams (individual words) in the generated text compared to the reference text(s). BLEU-4, Bilingual Evaluation Understudy (4-gram).
19 BLEU-4f
10,11
Text Generation
20 CodeBI.UUt
[0,1]
CodeBLFUU is metric designed to evaluate the quality of generated code by com-
Code Generation
paring it to reference code. CodeBLEU combines standard BLEU with additional features specific to code, such syntax matching, data flow alignment, and weighted D- n-gram matching.
21 ROUGE-LI
[0.1] ROUGE, Recall-Oricnted Understudy for Gisting Evaluation of Longest Common Image/ Mangge/Viieoooo Captioning
Subsequence, LCS, evaluates text generation tasks, which measures the overlap between predicted text and the reference text.
22 ROUGE-It
10,11
ROUGE in 1-grams (single words).
Text Generation
CIDErt
[0.1]
Consensus- based Image Description Evaluation (CIDEr), evaluates the quality of gen- Captioning erated sentences (e.g., image captions) by comparing them against set of reference sentences.
Image-related
24 PSNRT
(0,cc) PSNR, Peak Signal-to-Noiee Ratio, measures the quality of a reconstructed or com- Image Desnowing pressed signal, such images or videos, compared to the original signal.
25 MS-SSIMI
[-1.1] MS-SSIM, Multi-Scale Structural Similarity Index Measure, a metric used for Document
Image
Un-
image quality assessment that measures the structural similarity hetween two images warping across multiple scales.
CLIP-Scoref
[0.1] The CLIP score measures the similarity between an image and a textual description Image Editing using the CLIP model, commonly used for image-text matching tasks.
FID 4
(0,cc) FID, Fr cchet Inception Distance, measures the distance between two multivariate Text-to-Image Gaussian distributions: one representing the features of real images and the other tion representing the features of generated images.
Genera-
28 SAD/
[0,cc) SAD, Sum of Absolute Differences, measures the total absolute difference between Object Matting the predicted and ground truth values for all pixels in an image or region. is typically how closely the model replicates fine-grained
used in image matting tasks image details such as edges and textures.
Video-related
29 Fram-Acet
[0.1]
Frame-level Accuracy evaluates how many frames (or time steps) in the sequence are Video Translation correctly classified by comparing predictions with the ground truth on frame-by- frume basis.
30 FVD +
(0,cc) Fréchet Video Distance (FVD) is a metric used to evaluate the quality of generated Video Generation
video sequences, extending the principles of the FID to videos. FVD calculates the distance hetween the feature distributions of real and generated videos, taking into account both spatial and temporal dynamics.
31 MUSIQ1
[0.1] MUSIQ. Multi-scale Image Quality, quantifies the distance between the feature Video Superresolution
distributions real and generated videos.
32 absRel +
(0,c0) absRel (Absolute Relative Error) is a metric commonly used in depth estimation tasks,
Vidco Depth Estimation
which measures the average ratio of the prediction error to the ground truth depth.
33 EPE/
(0,cc End-Point Error (EPE) is metric commonly used inoptical flowtasks to evaluate the Optical Flow accuracy of predicted motion vectors (flow) between consecutive frames in a video or image sequence. It measures the Euclidean distance between the predicted and ground truth flow vectors at each pixel. providing an average error across the image.
34 DINO-Scoret
[0, The DINO Score is calculated as the cosine similarity between the DINOv2 class Subject-Driven embedding of two frames, effectively measuring the consistency of subject's 's identity Generation across frames. According the DreamBoott paper, the DINO Score captures more detailed aspects of subject identity compared to the CLIP Score.
Image
35 LI-Dist
[0. LI-Dis measures the L1 distance between two consecutive frames, evaluating the Static Video Generation model's ubility to generate static (temporally stable) videos by quantifying the
differences between adjacent frames.
LI-Dis 255-21,
255
36 OFSt
[0. 1] OFS. Optical Flow Score. captures the movement of pixels between two consecutive Dynamic Video Genera-
frames. By applying a threshold identify "moving pixels" based on optical tion
flow, calculate the ratio of these moving pixels. This ratio quantifies the degree of dynamism in the generated videos.
29

---

On Path to Multimodal Generalist: General-Level and General-Bende
#
Metric
Range
Calculation
Representative Tasks
Aesth-Scoref
[0. 1]
The aesthetic score (Aesth-score) calculated using a ViT with a linear head, as implemented in improved aesthetic predictor. It has similar calculation in CLIP score.
Artistic Content Text-to- Video Generation
38 Suc-Ratet
[0.
Successful Rate (ate) measures the performance of Video Generation. Suc-Rate leverages an open- vecablacyy detector to identify the subjects specified in the text prompts. If all the subjects corresponding frame as a successfully generated video frame, and then calculates the ratio of successful frames the total number of generated frames.
Multi-Class Saa--Clls-Coddttinee
Text-to-Videe Genera-
prompt are successfully detected, classifies the
tion, Spatial Relation
Video Generation, Cam-
era Motion Generation
39 ViCLIP-
[0. 1] We calculate the cosine similarity between the ViCLIP embeddings of text prompts and videos. Compared the CLIP Image model, ViCLIP enables a more com- prehensive assessment of the video's style and its overall consistency with the text prompts.
Image-to- -odeeo Genera-
Scoret
40 A CLIP+OFS +MSS)
[0. 1]
For the image-to- todgeeto-video generation task, we conduct a comprehensive evaluation based on Subject Consistency, Background Consistency, Motion Smooth and Dynamic tion Degree. We utilize the DINO-Score and CL.P--Score assess subject and background consistency, reflecting the model's ability to adhere to the image prompt. To evaluate the motion smoothness, measure the Motion Smooth Score (MSS) of the frame- by-frame motion prior via video frame interpolation models. Also, we employ the Optical Flow Score measure the dynamic degree of the generated videos, ensuring that our metrics do not favor static videos.
Image-to- Video Genera-
3D-related
41 AMOTA t
AMOTA (Average Multi-Objec Tracking Accuracy) is performance metric used to 3D Tracking evaluate multi-object tracking (MOT) systems. combines detection accuracy and tracking performance by ussessing how well system detects, associates, and tracks multiple objects over time. AMOTA is calculated by averaging tracking accuracy range of thresholds for the Intersection over Union (IoU) matching criteria. [0,cc) Relative Translation Error (RTE) a metric commonly used in roboties, pose esti- 3D Pose Estimation
[0.1]
42 RTE
mation. and SLAM (Simultancous Localization and Mupping) tasks. It evaluates the accuracy of system's 's estimated translation (movement) compared to the ground truth, typically in scenarios where spatial accuracy is erucial.
43 CD
[0,cc) Chamfer Distance (CD) is a metric widely used in 3D geometry processing, point Point Cloud Generation
cloud generation, and shape matching tasks. measures the similarity between two sets of points (e.g., two point clouds) by quantifying the average closest-point distance between them. CD particularly useful for evaluating the alignment and fidelity of reconstructed or generated 3D shupes compared to ground truth duta.
Segmeentation&&Detcctonnnn
44 m
[0.1] mloU, Mean Intersection over Union, calculates the ratio of the intersection area Image Semantic Segmen- to the union area between the predicted and ground truth segmentation masks for a tation single class.
45 m_vloUf
[0.1] m_ vn measures the spatiotemporal overlap between predicted and ground-truth Temporal Action Detec- object regions across multiple video frames
46 Inst-mloUt
[0.1] Inst-mloU computes the average IoU score for all part instances across dataset. It 3D Part Segmentation ensures that both over-segmentation and under-segmentation errors are penalized, focusing instance- level segmentation quality.
PQt
[0.1] PQ. Panoptic Quality, is used for panoptic segmentation tasks, combining both Panoptic Segmentation segmentation quality and detection quality. is comprehensive metric that evaluates both pixel-level segmentation and object detection quality.
48 DICE
[0.1] DICE, Dice Similarity Coefficient, a metrie commonly used image segmentation Bone Fracture Detection tasks, measuring the similarity between the predicted segmentation and the ground truth segmentation.
49 S-measuret
[0.1] Structure Measure (S-measure) is metric designed to evaluute the structural similar- Video Object Detection ity between a predicted binary map (e.g.. an object mask) and ground truth binary map. It halances both region-level and boundary-leve consistency, ensuring that the evaluation captures holistic structural integrity and fine-grained details.
Audio-related
30

---

On Path to Multimodal Generalist: General-Level and General-Benel
Metric
Range
Calculation
Representative Tasks
CLAP1
[0.1]
CLAP (Contrustive Longuage-Auding Pretraining) evaluates the aligmment between generated audio and text. It is derived from contrastive learning framework where embeddings of audio and text are trained to be close in a shared latent space they semantically related.
Audio Editing
51 Style-CLAP1
[0.1] Style-CLAP calculates the CLAP cosine similarity between the generated Mel spec- Music Style Transfer trograms and the corresponding textual description of the style to evaluate style fit.
52 MCD +
(0,cc) Mel-cepstral distortion (MCD) measures the spectral distance between the mel- Speech Synthesis cepstral coefficients (MCCs) of generated speech and reference speech, providing un indication of how closely generated speech resembles the reference terms acoustic characteristics.
WER 4
[0.1] WER (Word Error Rate) measures the percentage of errors in transcribed output TTS compared to the reference transcription.
FAD J
(0.cc) Frechet audio distance (FAD) evaluates the quality and realism of generated audio, Video-to-Audio
and measures the similarity between the distribution of features obtained by VGGish in generated audio and those in set real (reference) audio samples.
55 1
[0,1] Fitch-Class Consistency (PCC) metric used in the evaluation of generated music to assess how consistent the pitch classes (e.g., notes) are across pairs of bars a piece of music. It measures the overlapping between the pitch-class histograms of different bars, ensuring that the generated music maintains harmonic coherence.
Music Generation
Human-aware Evaluation
56 UPR f
[0.1]
UPR, User Preference Rates, UPR measures the proportion of times particular Video Style Transfer system or model is preferred over alternatives in a set user evaluations, reflects the subjective preferences of users and is often derived from pairwise comparisons ranking experiments.
57 MOS f
[1,5]
Mean Opinion Score (MOS), in which human raters listen synthesized speech and Speech Generation assess its naturalness, quality, and intelligibility using a 5-point Likert scale.
58 GPT-Score t
[0.1]
GPT-Score evaluates the instruction following with GPT assistance, alter- Audio Question Answer- native to humna evaluation.
ing
Mapping Functions of Scoring Metric. Most task evaluation scores, despite utilizing different metrics, fall within 0-100% range, such as FI, Accuracy (Acc), and ROUGE-L, and follow a monotonically increasing trend. However, certain task metrics produce scores outside this range. For example, regression-relatedd metrics, as well as FID, FVD, and similar metrics, range from 0 to infinity and follow a monotonically decreasing trend. In contrast, MOS scores are represented as a discrete 5-point scale. Due to these varying score ranges across tasks, it becomes intractable to normalize them to a unified scale for level score calculations. Thus, we design the following mapping functions to standardize these metrics into 1-100% range, therehy streamlining the computation of level scoring algorithms.
Normalizing MAE:
50 - 1, where x E [0, +00), y (0, 1).
= x sigmoid
Normalizing RMS:
[50
= x sigmoid
1, where x E [0,+00)) y E (0, 1).
Normalizing MSE:
5
y 2 x sigmoid
1, where x E {0,,00),, y E (0,
Normalizing RMSE:
y = 2 x sigmoid
1, where x E +00,+00), y E (0, 1).
Normalizing absRel:
0.1
y = 2 x sigmoid
1, where x E 10, +00), y E (0,
31

---

On Path to Multimodal Generalist: General-Level and General-Bende
Normalizing EPE:
1 -, where x E [0, +00), y E (0, 1).
y 2 x sigmoid
Normalizing FID:
(25
x sigmoid
1, where E (0,+000,
y (0, 1).
Normalizing FVD:
100
y sigmoid
1, where (0,) y E (0,
Normalizing FAD:
10
= x sigmoid
- 1, where x +00), y (0,
Normalizing PSNR:
x 20
= tanh
where E (00), y 10.
Normalizing SAD:
10
2 x sigmoid
1, where (0,+) y (0,
Normalizing RTE:
(0.5
y 2 sigmoid
where æ E (00), y E (0,
Normalizing CD:
y = 2 x sigmoid
- 1, where x E {0,+ y 6 (0, 1).
Normalizing MCD:
y = 2 x sigmoid
1, where x E {00)
E (0, 1).
Normalizing WER:
y Ir
where [0, 1]. y E [0.1].
Normalizing MS-SSIM:
(x 1)
y
where x E [-1, y [0.1].
2
Normalizing MOS:
y = 4
x
where x E [1,5),
y e [0,1].
32

---

On Path to Multimodal Generalist: General-Level and General-Benu
B.5
Data Format
To provide comprehensive understanding of how utilize our benchmark data, we present examples illustrating how the data files are stored and organized. Figure 8 displays the code snippets showcasing the structures of some representative tasks. Figure 9 illustrates how organize the benchmark datasets in the file system.
"task": "accent country classification", ,
Task Name
"data_soure :atrasssuuee:: "Speech Accent Archive" " Data source for constructing the benchmark
"type": "audio comprehension" Task paradigm: comprehension or generation
'modality": {
"in": ("audio", "text"
The input and output modality
"out": "text"
skills for completing the task
"skill": "Speech Accect Understanding
"domain": ("Literature", "Culture"],
The domain covered in the dataset
: ("Reasoning Ability", "Content Recognition") for completing the task
The general capability needed
"version": "1.0" "data":
Instance 000
"id": "ac "ac_000",, "input" {
Input content
"audio file": "german30... .mp3",
"prompt": "what is the accent in the audio?"
"output":
"text": "austria"
Output content
"id": "ac_001",
"input": {
"audio file": "english201.mp""
"prompt": "what is the accent in the audio?"
ootpput: { "text": "usa"
Instance 001
Figure 8: An illustrative example of file formats.
33

---

On Path to Multimodal Generalist: General-Level and Ceneral-Bench
General -Bench Bench
Image
comprehension
Bird-Detection
annotation.jse
images
Acadian Flycatcher_@001
Bottle-Anomaly Anomaly D-Deeectin
annotation.
images
generation
Layout-tor to-Face--ae Face-Image
innotation.jsom
images
Video
comprehension
Human-Ol
-bjtet-ttee www.ennniinnniiieeee www.iion-iiid--aaiiiiiiiiiiiieei annotation.3 json
videos
generation
Scene Image- annotation.
Generation
videos
3d
comprehension
30 Furniture Classification annotation.jsom
pointclouds
generation
Text-to- Tee--to-3D-Livv 30-Living-and-Arts wwt.rrtt-oonn----dd-eeeeeeee
annotation.jsom
pointclouds
Audio
comprehension
Accent-Cl ass clseettccasiiiiiaionn fication
annotation. json audios
generation
Video-
annotation. audios
History-
Misstory-Question Queettio Question-Answerie Answering annotation. json
Abstractive Summarization
annotation.jsom
Figure 9: The organization structure of the file system.
34

---

On Path to Multimodal Generalist: General-Level and General-Bene
Stecstine dee
Dn
Sor uo Peri Loorg
speckcm Sie
aa Bostar an dapaee
bog Toe
Betavior g
Sin Binkkh2im Masuin a
Tert baaadd Souin Soundiing minniiiing Con
algee Dol
Diness Recog
Code aeer
Doc VOA
Joig Cdly
Imig Eonaace
Inpaany
Gen
Be
Di
Hallucin Sirgh Os
Emiotion De
Ohe mone
De
Img Denoxe
Edit
ling Cap
Farr D4g
Bed sdgen Weathhr
Deum Est
Dang
mig Seg
Vis-Story Gen
Generation
Img OCR
Ing Rezog
Farne
Via Rel Inf
Conuue VIA
Sion Reoebg Scaee
mg Sam Seg
Safe Det
Ripe Rlcog
Image
Aona See
Seg
Rei Reason
RGBD, racuin
Pantopii 589
Img VQA
Word MGF
Aard ponei
VOA
PCA
Object Reco
Comprehensionigg
Ind Anemaly Det
Vis Gmd.
ligh
Object Mat
Keypoint D
(eal
MM Dialng
Object Det
Object Aaaun Buod
MM Reason
Med Sag.
Deet diee HG
Can
wWW
croed Crtam
i
aa Candber
Figure 10: Taxonomy and hierarchy of data in terms of Image modality.
B.6 Data Taxonomy and Hierarchy
We visualize a comprehensive hierarchical taxonomy of our benchmark. Due to space constraints, we have separately illustrated the taxonomy for five major modalities in Figure 10, Figure 11, Figure 12, Figure 13, and Figure 14, respectively. Each visualization includes comprehension and generation paradigms, skills (meta-tasks), and specific tasks for the respective modality.
35

---

On Path to Multimodal Generalist: General-Level and Seneral-Benel
solading sou poyeoooo seor
Human-Par C-Revos
pruppment C C-RNOS
netu (ouny Mon
Sue aosaan (s0
Buurei PM
Human C-Relos
Aniimal AhimaiC-Raala C-RaiVDs nevvss
Auto C RevOs
home homuned de munny Bomaieed
ingerd meger PIA unarpuee PM sagun WAS
Susseaa und
snazonng
Auto vos
Pumas VOS
Animall VOS Sports VOS
Autu VOS VOS
Nv. Wi-Mumm Humen VOS
My. Arimal vos hy Rumiture vas
Vid Hom-- Treck Long: Vid Obj Trads Long Vid-Animod Ae--nnmall Track -ir dil--aieeeee Track Long wid Grow Track
Col Oti Match Shape Mitch Size Position Match
Marker Match
Object Marker Match
Match
Belition Match
compe Vid Gan
Sceve Img2Vi Vid
Velicle- mmg2 Vid Gen
Gan
C-Revoo
Genernl vos
Vehicle Track
Track
Archi
Food img2 Vid Gan
Pet- Imgii VmgaVid Gan
Vid Edit
Long Vid Track
NW vos
Human
Animal Track
Generall O-bjj Track
Somman-Pare Part Track
Wild Animel
Animal-nmgaaenn -mg2 Vid Gen
schitess 1mg2 Vid
TxtzVid Gan
Other -Prrt Track
Symthetic Scane Flow Estimata Complex S Extimate Panoramic Flow Estimate
Obj Match
Human Human-Img22 4mg2 Rdd Gen
Plant-Ing Wid Gen
Material- TutZvid Gen
Ing2Vid Gen
Generation
Obj Track
Cofon Txt2Vid Gen
Human RVOS
Style-Txx2Vid Gen
Olow
Animal RVOS
Human ReVOS
Weter-Sports- Sxtrrss--t22iii
Condt Gen
RVOS
Concert: Tx12Vid
Video
Animal
AnimalReVOS ReVOS ReVOS
Athletics Tx12Vid
Txt2Vid Gen
ReVOS
Human-Txx2Via Gan
SOS
Txt2Vid Gen
Temp Act Det
Auto: Street Human--Ste Strnant--Stettt VOS
VOs VOS
Interact-Vid Pedact-ViidO
Edu-Vid
Video QA
UA V Track
Bicycle- VOS
Oej-Lcc-Xd QA
Dbj-Dk-Aidd
Comprehension
UW Track
Static-Action Action Det Dynamic Action Det
Otiy Motion-Vie QA
Obij Cod- GA
Gymnastic-V52 Vid QA
Bal-Vid Vid
Pet-Vid QA
vid Recog
Vid-Event Recog
ni Understand
Vid Act Racog
UAV UAV-Build Build
UAY MAV-UAVToR Track
UAV Human Track
WAV-Vehicle Track
Science- Mid OA
Spon Vid
vid Go-Gro
UAV-Obi Track
Move- GA
Blue Water Track
Yallow Wate Track
Comedy Vid
Miltery- sid Q.
Green Water Track
Survive--j Vid OA
Crowed Track
White White-Watee Water Track
Geo Vid
Agei GA
N-O C-DD--nttract--- Vid Cap likk-itterttt-- vid
Teressiial ii Recod
Aquati COD Recod
Body Motttn Cap
Hum Salent Obi Dat cog
fnstrum Cap
Salient- Det Recong
Food Cap Becoog
Spronts Cup
Vehicies Cap Becong
Becodd
Per Care Cap
ArteeCrafts Cap
Faca. Operatin Cap
Action
Food! Racon
feccoll
Entertain Cap
Animall
Vid-06 aegaette serrec foor
Contine coreee TV- Saoonn corr
Action Orderigg
Action Countine
Artion Pred
Acdion Sciin--aas Joaly
Langaangg unguuge Recog
Cap
Auto MG
Mouse
Gane OA
Musc OA
Salfeed cap
Sport Cap
pues comman
Sone sel per Prr
d Cop
301 Serie
Siaee
Syn
Humor
Business
Francc
Vidco
VoE
Cap
Figure 11: Taxonomy and hierarchy of data in terms of Video modality.
36

---

On Path to Multimodal Generalist: General-Level and General- Benchh
RGBD2Mesh Recon
Txt2Mation Gen
3D Ap-Appiincce Cls
3D To-Talleware Cls
3D-Accest Cls
RGBD2PC Recon
Culture Img2M Gen
3D-Person Cls
Nature-Inee Neture-Img2M Gen
3D-MI Cls
30 Fo-Fuuiittre as
30 30-Struct Cls
Sci-Img2M Gen
Arts Arts-Img2M Img2M Gen
3D Electranic cis
3D-Vehicle Cls
Culture -mgg2CggGGGa Gen
Nature-Img2PC Gen
3D 3D-Appliance Seg
RGBD 2PC Recon
RGBD 2Mesh Recon
Txt2Motion Gen
30 Human Cls
Sci-Img2PC Gen
3D Furniuure Seg
30 Struct Cls
ImgZM Gen
2Mesh
Arts-Img2PC Gen
3D Structure Seg
Tech Cls
Img2PC Gen Generation
Culture-Ti Culture-Txt2M Gen
Indoor Scene- Seg
3D-Outdr Seg
Nature-Txt2M Gen
Outdear Scene Seg
Txt2M Gen
3D
Indoor-Inst Seg
Sci-Txt2M
I--I--nnstance Seg
Pose Est
Arts-Txt2M Gen
TRt2PC Gen
Comprehension
Part Seg
3D Od-metry
Culture- Mituree-Tx2PC Txt2PC Gen
PC2M Recon
3D Ap-Motion Analy
BD Geo Analy
PC Complt
3D Track
Nature- Txt2PC Nature-Txt2PCC Gen
3D 3D-Aircraft Seg
Sci- CG-TTxtPPC Gen
3D Det
3D-Person Person Seg
Art- RR-TTx2PCCCC Gen
3D 3D-Ve Vehicle Seg
3D Deapee Seg
Scene-PC2M Recon
Object-PCZM Recon
3D -r-Track
3D-PC Completion
3D 3D-Normal Est
3D ID-Detection
3D Mation Cap
3D- O0thrr QA
3D- Which' QA
3D-'Can' QA
aD-% Ip-i QA
3D l--Whht' QA
3D-Space QA
Sottto QA
Figure 12: Taxonomy and hierarchy of data in terms of 3D modality.
37

---

On Path to Multimodal Generalist: General-Level and General-Bend Bennch
Accent Accent-Sex Sex Recog
Vocal Vocal-Sound Sound Recog
Accent cog
2Caps TaAdio
VideoZAudio
1CapToAudio
Spe Speaker
Bird Sound Detect
AnimalSoundDetted
MTTS
VoiceConversion
TTS
Long AudioCaps apps
Wild AudioCaps
SpeechTrans (En-Mo)
SpeechTrans (En-Zh)
Txt2Aud Gen
V2A
Acnt Analy
TTS
Animal- Sound Det
OpenAQA
SpeechTrans (Zh-En)
Style Trans
Aud Analy
AudioQA
Spe Speech ech Trans
Aud QA
Chord Music Style Trans
Note-Quality Anal
Generation
Audio
Music Style Transfer
Music Trans Trans
Aud- Analy
Singer
Vocal-Tech Detect
Music Gen
Comprehension
Cint Analy
Song
Img2Aud Gen
EmoSpeech Gen
Envir-Sound Sound Det
Singing Voice Syn
keuy SpeechEme
Intent R Recog
PopMusic Gen
Dialog Gen
Music Analy
Speech Cmd
AudioEdit
Piano2Orchestra Gen
Spe SpeechEE
Sopeech
Acoustic Scene Recog
Env-Sound Recog
EmoStyle SmoStyleTransfer Transfer
Sound Sound-Eveny -Event Recog
EmoSpeech Syn
Music Genre Racog
DailyTalk Talk Gen
Instrument Recog
AudioE
Emation Recog
Pitch Anal
Instrument-Se Instumment-Suure Source Anal
Figure 13: Taxonomy and hierarchy of data in terms of Audio modality.
38

---

On Path to Multimodal Generalist: General-Level and General- BBrch
Open Domain QA
Conversation QA
Extract Summ
Abstract Sum
MultiD Doc Sum
Multi- lang Trans
Low-Ree
En-Zh Trans
En Cn-F raie mas
Table QA
Multi Lang QA
Code Code-Switee Swit-SSichh Q4
Linguistics QA
Humon Detect
MenaeL Mena@L-Tosin Tosic Det
Sarcasm Det
Trans
Grammar Correct
Question Cls Tapic Cls
Sentiennt
Finance cls
Meaphon Det
NU
Aspect Det
Aspect Senti
Aspect Term
Table-to to-1 Text
Style Transfer
Story Gen
Paraphrase
Opinion Ext
AO-Pair Ext
Senti RentiTrripeen TripTripett Ext
Time Series Pred
ABSA
Senti Quad Pred
Det
Dialog Sent Quad
History QA
Business QA
Soccer SA Opinion- Quinnnnniii Ext
Philosophy QA
Art QA
Humanities QA
Socia#L-So QA
Txt Cls
Txt Ext Entall
Txt Sum
X-L X-LanNNM M
Advance QA
Code Explain
Code-Defect Deffcct Detect
TxT Gen
Affect Computing
Code Gen
Code Repair
AMR
Time Series
Txt25QL Gen
Event Event-Coref Coref Res
Social QA
Code Ability
Common Reason
Person Analy
Sem Par
Causal Reason
Stance Detect
NLP
Doc Poc-Causal Reason
Intent Det
Sem Analy
Cog QA
Counterfact Reason
Sent Similar
Dialog Gen
Analog Reason
Rei Ext
Non- Trad QA
Domain QA
Multi-Hop QA
ADR Det
Temporal Reason
Chem- Chem-Protein RE
Genetic Assoc
Math Ability
Ethics NLP
Numeria Reason Reason
Spatial Reason
DDI Ext
Event Ext
PPI Ext
NER
Ling Par
Daily Dialogue Gen
Dialog RE
Doc RE
Biomedical QA
Joint- Joint-NER-Re NER-RE
Medical QA
News RE
Tech QA
Sci RE
Eng QA
Science QA
Trivial QA
Earth QA
Tweet QA
Nature QA
Politics QA
Fairytale
Disease NER
Ethical Reason
Chem NER
Truthful QA
Gene&Protein NER
Cámatte NER
Blas Detect
Legal
Location NER
Person NER
Hate Hate-Speer Speech Detect
Offensive Classity
Organ NER
Legal NER
Goo NER
Cyber INER
Path NER
Temporal INER
Scl NER
Math Proof Gen
Math Word Prob
Math QA
POS
Const Parse
Temp- Say--Event Event Reason
Event: Tnt-Trrigunn Trigger Det Event: TrisgerBAArg Extract
Fake- News Detect Fact Verify
Spam Detect
Parse
Figure 14: Taxonomy and hierarchy of data in terms of Language modality.
39

---

On Path to Multimodal Generalist: General-Level and General-Benu
B.7 Data Distributions
Capability. In Figure 15, we present the distribution of capability evaluations across all tasks in General -Brrch These capabilitiie include: Content Recognition, Commonsense Understanding, Reasoning Ability, Causality Discrimination, Affective Analysis, Problem Solving, Creativity and Innovation, Interactive Capability, and others. As observed, the majority of tasks focus on Content Recognition or perception-related abilities. This emphasis aligns with the current stage of MILM development, where models are not yet equipped with highly advanced cognitive capabilities. We plan to continuously update the benchmark in the future to accommodate the evolving strengths of more powerful models.
Creativity and Innovation: 125 (13.48%)
A ffective A 34 (3.67%)
Commonsense K Knowledge: 129 (13.92%)
Interactive Capability: 22 (2.37%)
Causality Discrimination: 15 (1.62%)
Ethical wareness: 14 (1.51%)
Planning Ability: 18 (1.94%)
Cognition Understanding: 25 (2.7%)
Temporal D Determination: 10 (1.08%)
Reasoning Abiliy: 124 (13.38%)
Content Recognition: 399 (43.04%)
Figure 15: Distribution of various capabilities evaluated in General -Bnch.
Daily: (3.88%)
Physios: (1.9684)
Sports: 18 (4.41%)
Mallu 10 (2.65%)
Philosophy: (1.22%)
Geometry: 8 (.966)
Lawe: (2.45%)
Engineeting: 15.38%)
Finznoe: 12 (2.90%)
Googriphy: 611,476)
Nature: (1.72%)
Physics
Math
Climate: 7(1.72%)
Geometty
Engineering
Art: 43 (10.54%)
Biology
Medicine
Biology
Chemistry
Astronomy
Social 248 (31.09%)
Geography
Nature
General: (69.06%)
Animal
Code
Climate
23 (5.64%)
Natural Sciezce: 159 85%)
Social
Féiles (2.49%)
Chemistry: (1.72%
Humanities
Linguistios
Social: 10 (2.55%)
Astruzomy: 4 (0.98%)
History
Politics
Eurth (0.73%)
Animal: 12 (2.94%)
Business
Social
Besiness: 15 (3.68%)
Code: (1.72%)
Phics
Culture
(1.718))
Art
Pinance
Histor:: (3.19%)
Law
Philosophy
Linguistio: (8(4441))
EamoriLie: (13.239))
Sports
Daily
Figure 16: Distribution of various domains and disciplines covered by General-Bence
Domains and Discipline. Figure 16 illustrates the domains and disciplines covered by our benchmark. While the majority of tasks belong to the general domain, the benchmark also encompasses significant fields from both Physical Sciences and Social Sciences. For Physical Sciences, the benchmark includes disciplines such Physics, Geometry, Biology, Medicine,
40

---

On Path to Multimodal Generalist: General-Level and General-Bench
Chemistry, Astronomy, and Geography. For Social Sciences, it spans including Humanities, Linguisties, History, Politics, Culture, Art, and Economics. In other words, our benchmark is designed to evaluate the capabilities of MILMs across wide range of scientific fields and domains. This ensures the broad evaluative advantage of our benchmark, enabling comprehensive assessment of multimodal generalist models.
Comprehension ys. Generation. We illustrate the task distribution across the two critical paradigms, Comprehension and Generation, in Figure 17. Currently, the majority of tasks are centered on comprehension, which aligns with the present capabilities of most MLLMs.
Comprehension
Generation
3D
udio
24
20
Video
128
46
Image
50
100
150
200
250
300
Figure 17: Distribution of various domains and discipline covered by General -ench.
Modality. Finally, we present the distribution of tasks across different modalities in Figure 18. Overall, the image modality constitutes the largest proportion of tasks. We note that beyond the five major modalities Image, Video, 3D (3D-RGB and Point-Cloud), Audio, and Language- benchmark also includes tasks other modalities such as Time Series, Depth, Infrared, Spectrogram, Radar, Code, Document, and Graph. These additional modalities play important roles in specific domains. However, due to the limited number of tasks these modalities, we have merged and classified their data under broader categories like Image and Language for of management.
3D: 53 (7.47%)
Audio: (6.21%)
Language: 121 (17.07%)
Video: 174 (24.54%)
Image: 317 (44.71%)
Figure 18: Distribution of different modalities covered in General -
B.8 Extended Data Insights
General-Bench particularly places a strong emphasis on the diversity of its evaluation data, covering a wide range of fields and scenarios to assess different aspects of model capabilities, as depicted in Figure 19. First, Table 6 summarizes the statistics of task and skill numbers in Genera -BBech..

---

On Path to Multimodal Generalist: General-Level and General-Bec -Brnch
Domain & Discipline
Natural Sciences
Physics
Math
G Geometry
B Biology
Engineering
Chemistry
Geogmahhy
General
fuht
Earth
Medicine
Nature
nimal
Climake
Code
Astronomy
ALL
Social Sciences
Humanities
Linguistics
History
Politics
Culture
Economics
Philosophy
Sports
Business
Social
Finance
Daily
Art
Capability
Content Recogmition
Cognition Understanding
Reasoning A bility
Causality Discrimination
Identifying objects, entities, events within the given multimo-dal data precisely
Interpreting intents, subtext, and metaphors contextial and nuanced manner
Solving complex problems or questions (e.g. logical, math th ematical) using reasoning
D Detecting causal relati relationships and distinguishing betw between een cause and e effect
Commonsense K wnowledge
Spatial Perception on
Creativity and Innovation
Temporal D Determination
Understanding everyday soen- arios and basic facts across diverse domains
Understanding and reasoning bout spatial relaionnsiiii atioonshps in various context
G Generating creative ideas or content and synthesizing cross-domain information
Underttaddigg and reasoning temporal sequences and rela- tionshiss data
Affective A nalysis
Planning A bility
Ethical A wareness
Interactive Capability
Understanding buman emo- bons, sentiments, and empathy in various modalities
Formulating plans and strate- gies achieve defined goals.
Evaluating ethical considera- ensuring responsible Docision-makinng
Engaging in mul ti- turn intera- ctions and managing context effectively
Modality-specific Skill
Image
Video
Audio
3D
Language
Comprehension on
Comprehension
Comprehension
Comprehension
Linguaitiic Parsing
Semantic Pansing
Image Captioning
Adion Prediction
Detection
Leetive Compuling
Depth Estimation
Video
Azimal Sound Analysis
3D QA
Opinion Mining
Im OCR
Object Matching
Music Understanding
30 Motion A nakysts
Relation Extmactin
Image Recogrition
Objest Tmakkng
adioo nilysis
Estimation
Event Extraction
Semantic Segmentation
Video G Gevunding
Eaviron So und A palysis
3D Tracking
Aalysis
Imt age Visual Grounding
Long Video meking
Speech Accent mayy
30 Humag- Olject Clasilication
Named Entiy
Image Visual
Video Deyth Fstimation
Speach
nalyas
Recommtton
Soane Recogmition
Video Attio Racog
Speech Emotion
3D Indbor Scame Seman- tc Segmentation
Cognitive QA
Multimodal Ressoning
Viden Event Recog
A nalysss
3D Outioor Scone Sem- antic Segm entaaiiii
Code Problem Solving
Mulli- image isual
Video Object Recuy
lons--ingual NLPP
Object etection
Optical Flow
Traralation
Dialogue eneration
dranced QA
Generation
Generation
Generation
Generation
Ethical NLP
Math Problem Solving
Text-based Img Editing
Conditional Video Gen
TTS
Image to Mesh Gen I'm age to Point Cloud Geteration
Numerical Prediction
Text to Img Gonemtion
Image2V koea Gmmention
cxfin Rdit
Social QA
Image Inpaitling
Tex Generaion
Slyle Transter
Summanization
Image Ethancement
Video ction Generation
Music Syuthesis
RGB-D Mash Recon
Text Entailment
Image S Transer
Viden Editing
Speech Style Transfor
Puint Cloud to Mesh Recon
Text Generation
L2yau Genention
Skotch21mg Generation
Viden Entencement
F'm ontonnall Speech Gan
Image2A udio Syathesi
Text Motion Generation
Semanbl Simulantt
July sis
Figure 19: General-Bench covers over 29 domains, evaluating more than 12 ceddality-pprsistent capabilities of generalists, as well as 145 modality-specift skills. In Appendix $B.7 we showcase all tasks and data specification detail.
B.9
Comparisons with Existing Benchmarks
Table 7 provides a comprehensive comparison of General -Bech with existing MLLM benchmarks. Compared to these benchmarks, General -encch demonstrates absolute superiority across all comparison aspects. For instance, is the first MILMM benchmark to cover nearly all commonly used modalities while emphasizing both Comprehension and Generation
42

---

On Path to Multimodal Generalist: General-Leve and General-Bene
Table 6: Summary of numbers of skills, tasks and data instances across modalities.
Image
Video
Audio
3D
Language
TOTAL
Comp
Gen
Comp
Gen
Comp
Gen
Comp
Gen
Single
11
#Skill
Sum
35
26
20
22
Single
271
45
126
46
24
20
30
22
#Task
118
702
Sum
44
Single
124,880
26,610
44,442
16,430
11,247
9,516
23,705
10,614
#Instance
58,432
325,876
Sum
151,490
60,872
20,763
34,319
as critical evaluation paradigms. Most notably, General -Bench boasts the largest dataset, encompassing 550 tasks with over 275K instances. Also, it evaluates the largest number of MILMs tested date, setting new standard for benchmark comprehensivennss and scale.
Table 7: A full comparison of General-Bench with other popular MLLM benchmarks.
Aspect
Answer
#eested Models
Modality
#Domain #Skill #Task #Sample
#Metric
Annotation
Rench.
Scheme
Form
SifeeceeDDscipllle
ScienceQA
Txt.in
Comp.
12
21K
MC-QA
Acc.
Repurposed
10
(Lu et al., 2022)
VisAidMath (Ma el 2024)
Txt.Ing
Comp.
1.2K
MC-QA
Repurposed
10
MMMU
Txt, Img
Comp.
30
11.5K
MC-QA
Manual
24
(Yue ct al., 2024)
NaturalBench (Li ct al. 2024c)
Txt.Ing
Comp.
27
7.6K
MC-QA
Aoc.
Repurposed
32
Audio
AudioBench (Wang et al., 2024h)
WER,METEOR
Txt, Aud
Comp.
3
100K
Free-Form
Repurposed
Llama-score
MARBLE (Yuan et al., 2023b)
Txt, Aud
Comp.
12
18
Free-Form
Origin(19)
Repurposed
MMAU (Sakshi al., 2024)
Repurposed Manual
Txt, Aud
Comp.
12
27
10K
MC-QA
Aec.
+3D
Bench (He et al., 2023)
Txt, 3D
Gen.
3
3
300
Free-From
Origin(2)
Repurposed
3DBench
Txt, 3D
Comp.
12
12
8K
Freo-From
Origin(6)
Repurposed
3
(Zhang et al., 2024b)
Video
Video-Bench (Ning 2023)
Txt, Vid
Cump.
/
3
7
17K
MC-QA
Acc.
Repurposed
8
VideoMME ct 2024b)
TAL.Vid
Comp.
6
12
12
900
MC-QA
Acc.
Manual
13
MC-QA
Acc.
(Zhou al., 2024a)
Txt. Vid
Comp.
7
9
9
2593
Open
GPT-Ranking
Manual
20
MVBench (Li et al., 2024f)
Txt.Vid Vid
Comp.
20
20
4k
MC-QA
Acc.
Repurposed
14
AutoEval-Videe (Chen et al., 2024d)
Txt.Vd
Comp.
12
9
9
327
Open
GPT-scure
Manual
11
VBench
(Ilungg el al.,, 2024)
Txt.Vid
Gen.
8
16
16
100
Open
Manual
Image
MMEE
Txt.Ing
Comp.
14
14
2.2K
MC-QA
Acc.
Repurposed
30
(Fu el al., 2023)
43

---

On Path to Multimodal Generalist: General-Level and General-Bench
Aspect
Task Scheme
Answer Form
#Tested Model
Modality
#Domain #Skill #Task #Sample
#Metric
Annotation
Bench.
Lub
MC-QA Open
Acc. CIDEr
Txt, Img
Comp.
6
47
2.1K
Repurposed
(Xu et al., 2023b)
top-1 Acc.
MM-Vet (Yu et al., 2024)
Txl.mg
Comp.
/
6
1
205
MC-QA
GPT-score
Repurposed
16
V*Bench (Wu and Xie, 2024)
Txt.Ing
Comp.
1
2
2
191
MC-QA
Acc.
Manual
13
MMIE (Xia 2024b)
Comp. Gen.
MC-QA Open
Txt, Img
10
4
4
20K
Manual
8
Mia-bench (Qian al., 2024)
Txt. Img
Comp.
15
8
8
400
Open
GPT-score
Manual
29
MME-RealWorld (Zhang et al., 2024c)
Txt, Img
Comp.
5
43
43
29.4K
MC-QA
Acc.
Manual
29
MLLM-Bennch et al., 2024)
MC-QA Open
Txt. Img
Comp.
6
420
GPT-score
Manual
21
Q-Bench (Wu et al., 2024b)
MC-QA Open
Manual+
Txt, Img
Comp.
3
3
84.7K
GPT-score
Repurposed
15
MUIRBench (Wang al., 2024c)
Manual+
Txt.ing
Comp.
12
12
12
2.6K
MC-QA
Acc.
Repurposed
20
MileBench (Song et al., 2024)
MC-QA
Ace.
Manual+
Txt,Ing
Comp.
2
2
6.4K
22
Open
ROUGE-L
Repurposed
MMBench (Liu ct al., 2024a)
Txt.Ing
Comp.
2
20
MC-QA
Repurposed
21
Omini
SE..-BBench (Li ct al., 2023b)
Txi,Img
Comp.
12
19K
MC-QA
Manual
SEED-Bench-2
Comp. Gen.
Manual+ Repurposed
(1.i et al., 2023c)
Txt.img Vid
3
22
22
24K
MC-QA
Acc.
23
CV-Bench (Tong et al., 2024a)
Txt, Img. 3D
Comp.
2
4
2.6K
MC-QA
Acc.
Repurposed
MMIU (Meng al., 2024b)
Txi.in Vid,
Point-Cloud, Dloth
Comp.
7
52
11.7K
MC-QA
Acc.
Repurposed
22
MMTTBench (Ying al., 2024b)
Txt.Ing. Vid.
Point-Cloud
Comp.
32
162
31K
MC-QA
Acc.
Repurposed
30
MEGA-Bench (Chen 2024c)
Txt,Img, Vid
Comp.
10
505
8K
Free-Form
Origin (45)
Manual
22
Txi.mng Vid,Aud,
172 x
Demh.Depth.3D-RR DR-GB,
Specialists
Gemeral-Bence (Ours)
Reannotated +Manual
Point-Cloud, Mooud,nffrared, Comp.+Gen.
29
145
702
325.8K
Free-Form
Origin (58)
&
Spectrogram, Radar,
Code. Sade_Doc,Grut Graph...
Generalists
44

---

On Path to Multimodal Generalist: General-Level and General-Benu
C Extension on Experimental Results
In the main text, we presented a subset the evaluation performance of MLLMs at the skill level due to space constraints. Here, we provide the complete results on all specific tasks. The results are organized based on modality and task paradigm distinctions.
It is important to note that we conducted inference using different MLLMs' open-source codebases or APIs. However, the choice of prompts for different tasks can significantly impact the model's performance on a given task. For instance, some models require highly detailed and specific in-context instructions in the input prompt to achieve their best performance. To ensure faimess, our team applied a uniform prompt across all MLLMs for given task, without any model-specific prompt tuning. the model developers are unsatisfied with the presented results, we welcome them to adjust the prompts to obtain more representative and improved scores.
C.1 Multimodal Specialist and Generalist Systems
SoTA Specialist. For each specific task under a specific modality, we select a SoTA specialist to generate benchmark results. The selection of specialists is determined based on two criteria: 1) their performance on each task using public benchmarks and leaderboards, i.e., they must demonstrate top performance; and whether they are widely recognized and utilized by the community. Meanwhile, we exclude models that lack reliable open-source code or parameters (as unable to run our own data through them), even if such models claim to be SoTA in their own papers. It is important to note that the specialists we use must have undergone large-scale supervised pretraining on the corresponding tasks, enabling them to achieve SoTA performances. In our implementation, we directly load their released parameters and perform inference on the General- Benrraa-Beence Bencch test sets. In total, we have 172 specialists.
Multimodal Generalists. We consider a diverse set of existing popular MLLMs that are capable of handling specific or various modalities and tasks. This includes both open-source systems and closed-source ones (such as the OpenAl GPT series). For open-source models, implement them by loading their released parameters and directly performing inference on the General Seeral-Beende Bench test sets. For closed-source models, we utilize their APIs to access the services. We note that, despite the release of a vast number of MLLMs in the community, due to resource constraints, we only consider subset of MIL.Ms that demonstrate strong and stable capabilities and are widely recognized and utilized. However, our evaluation system remains open, and we encourage more MILMs interested in our benchmarking system to participate by running their own evaluations and submitting their scores. Table 9 summarizes the multimodal generalists employed, including their corresponding modality support, characterized skills, parameter sizes, and backbone LLM architectures.
Table 9: A complete list of (multimodal) generalists evaluated on General-Bench.
Model
Backbone
Size
Modality Support
Paradigm
Lorguege-oriented (Close//Open--oureee) Models
Meta-L-ama-3311-33
Ilama
8B
Language
Instruct (Touvron et al., 2023)
2 Gemma-2-9b-it (Team Gemma et ul., 2024b)
9B
Language
3 GPT-J (Wang and Komat- GPT-J suzaki, 2021)
6B
Language
ChaiGLM-6B (GLM al. ChalGLM 2024)
Language
5
Owen2.5-7B-IInstruct (Yang al., 2024)
Qwen2.5
7B
Language
6
I2 (Cai InternLM2 et al., 2024)
7B
Language
7 Baichuan2-7B-Chhe (Yang Baichuan2 et al., 2023)
7B
Language
Sicuna-bb-IL. (Chiang Vicuna
7B
Language
et al., 2023)
Ci-oon3-7B-Instruue (Almazrouei et al., 2023)
Falcon3
7B
Language
45

---

On Path to Multimodal Generalist: General-Level and General-Bence
# Model
Backbone
Size
Modality Support
Paradigm
10
Ministral-8B-Instruue
Ministral
8B
Language
2410 (Jiagg al., 2024a) Yi-lightning (Young et al., Llama 2024)
11
6B
Language
12 GPT-..5-turbo (OpenAl, GPT3.5
Language
2022a)
Multimodal Close-sourced Models
GPT4-V (OpenAI, 2022b)
GPT4
Language, Image
Comprehension
GPT4-o-mini (OpenAI. 2022b)
GPT4
Language, Image
Comprehension
3
GPT4-o (OpenAI, 2022b)
GPT4
Language, Image
Comprehension
4
GPT4-o-409
(OpenAI,
GPT4
Language, Image
Comprehension
2022b)
5
ChatGPT-o-latest (OpenAl, GPT4
Language, Image
Comprehension
2022b)
Claude-3.5-Sonnet (Team, Claude-3.5-Sonec
Language, Image
Comprehension
2024)
7
Claude-3.5-Opus (Team, Claude-3.5-Opus 2024)
Language, Image
Comprehension
8
Gemini-1.5-Pro al., 2024a)
(Team Gemini
Language, Image
Comprehension
9
Gemini- 1,5--i--,,Faah (Team Gemini
Language, Image
Comprehension
et al., 2024a)
Multimodal Open-sourced Models
Yi-vision-v2 (Young et al., I.LaVa 2024)
6B
Language, Image
Comprehension
Fmu2-378 (Sun al., LLaMA-33B 2024)
37B
Language, Image
Compeheesion--Ceenrattoo
3
IntemVV.2.5-2B et ul., 2024c)
(Chen Internlm2_5-1_8hhhhh
2B
Language, Image
Comprehension
4
V1.e5mV1.2.-4BB
(Chen Qwen2.5-3B-Instrucc
4B
Language, Image
Comprehension
et al., 2024c)
5
VL2.58B et al., 2024c)
(Chen internlm_5577b-00
8B
Language, Image
Comprehension
VL-C-ht 2B-V (Gao al., 2024)
IntemLM2-Chaa L.BB
2B
Language, Image
Comprehension
7
Mini-IntemVL-Chat 4B-V1-5 (Gao et al., 2024)
Whi-3-mini- 1ini--12k--nntruue
4B
Language, Image
Comprehension
8
MiemLM-XComposerr
IntemLM2-Chn Chat-118B
1.8B
Language, Image
Comprehension
VI-1.8B (Dong et al.,
2024)
9
Phi2
2.7B
Language, Image
Comprehension
4e-384 (Lin et al., 2024)
10
Monkcy-1OB-chat (Lictal,, Qwev-7B 2024g)
10B
Language, Image
Comprehension
11
JPPLUGG-OWI/-LLLMMA3 7b (Ye et ul., 2024)
ILaMA2-7b
7B
Language, Image
Comprehension
12
Phi-3.5-Vision-Instmaa
Phi-3 Mini
4.2B
Language, Image
Comprehension
(Abdin et al., 2024)
13
Cambrian-1 al., 2024b)
(Tong LLaMA3-8B-Instmee
8B
Language, Image
Comprehension
14
DetGPT (Pi al., 2023)
Vicuna-7B
7B
Language, Image
Comprehension
15
Otter (Li ct al., 2023d)
LLaMA-7B
7B
Language, image
Comprehension
16
NExT-Chat (Zhang al., L.LVVA
7B
Language, Image
Comprehension
2023c)
17
GPT4RoI-7B (Zhang al., LLAMA-7B 2023d)
7B
Language, Image
Comprehension
18
GLaMM (Rasheed et al., Vicuna-7B 2024)
7B
Language, Image
Comprehension
46

---

On Path to Multimodal Generalist: General-Level and General-Bench
# Model
Backbone
Size
Modality Support
Paradigm
19
Pixtral-12B (Agrawal al.,
Mistral-Nemo-1Z2
12B
Language, Image
Comprehension
2024)
20
BLIP-2 (Li et al., 2023a) Flan T5-xl
3B
Language, Image
Comprehension
21
BLIP-3 (XGen-MM) (Xue Phi3-mini
4B
Language, Image
Comprehension
et al., 2024)
22
miniMonkey (Li et al., Qwev-7B 2024g)
7B
Language, Image
Comprehension
MiniGPT4LLLMA2-775 LLaMA2-7B-instoe
7B
Language, Image
Comprehension
(Zhu 2023a)
24
Show-o (Xie et al., 2024)
Show-o
1.3B
Language, Image
ComppehensionnnCCenrrttio
25
DeepSeek-V --.ec--V..-BBBBae-
7B
Language, Image
Comprehension
(Lu et al., 2024b)
26
DeepSeck-V1--7-CCC
DeepSeek
7B
Language, Image
Comprehension
(Lu et ul., 2024b)
27
LISA (l.ai al., 2024)
L.aMA-7B
7B
Language, Image
Comprehension
CogVLM-Chat
(Wang
Vicuna-VV -..n7-v1..--BB
Language, Image
Comprehension
al., 2023h)
29
ShareGPT4V-7B al., 2025)
(Chen Vicuna-v -
7B
Language, Image
Comprehension
30
ShareGPT4V-13B (Chen Vicuna-v1.5-135 et al., 2025)
13B
Language, Image
Comprehension
31
GLM-VL-Chat (Du et al., GLM-4V 2021)
9B
Language, Image
Comprehension
32
OMG-LLaVA-
intemlm2-7b
7B
Language, Image
Comprehension
IntemLM20B
(Zhang
al., 2024a)
33
Idefics3-8B- Deeies3-8B-LLamm3 3B-Lmmma3
Llama-3.1
8B
Language, Image
Comprehension
(l.aurengon al., 2024)
34
MiniCPM3-4B (Hu et al., MiniCPM3-4B
4B
Language, Image
Comprehension
2024a)
35
SEED-LLaMA-IE (Ge Llama2-chat-133
14B
Language, Image
Comppehinnsoon-GGenerttiio
et al., 2023)
36
LaVIT-V2 (7B) (Jin et al.) LLaMA-7B
7B
Language, Image
Comprehennioon-CCeerattiii
37
LM4LV (Zheng al., LLaMA2-7B instruct 2024)
7B
Language, Vidco
Generation
38
CoLVA-2B (Zhou et ul.. Qwen2-2B 2025)
2B
Language, Image, Video
Comprehension
39
CoLVA-4B (Zhou et al., Phi3-3.8B 2025)
4.1B
Language, Image, Video
Comprehension
40
Long-LLaVA-99 (Wang Jamba In-Itmcctt ul., 2024d)
9B
Language, Video
Comprehension
VeepSeek-VL-22smme (Iu 160 al., 2024b)
2.8B
Language, Image
Comprehension
42
DeepSeck-V1-22 (I.u al., DeepScekMoF-278 2024b)
4.5B
Language, Image
Comprehension
43
Qwen-VLL-Chat Vh.-Chat (Bai et al., Qwen-7B 2023)
7B
Language, Image, Video
Comprehension
Qwen- Audin-Audi- (Chu al., 2023)
Qwen-7B
7B
Language, Audio
Comprehension
45
Qwen2-V1.-7B al., 2024a)
(Wang Qwen2-7B
7B
Language, Image, Video
Comprehension
46
Dwven-AAudioIIsstmect
Qwen-7B
7B
Language, Audio
Comprehension
(Chu al., 2024)
47
Qwen2-V1L-72B
(Wang Qwen2-72B
72B
Language, Image, Video
Comprehension
et al., 2024a)
48
LLaVA-NeXT-13B al., 2024b)
(Liu Vicuna-130
13B
Language, Image
Comprehension
LLaVA-NeXT:34B al., 2024h)
(Liu Nous-Hermes-22 2-Yi-34B
34B
Language, Image
Comprehension
47

---

On Path to Multimodal Generalist: General-Level and General-Bench
#
Model
Backbone
Size
Modality Support
Paradigm
50
MaAVA-OneVViiion-- (l.i Qwen2-7B
7B
Language, Image, Video
Comprehension
al., 2024d)
51
LLaVA-One-VViion- Vision-72B Qwen2-72B
72B
Language, Image, Video
Comprehension
(Li et al., 2024d)
52
Sa2VA-8B (Yuan et al., Interml.M2-7B 2025)
8B
Language, Image, Video
Comprehension
53
Sa2VA-26B (Yuan et al., MM-22 2025)
26B
Language, Image, Video
Comprehension
54
IntemVLL-2-8B (Chenctal, Intern. 2024c)
8B
Language, Image, Video
Comprehension
55
Intemn VL-2.5-8B
(Chen internlm2.5-7b-cle
8B
Language, Image, Video
Comprehension
et al., 2024c)
56
IntemVL-2226B
(Chen IntemLMM-200
26B
Language, Image, Video
Comprehension
et al., 2024c)
57
InterVL. VL-2.-2-25 (Chen internlm2.5-206 5-bbchhhat
26B
Language, Image, Video
Comprehension
al., 2024c)
Vitron-V VI (Fei al., vicuna-7b-v0 2024a)
7B
Language, Image, Video
Comprehensioon-CCeerattioo
59
Mini-Gemini (Li al., 2 2024c)
34B
Language, Image
Comppehensionn-CCenratiii
60
3D-LLM-2.IB (Hong al., BLIP2 2023)
2.
Language, 3D
Comprehension
Poin (Xu et al., LLaMA 2025)
7B
Language, 3D
Comrehension
62
PointLLM-13B (Xu al., LLaMA 2025)
Language, 3D
Comprehension
63
VisTA (Zhu al., BERT 2023b)
1.3B
Language, 3D
Comprehension
64
Ava (Zhou et ul., T5-large 2024b)
770M
Language, 3D
Comprehension
65
MotionGPT-TS
(Jiang T5
220M
Language, 3D
Generation
et al., 2024b)
66
MotionGPT-LLaMA
LLaMA
13B
Language, 3D
Generation
(Zhang al., 2023e)
67
LLaMA-mesh
(Zhang LLaMA
7B
Language, 3D
Generation
al., 2023e)
68
GAMA (Ghosh et al.. Llama-2-7b-chal 2024)
7B
Language, Audio
Comprehension
69
Pengi (Deshmukh et al., GPT2-base 2023)
124M
Language, Audio
Comprehension
70
WavLLM (Hu al., LLaMA-2-7B-chat 2024b)
Language, Audio
Comprehension
71
SALMONN-7B
(Tang Vicuna-7B
7B
Language, Audio (Speech)
Comprehension
al, 2023)
72
SALMONN-13B
(Tang Vicuna-13B
13B
Language, Audio (Speech)
Comprehension
et al., 2023)
73
SpeechGPT-7B-com
LLaMA-2
7B
Language, Audio (Speech)
Generation
(Zhang et al., 2023a)
74
AudioGPT-GPT4 (Huang GPT-4
Language, Audio (Speech, Sound)
Generation
al., 2023)
75
AnyGPT 2024)
(Zhan et al., LLaMA-2-7B
8B
Language, Image, Audio (Speech, Music)
Comprchension-GGeerattion
76
PandaGPT-13B (Su al., Vicuna-13B.o. 2023)
13B
Language, Image, Video, Audio
Comprehension
77
ImageBind-LLM et al., 2023)
(Han LLama-1-7B
7B
Language, Image, Vidco, Audio
Comprehension
Moda Verse-7b-o (Wang Vicuna-7b-VV
7B
Language, Image, Vidco, Audio
Comppehensionn-Geenrattio
al., 2024e)
79
Unified-io-2-XXI. al., 2024a)
(Lu UIO-2-XXL
6.8B
Language, Image, Vidco, Audio
Compechension-GGeerration
48

---

On Path to Multimodal Generalist: General-Level and General-Benel Bench
Table 11: Performance of multimodal generalists on video comprehension and generation skills.
Video Comprehension Skill (Avg within each WV-CC Group)
TTask Completion
Level Score on Video
Model
#1
#2
#3
#4
#5
#6
#7
#8
#9
#10
#Supported #Win-over- Level-2
Level-3
Level-4
#11
#12
#13
#14
#15
H16
#17
#18
#19
#20
Task
Specialist
So Specialist
37.43
49.64
21.31
23.06
81.85
85.43
54.53
64.83
40.65
30.80
13.92
0.14
48.06
68.96
63.62
77.02
75.08
37.20
44.00
Intern VL-2-L--2.5-8B
27.54
14.51
0.00
0.00
0.000
0.00
55 (43.7%%
5 (4.0%)
5.76
1.24
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
4.85
Intern InternVL-22.5-222 2L-22.5-2B
37,03
32.01
18.71
21.57
0.00
0.00
0.00
0.00
0.00
0.00
55 (43.7%)
26 (20.6%)
6.70
3.76
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
5.30
Qwen2-VL. VLL-2BB
38.22
32.32
19.35
22.70
0.00
0.00
0.00
0.00
0.00
0.00
55 (43.7%)
22 (17.5%)
6.89
5.22
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
5.70
DeepSeck-VL-2
21.30
18,90
12.10
12.10
0.00
0.00
0.00
0.00
0.00
0.00
55 (43.7%)
(4.0%)
3.98
0.64
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
3.20
LLaVA-One-
31,20
31.30
19.10
10.60
0.00
0.00
0.00
0.00
0.00
0.00
56 (44.4%)
21 (16.7%)
5.83
3.75
0.00
Vision-72B
0.00
0.00
0.00
1.70
33.19
25.11
16.75
8.67
0.00
0.00
0.00
71.03
50.95
0.00
Sa2VA-8B
91 (72.2%)
32 (25.4%)
8.31
4.38
0.00
0.00
611.28
0,00
0.00
19.85
37.83
46.36
42.58
48.02
1.48
Sa2VA-26B
35,33
26.33
17.58
10.39
0.00
0.00
0.00
0.00
0.00
0.00
81 (64.3%)
27 (21.4%)
8.81
4.58
0.00
0.00
0.00
0.00
0,00
28.41
38.91
47.10
43.12
48.42
1.70
CoLVA-4B
32.68
26.45
13.55
17.62
0.00
0.00
0.00
0.00
0.00
0.00
63 (50.04)
(6.3%)
4.78
1.24
0.00
0.00
0.00
0.00
45.81
0.00
0.00
0.00
0.00
4.23
Inter IntemVL-2288)
32.69
27.09
14.24
17.61
0.00
0.00
0.00
0,00
0.00
0.00
55 (43.7%)
0 (0.0%)
5.64
0.46
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
4.85
LLV--9
36.14
26.25
15.89
15.53
0.00
0.00
0.00
0.00
0.00
0.00
54 (42.9%)
22 (17.5%)
5.84
3.81
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
4,20
Video
Generation
Skill (Avg within each #V-G Group)
Task Completion
Level Score Vidco
Model
#1
#2
#3
#4
#5
#6
#Task- Sappit
#Win-pplls
Level-2
Level-3
Level-4
SoTA Specialist
69.09
55.79
88.94
62.90
37.79
51.46
VidAgent
52.42
47.73
88.84
63.61
0.00
0.00
30 (65.2%)
0 (0.0%)
25.00
0.00
0.00
LM4LV
0.00
0.00
0,00
0.00
25.90
5.93
8 (17.4%)
0 (0.0%)
6.74
0.00
0.00
NEXT-GPT-VILS
26.78
6.72
130.22
16.03
0.08
0.06
40 (87.0%)
0 (0.0%)
8.34
0.71
0.00
Vitron-V V1
36.74
19.32
116.31
25.09
0.08
0.06
40 (87.0%%
0 (0.0%)
18.72
3.04
0..00
#
Model
Backbone
Size
Modality Support
Paradigm
80
GPX--GG eWueral, vicuna-7b-v1.3
7B
Language, Image, Video, Audio
Comprehenssoon-GGGnerttiio
2024a)
81 VidAgent (Shen et al., vicuna-7b-v0 2023)
7B
Language, Image, Video
Comprehensionn-GGenrrttioo
Note that, for VidAgent we implement HuggingGPT as the prototype agent, and integrate Intern VL-2.558 (Chen et al., 2024c) as video comprehension module, and integrate CogVideo (Hong et al., 2022) as video generation module.
C.2 Full Main Evaluation Results
We note that all the generalists run the evaluation on our General Beseral--Benne Bench data set under zero-shot setting. The overall results of part of the models on image comprehension and generation are presented in Table and Table 3, respectively; video results are shown in Table I; audio results are shown in Table 12; 3D results are shown Table 13; The results of all generalists on NLP tasks are shown in Table 14. The complete performing scores of all MLLMs across all tasks and datasets presented in Appendix JC. Overall, have the following observations.
Table 15: Leaderboard of multimodal generalists (MLLMs) at level-2.
Level 2 Score
Model
Modality
Paradigm
Ranking
of Image
of Video
of Audio
of 3D
of Overall
Unified-io-2XXXI
C+G
20.62
8.56
25.63
0.00
13.70
12
AnyGPT
x//
C+G
23.10
0.00
29.06
0.00
13.04
2
NEXT-GPT-V1.5
C+G
18.69
8.34
25.05
0.00
13.02
3
ImageBind-LLN LLM
C
19.54
12.54
17.52
0.00
12.40
4
49

---

On Path to Multimodal Generalist: General-Level and General-Bende
Level 2 Score
Model
Modality
Paradigm
Ranking
of Image
of Video
of Audio
of 3D
of Overall
Moda Verse-7b-vD
C+G
15.56
7.32
26.10
0.00
12.25
5
Vitron-VI
C+G
30.13
18.72
0.00
0.00
12.21
PandaGPT-13B
C
20.78
9.34
16.98
0.00
11.78
7
VidAgent
C+G
18.21
25.00
0.00
0.00
10.80
InterVL2. 5-8B
C
25.20
8.44
0.00
0.00
8.41
9
Emu2-37B
C+G
30.90
0.00
0.00
0.00
7.73
Sa2VA-26B
C
21.88
8.81
0.00
0.00
7.67
11
LaVIT-V2 (7B)
C+G
29.50
0.00
0.00
0.00
LLaVA- MaAVA-OOe-- Sie-Vision--225
C
23.12
5.83
0.00
0.00
7.24
13
Ooven-AAdio--ian
C
28.61
0.00
Qwen-Audio-Chat
C
0.00
0,00
28.39
0.00
7.10
15
Mini-Gemint
C+G
0.00
0.00
6.975
SEHD-IJaaMA-138
C+G
26.81
0.00
0.00
0.00
6.70
17
GAMA
C
0.00
0.00
26.35
0.00
Qwen2-VL-72 VL-72B
C
19.41
6.89
0.00
0.00
6.58
19
Sa2 VA-BB-
C
17.33
8.31
0.00
0.00
6.41
20
Intern IntemVLL-22522 VL-2.--26B
C
18.73
6.70
0.00
0.00
6.36
21
Qwen2- VL-7B
C
18.42
6.00
0.00
0.00
6.11
22
InterVLL2 5-4B
C
24.41
0.00
0.00
0.00
6.10
23
SALMONN-13B
C
0.00
0.00
23.95
0.00
5.99
24
InternVV.-2-266.
17.55
6.36
0.00
0.00
5.98
25
WavLLM
0.00
0.00
23.49
0,00
5.87
26
Lonkey--OB-chhe
0.00
0.00
VL2.52LL..5-B
23.32
0.00
0.00
0.00
5.83
28
Pengi
23.29
0.00
-LaVA-One-Vision
C
18.32
4.34
0.00
0.00
5.67
30
SALMONN-7B
0.00
0.00
21.09
0.00
5.27
31
InternVLL-225-22
14.70
5.76
0.00
0.00
5.12
32
VL--Seck-VLL-B--htt
19.89
0.00
0.00
0.00
4.97
33
InternVL-2-8B
14.06
5.64
0.00
0.00
4.93
34
GPT4-o
19.67
0.00
0.00
0.00
4.92
35
GPT4-o-4096
19.68
0.00
0.00
0.00
4.92
36
Gemini-1.55 1.5-Pro
19.67
0.00
0.00
0.00
4.92
37
Claude-3.5-Somee
19.38
0.00
0.00
Claude-3.55O0pg
19.00
0.00
0.00
0.00
4.75
39
chatgpt4-o-latest
0.00
0.00
Gemini-1.5- 1.5-Flash
18.54
0,00
0.00
0.00
4.64
41
Col VA-4B
C
13.59
4.78
0.00
0.00
4.59
GPT4-V
C
18.16
0.00
0.00
0.00
4.54
43
GPT4-o-mini
17.79
0.00
0.00
0.00
4.45
44
GLM-VL--hat
C
17.00
0.00
0.00
0.00
4.25
45
Idefes33-BB-Llama3
C
16.71
0.00
0.00
0.00
4.18
46
LLaVA-NeXT-34B
C
16.58
0.00
0.00
0.00
4.15
47
Phi-3.5-Vision-Instre
C
16.46
0.00
0.00
0.00
4.12
48
MiniCPM3-4B
16.46
0.00
0.00
0.00
CogVa.M-Chat
16.31
0.00
0.00
0.00
4.08
50
CoLVA-2B
11.73
0.00
0.00
InternL--Chan CLL-Chtt-VV-S
16.16
0.00
0.00
0.00
4.04
52
DetGPT
16.05
0.00
0.00
0.00
BLIP-3 (XGen-MM)
15.40
0.00
0.00
0.00
3.85
54
LLaVA-NeXT-13B
15.11
0.00
0.00
0.00
3.78
55
Pixtral-12B
C
14.74
0.00
0.00
0.00
3.69
56
ShareGPT4V-133
C
14.72
0.00
0.00
0.00
3.68
57
Yi-vision-v2
C
14.61
0.00
0.00
0.00
3.65
58
Owen-VL-CCat VL-Chat
13.91
5.34
0.00
0.00
3.48
59
ShareGPT4V-77B
13.78
0.00
0.00
0.00
3.45
60
50

---

On Path to Multimodal Generalist: General-Level and General-Bende
Level 2 Score
Model
Modality
Paradigm
Ranking
of Image
of Video
of Audio
of 3D
of Overall
Mini-InnemVL-Chhat4BBBB
13.53
0.00
0.00
0.00
3.38
61
MieemlM--CComgsserr-Viii
C
13.31
0.00
0.00
0.00
3.33
62
DeepSeek-VL.ce
-1.Bbek--11-BBBBaa
C
13.13
0.00
0.00
0.00
3.28
MiniGPT4-LLaMA22-E
C
12.89
0.00
0.00
0.00
3.22
64
MoE-LLLWAA-hh222277--444
12.55
0.00
0.00
0.00
3.14
65
mPLUUG-OW.22LLMMAA227
C
12.21
0.00
0.00
0.00
3.05
66
Cambrian-1-8B
C
11.76
0.00
0.00
0.00
2.94
67
BLIP2
11.65
0.00
0.00
0.00
2.91
miniMonkey
C
11.31
0.00
0.00
0.00
2.83
69
NExT-Chat
0.00
0.00
Audio-GPT4
G
0.00
0,00
8.80
0.00
2.20
71
GPT4RoI-7B
8.49
0.00
0.00
Show-o
C+G
7.78
0.00
0.00
0.00
1.95
73
SpecchGPT-7B-com
G
0.00
0.00
7.22
0.00
PointLLM-13B
C
0.00
0.00
0.00
7.00
1.75
75
LM4LV
G 0.00
6.74
0.00
0.00
1.69
76
PointLLM-7B
C
0.00
0.00
0.00
6.53
1.63
77
Long-LLaVA-9B
C
10.23
5.84
0.00
0.00
1.46
78
3D-VisTA
C
0.00
0.00
0.00
5.41
1.35
79
3D-I.LM-2.18
C
0.00
0.00
0.00
5.41
1.35
80
AMG--LLaVA-Intee M20B
C
4.56
0.00
0.00
0.00
1.14
81
DeepSeek-VL. VV--2
19.21
3.98
0.00
0.00
1.00
82
DeepSeek-V VL-2-small
17.40
3.64
0.00
0.00
Otter
C
3.15
0,00
0.00
0.00
0.79
84
LLaMA-mesh
0.00
1.60
0.40
LISA
1.27
0.00
0.00
0.00
0.32
86
GLaMM
0.94
0.00
0.00
0.00
0.24
87
AvatarGPT
0.00
0.00
0.00
0.21
0.05
88
MotionGPPT-TS
G
0.00
0.00
0.00
0.00
0.00
MotionGPT-LLaMA
G
0.00
0.00
0.00
0.00
0.00
Meta- Metaa-lamaa-.1-a Instrrct
0.00
0.00
0.00
0.00
0.00
Gemma-2-9b-it
0.00
0.00
0.00
0.00
GPT-J
0.00
0.00
0.00
0.00
0.00
ChatGLM-6B
0.00
0.00
Breen255-BBIIntrree
0.00
0.00
0.00
0.00
0.00
InternI. M2-Chat-7B
0.00
0.00
0.00
0.00
0.00
Baichuan2-7B-
0.00
0.00
0.00
0.00
0.00
Vicuna-75 Soaua--b--1.. 11.5
0.00
0.00
0.00
0.00
0.00
Falcon3-7B- Instrrctt
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Yi-lightning
0.00
0.00
0.00
0.00
0.00
GPT-3.5-turbo
0.00
0.00
0.00
0.00
0.00
Table 17: Leaderboard of multimodal generalists (MLLMs) at level-3 where Comprehension and Generation.
Level 3 Score
Model
Modality
Paradigm
Ranking
of Image
of Video
of Audio
of 3D)
of Overall
Sa2VA-26B
14.65
4.58
0.00
0.00
C
15.21
3.75
0..0)
0.00
4.74
2
Owen2-V1--72B
C
12.34
5.22
0.00
0.00
4.39
3
Mini-Gemin Gemini
C+G
17.23
0.00
0.00
0.00
4.31
4
Sa2VA-8B
C
12.39
4.38
0.00
0.00
4.19
5
IntemVL22.559)
C
13.09
1.82
0.00
0.00
3.73
GPT4-o-4096
C
14.68
0.00
0.00
0.00
3.67
7
Qwen2-VL--7
C
12.13
2.47
0.00
0.00
3.65

---

On Path to Multimodal Generalist: General-Level and General-Bende
Level 3 Score
Model
Modality
Paradigm
Ranking
of Image
of Video
of Audio
of 3D
of Overall
GPT4-0
C
14.51
0.00
0.00
0.00
3.63
9
Intern InternV1.-2-2 VL--2-26B
C
8.81
4.81
0.00
0.00
3.41
10
Intern V1.22.5-26
C
9.51
3.76
0.00
0.00
3.32
ChatGPT-o-latest
C
13.02
0.00
0.00
0.00
3.26
12
GPT4-V
C
12.85
0.00
0.00
0.00
3.21
13
Gemini-1.5-Pro
C
12.66
0.00
0.00
0.00
3.17
Claude-3.5-Sonnet
C
11.98
0.00
0.00
0.00
3.00
15
GPT4-o-mini
C
0.00
0.00
0.00
LLaVA- SaeAA-Oe-Wiie VieeViiiin--B
C
10.21
1.54
0.00
0.00
2.94
17
Inter 5-4B
11.59
0.00
0.00
2.90
Monkey-10B-chat
11.59
0,00
0.00
0.00
2.90
19
LL2 5-2B
11.45
0.00
0.00
Claude-3.5-Opus
C
11.08
0.00
0.00
0.00
2.77
21
Gemini-1.5- 1.5-Flash
C
10.85
0.00
0.00
0.00
22
Vitron-VI VI
C+G
7.65
3.04
0.00
0.00
2.67
23
CoLVA-4B
C
9.45
1.24
0.00
0.00
2.67
24
Qwen-Audio-Chal
C
0.00
0.00
10.57
0.00
2.64
25
VL-catL-CCha-V155
C
9.42
0.00
0.00
0.00
2.36
26
Phi-3.5-Vision-Inssruce
C
9.39
0.00
0.00
0.00
2.35
27
DeepSeck-V1-2
C
8.32
0.64
0.00
0.00
2.24
28
I
7.63
1.24
0.00
0.00
2.22
29
GLM-VL-Char
8.67
0.00
0.00
0.00
2.17
30
Owen2-Audio-Instuue
8.53
0.00
31
LLaVA-NeXT-34B
8.24
0,00
0.00
0.00
2.06
32
DeepSeck-VL-7B--Ch.
8.19
0.00
0.00
MiniCPM3-4BB
C
8.11
0.00
0.00
0.00
2.03
34
Long-LLaVA-99
4.21
3.81
0.00
0.00
2.01
35
Yi-vision-v2
C
7.85
0.00
0.00
0.00
1.96
36
CogVLM-Chat
7.77
0.00
0.00
0.00
1.94
37
IntemVL-22-8
7.28
0.46
0.00
0.00
1.94
38
Idefies3-BB-Llama3
7.70
0.00
0.00
0.00
1.93
39
CoLVA-2B
6.60
1.04
0.00
0.00
1.91
40
GAMA
0.00
0,00
7.15
0.00
1.79
41
LLaVA-NeXT-13B
0.00
0.00
0.00
1.72
BLIP-3 (XGen-MM)
6.42
0,00
0.00
0.00
1.61
43
ShareGPT4V-13B
0.00
0.00
1.49
Owen-VL-Chie
C
5.88
0.00
0.00
0.00
1.47
45
DeepSeek- B1.---V1.-BB-B-
C
5.75
0.00
0.00
0.00
1 1.44
Pixtral-12B
C
5.72
0.00
0.00
0.00
1.43
47
DeepSeek- VL-22small
C
5.12
0.52
0.00
0.00
1.11
48
C
5.47
0.00
0.00
0.00
1.37
49
NEXT-GPT-V1.5
C+G
3.24
0.71
1.34
0.00
1.32
50
Mini-IneemVL-Chht--B--W
C
5.21
0.00
0.00
0.00
1.30
51
Emu2-37B
C+G
5.18
0.00
0.00
0.00
1.30
52
C
0.00
0.00
0.00
1.20
ShaneGPT4v.7B
C
4.78
0.00
0.00
0.00
1.20
54
MiniGPTA-LLMMA--.5
C
0.00
0.00
mPLUG-OWI2-LLAMASS
C
4.60
0.00
0.00
0.00
1.15
56
AnyGPT
C+G
1.29
0.00
3.29
0.00
1.15
miniMonkey
C
4.51
0.00
0.00
0.00
1.13
58
Cambrian- 1-8B
C
3.84
0.00
0.00
0.00
0.96
59
DetGPT
C
3.77
0.00
0.00
0.00
0.94
60
LavIT-V2 (7B)
C+G
3.71
0.00
0.00
0.00
0.93
61
SALMONN-13B
C
0.00
0.00
3.61
0.00
0.90
62
ImageBind-LLM
C
1.56
0.72
1.26
0.00
0.89
63
NExT-Chat
C
3.51
0.00
0.00
0.00
0.88
64
52

---

On Path to Multimodal Generalist: General-Level and General-Bende
Level 3 Score
Model
Modality
Paradigm
Ranking
of Image
of Video
of Audio
of 3D
of Overall
SEED-LLaMA-138
C+G
3.49
0.00
0.00
0.00
0.87
65
WavLLM
C
0.00
0.00
3.28
0.00
0.82
66
Unified-io-2-XXI.
C+G
2.11
0.14
1.01
0.00
0.82
67
Moda 7brse--
C+G
0.98
0.23
1.14
0.78
0.78
68
PandaGPT-13B
C
2.35
0.05
0.65
0.00
0.76
69
Audio-GPT4
G
0.00
0.00
3.02
0.00
0.76
70
BLIP2
C
2.79
0.00
0.00
0.00
0.70
71
GPT4RoI-7B
C
2.36
0.00
0.00
0.00
0.59
Pengi
C
0.00
0.00
1.74
0.00
0.44
73
3D-LLM-2.IB
C
0.00
0.00
1.38
3D- VisTA
C
0.00
0.00
0.00
1.07
0.27
75
Show-o
C+G
0.00
0.00
IJSA
C
0.82
0.00
0.00
0.00
0.21
77
Otter
C
0.68
0.00
0.00
0.00
0.17
C
0.44
0.00
0.00
0.00
0.11
79
GLaMM
C
0.41
0.00
0.00
0.00
0.10
80
AvatarGPT
0.00
0.00
0.00
0.21
0.05
81
PoimLLLM-7B
C 0.00
0.00
0.00
0.00
0.00
PoimLLM-113
C
0.00
0.00
0.00
0.00
0.00
MotionGPTTT5
G
0.00
0.00
0.00
0.00
0.00
MoLIMM
G
0.00
0.00
0.00
0.00
0.00
LLaMA-mesh
.4.
G
0.00
0.00
0.00
0,00
0.00
SALMONN-7B
0.00
0.00
0.00
SpecchGPT-7B-com
K(4
G
0.00
0.00
0.00
0.00
0.00
LM4LV
G
0.00
0.00
0.00
VidAgent
C+G
0.00
0.00
0.00
0.00
0.00
Meta-Llama-3.--8BBmmas
0.00
0.00
0.00
0.00
0.00
Gemma-2-9h-it
0.00
0.00
0.00
0.00
0.00
GPT-J
/
0.00
0.00
0.00
0.00
0.00
ChaiGLM-6R
0.00
0.00
0.00
0.00
0.00
Qwen2.5-7B Sannd.5--B-Instruee
0.00
0.00
0.00
0.00
0.00
InternL Chat-7B
0.00
0.00
0.00
Baichuan2-7B-BB.
0.00
0.00
0.00
0.00
0.00
Vicuma-7b-VI.
0.00
0.00
0.00
Biecon3-BB-Intruae
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Yi-lightning
0.00
0.00
0.00
0.00
0.00
GPT-3.5-turbo
0.00
0.00
0.00
0.00
0.00
C.3
Level and Leaderboard of Multimodal Generalists
Based on the overall performance of each model across the various modalities and tasks, we rank all the compared models according to the General-Leve scoring defined in $ 3.2. Tables 15, 17 and 19 present the specific scores and rankings of multimodal generalists at different General-Levels. Note that no generalists score non-zero at Level-5, and thus we do not show a rank at Level-5. Figure visualizes these leaderboards.
As shown, for all the current MILMs at level 2, Unified-io-2-XXI. (Lu et al., 2024a) ranks the best, followed by AnyGPT (Zhan al., 2024). Surprisingly, GPT-4V and GPT-4o did achieve the expected rankings at level 2. While the GPT series excels in the individual tasks it supports, generalists, they fall short in skill coverage compared to some open-source MLLMs. This is because, to rank higher at level 2, models must not only perform well on different tasks but also support as many modalities and tasks possible.
Next, MLLMs that can manage to reach level become different. Sa2VA-26B (Yuan et al., 2025) ranks at the top, while LLa Vision-72B (Li et al., 2024d) and Qwen2-VL-72B (Wang et al., 2024a) achieve the second and third places, respectively. Some high-ranking level-2 models lost their places at level 3. This lies in the fact that most MLLMs are limited
53

---

On Path to Multimodal Generalist: General-Level and General-
Table 12: Performance of multimodal generalists on audio comprehension and generation skills.
Audio Comprehension Skill (Avg within each NA-C Group)
Tass Completion
Level Score on Audio
Model
#1
#2
#3
#4
#5
#6
#7
#8
#9
STask-Supprt
#Win-Spelst
Level-2 Level-3
Level-4
SoTA Specialist
87.27
79.08
70.62
79.00
71.87
62.90
58.70
77.90
78.07
Owen-AAdio-Chai
56.93
68.77
76.80
37.70
47.71
19.79
56.44
85.15
78.50
30 (100.0%)
6 (25.0%)
28.39
10.57
0.00
Qwen2-Audio-He -e--Audio--aa Instru
72.65
74.80
61.40
36.80
45.82
13.45
61.68
78.95
67.99
24 (100.0%)
6 (25.0%)
28.61
8.53
0.00
GAMA
57,00
64.20
68.00
53,20
18.43
26.95
48.85
85,55
61.80
23 (95.8%)
4 (16.7%)
26.35
7.15
0.00
Pengi
52.88
60.07
56.70
36.78
19.77
19.55
42.95
77.40
61.17
23 (95.8%)
(4.2%)
23.29
1.74
0.00
SALMONN- 13B
67.89
56.33
67.80
29.45
24.67
19.36
43,95
76.55
56.67
23 (95.8%)
(8.3%)
23.95
3.61
0.00
WavLLM
64.45
41.07
71,20
30.08
31.30
26.55
45.75
61.40
64.57
24 (100.0%)
(8.3%)
23.49
3.28
0.00
NEXT-GPT-V1.5
43.23
29.13
65.80
26.70
14.47
25.65
47.95
70.20
69.43
24 (100.05%)
(0.0%)
25.05
1.34
0.00
PandaGPT (133)
41.80
20.23
45.20
20.98
8.47
20.50
42.25
54.80
65.83
24 (100.0%)
(0.0%)
16.98
0.65
0.00
Moda Verse-7b-ve 7b-v0
34.10
16.37
32.80
15.20
6.60
8.90
35.05
49.20
60.13
23 (95.8%)
(0.0%)
26.10
1.14
0.00
Any-GPT
44,50
32.13
63,40
48.08
16.27
36.40
52.65
67.95
44.63
23 (95,89)
(4.2%)
29.06
3.29
0.00
Unified-in- I-XXXX.
30.15
27.60
56.10
28.58
15.47
38.35
38.70
63,50
60.63
24 (100.05%)
0 (0.0%)
25.63
1.01
0.00
Audio Generation Skill (Avg within
each #A-G Group)
Task Completion #Win-Spelst
Level Score Audio
Model
#1
#2
#3
04
#5
N6
N7
48
#9
#10
#11
#Task-Supprt
Level-2 Level-3 Level-4
SoTA Specialist
31,50 3.82
3.64
4.68
41.54
51.40
11.52
6.80
8.33
22.88
20.33
2n XXII. 18.36 2.03
5.11 40.52
16.41
24.31
16.97
86,23
94.52
0.25
2.24
17 (85.0%)
O (0.0%)
25.63
1.01
0.00
Any-GPT
23:50 3.24 4.57 33.58
13.38
14.05
27.49
45,36
83.89
0.25
2.47
17 (85.0%)
1 (5.0%)
29.06
3.29
0.00
NEXT-GPTEVIL5
13.60 1,15 4.07 50.51
34.51
1.35
12.36
96,70
99.23
0.25
7.77
17 (85.0%)
1 (5.0%)
25.05
1.34
0.00
AudioGPT
0,50 1.32 4.61 23.10
29.48
0.00
0.00
46,30
79.98
0.25
0.00
(65.0%)
1 (5.0%)
8.80
3.02
0.00
SpeechGPT
0.10 2.79
4.44 32.35
0.00
0.00
30.24
85.54
0.25
0.00
(55.0%)
0 (0.0%)
7.22
0.00
0.00
VodaV
12.30 1.15 4.29 50.50 28.99
1.05
16.45
100.00
100.00
0.25
4.17
(85.0%)
(10.0%)
26.10
1.14
0.00
Table 13: Performance of multimodal generalists on 3D comprehension and generation skills.
3D
Comprehension
Skill (Avg within each #D-C Group)
Task Completion
Level Score on 3D
Model
#1
#2
#3
#4
#5
#6
#7
#8
#9
#10
#11
#12 #13
#Task #Task-Suppet #Win Spelst
Level-2 Level Level-4
SoTA Specialist
96.24
98.35
97.78
78.50
70.02 81. 55.00
88.28
75.20
9.96 68.52
47.14 22.30
/
3D-VisTA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
46.37
0.00
(23.3%)
(6677%
5.41
1.07
0,00
MMi7L
46.16
7.50
72.86
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
8 (26.7%)
(0.0%)
6,53
0.00
0.00
LMinILLM-133
48.79
10.00
78.14
0.00 0.00
0.00
0.00
0.00
0.00
0.00 0.00 0.00
0.00
9 (30.0%)
0 (0.0%)
7.00
0.00
0.00
3D-LLM
0.00
0.00
0.00 0.00 0.00
0.00 0.00
0.00
0.00 0.00 0.00 46.34
0.00
7 (23.3%)
1 (3.3%)
5.41
1.38
0.00
AvatarGPT
0.00
0.00
0.00 0.00
0.00
0.00 0,00
0.00
0.00 0.00 0.00 0.00
12.70
(3.3%)
(0.0%)
0.21
0.21
0.00
3D Generation
Skill (Avg within each #D3G Group)
Task Completion #Win-Spelst
Level Score on 3D)
Model
#1
#2
83
#4
M5
#6
#7
#8
89
STask-
Level-2 Level-3 Level-4
SoTA Specialist
0.22
7.12F
24.42
25.69
78.06
83.64
65:00.02
6540.02
0.23
MotionGPT-TS
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.51
(4.5%)
(0.0%)
0.00
0.00
0.00
MotionGPT-LIaMA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.60
(4.5%)
0 (0.0%)
0.00
0.00
0.00
LLaMA-Mesl
0.00
0.00
0.00
17.55
0.00
0.00
0.00
0.00
0.00
(4.5%)
(0.0%)
1.60
0.00
0.00
to multimodal content comprehension and lack support for generation tasks. GPT-4V and GPT-4o win top-10 positions here.
Finally, only 3 MLLMs that reach level 4 can be seen, i.c., Mini-Gemini, Emu2-37B, and Vitron-VI At this level, these models exhibit synergy across both comprehension and generation. Besides these three models, no other systems exhibit such capability.
Most critically, no model has yet demonstrated the ability to enhance language intelligence through non-language modalities, underscoring the significant challenges in the pursuit of true AGI. And this definitely our goal to reach the most capable multimodal generalists.
C.4 Capability BreakDown
We now take closer look, as multimodal generalists, at how well different MILMSs support tasks and modalities.
54

---

On Path to Multimodal Generalist: General-Level and General-Bence
Table 14: Performance of generalists on language-related (NLP) skills.
Language Skill (Avg within each III. Group)
Task Completion
Level Score Level-5
Model
#1
#2
#3
#4
#5
#6
#7
#8
#9
#10 #21
#11
#Supported #Win-over-
#12
#13
#14
#15
H16
#17
#18
#19
#20
#22
Task
Specialist
So Specialist
62.62
86.23
76.78
71.00
58.02
62. 80
75.11
77.84
79.70
71.91
28.27
86.95
0.31
94.40
91.41
86.05
86.03
84.
83.67
58.61
77.73
92.38
Meta- -lat--
39.75
56.76
54.21
60.52
20:01
37.17
12
53.23
44.49
14,80
113 (98.3%)
0 (0.0%)
0.00
8B-Instruct
45.34
7,95
76.40
51.80
65.90
41.10
24.49
30.70
8.08
32.40
54.35
CharGLM-6b
28.97
33.24
37.24
46.10 10
19.39
27.84
18.85
35.88
27.85
38.51
13.93
96 (83.5%)
0 (0.0%)
0.00
10.91
41.80
45.81
24.50
16.45
0.12
8.41
2.70
23.80
45.37
Vicuna-7b-v1.5
24.78
11.18
33.44
41.19
4.51
13.25
19,94
35.27
54.81
40.58
5,06
72 (62.6%)
0 (0.0%)
0.00
43.98
11.41
0.00
0.00
0.000
0.96
0.07
0.47
0.00
23.13
15.440
Falcon3-7B-Instrue
36.79
58.36
49.91
56. 80
21.38
37. 12
32.03
42.11
55.79
42.07
15.56
112 (97.4%)
0 (0.0%)
0.00
48.15
5.15
88.80
85.89
45.65
42. 86
27.
34,22
11.19
39.80
58.75
Ministrul-SB-
41.74
54.21
49.53
51.92
39.32
40.49
1.300
22.86
56.87
43.46
13.73
112 (97.4%%)
(0.0%)
Instruct-2410
23.39
11.08
84.80
72.60 60
56.70
37.1
6.28
31.38
9.37
25.53
40.44
Yi-l.ightming
41.73
60.54
55.39
60.51
20.53
39.83
22.45
43.57
62.52
42.03
15.29
113 (98.3%%
0 (0.0%)
0.00
52.68
5.37
72.60
56.24
64.75
43.59
28.27
42. 84
25.34
29.27
60.49
GPT-4V
27.55
62.40
34.57
32.55
14.43
27.84
27.79
36.07
65.36
42.11
13.96
113 (98.3%))
0 (0.0%)
0.00
44.56
3,16
86,20
83.23
65.10
53.82
54.14
45.45
33.86
26.46
24.24
GPT-4o
26.25
62.57
33.98
31.50
16.20
26.26
27.14
36.64
66.86
42.69
14.49
113 (98.3%)
0 (0.0%)
0.00
46.41
2.58
85.40
86.30
67.50
56.10
57.42
46.97
39.52
32.07
28.50
Emu2-32B
32.91
45.43
47.04
39.56
27.74
31.24
39.04
41.72
45.48
46.35
13:05
113 (98.3%)
(0.0%)
0.00
50.15
9.53
57.54
48.78
43.76
36.67
19.84
24.01
13.78
26.47
31,72
DeepSeck-VL-72
29.97
44.39
55.55
20.36
40.49
57.93
49.85
48.73
27.03
76
10.37
114 (99.1%)
0 (0.0%)
0.00
79.68
83.00
62.20
50.60
62.30
46.87
4.12
28.46
8.11
31.80
40.97
(wen2-V1.-78
23.91
27.51
37.68
46.40
17.84
20.96
36.25
29.29
35.42
35.58
12.62
94 (81.7%)
0 (0.0%)
0.00
37.23
6.48
64.00
37.00
3,50
20.50
0.24
4.87
6.00
20.87
21.79
LLaVA-One-
50.44
41.98
54.55
61.13
29.87
$6.99
35.24
43.27
55.23
41.49
17.73
110 (95.7%)
0 (0.0%)
0.00
Vision-72B
43.81
33.55
84,80
10.43
59.35
34.91
42.94
28.63
19.26
52.20
71.95
LL25m_L2
42.93
47.76
59.54
31.17
42.86
32.72
50.98
43.02
30.85
51.23
9.07
114 (99.1%)
0 (0.0%)
0.00
71.96
75.20
55.40
68.40
56.75
55.60
22.12
36.48
9.80
32.13
53.67
Long-Ilava
26.50
49.49
34.81
39.62
17.83
33.14
20.63
38.44
48.90
38.
6.30
107 (93.0%)
(0.0%)
0.00
48.44
11.40
68.60
41.70
52.65
31.42
2.33
21.52
7.40
29.47
42.07
NEXT-GPT-V1.5
20.66
22.42
32..55
39.51
4.19
16.47
16.49
32.67
51.49
37.73
5.06
79 (68.7%)
0 (0.0%)
0.00
42.09
1.06
68.90
43.20
28.78
9.24
4.44
6.16
7.22
24.17
18.86
SEED-LLaMA-I3B
18.11
32.55
26.54
25.19
8.80
18.85
11.68
15.89
21.64
23.56
4.80
109 (94.8%)
(0.0%)
0.00
20.84
11.16
13.20
34.80
28.98
19.93
2.59
10.31
2.10
12.07
12.41
29.34
16.70
47.05
56.85
5.09
14.85
21.27
39.24
57.40
40.65
4.93
LLaMA-Mesh
84 (73.0%)
0 (0.0%)
0.00
44.19
11.41
0.00
0.00
0.000
1.50
0.65
0.56
0.01
23.13
21.57
MinIGPT-LLMMA2
28.17
1,5.73
40.98
45.15
3,71
10.99
25.97 97
43.03
35.22
36. 14
10.42
84 (73.0%)
0 (0.0%)
0.00
42.56
7.46
0.00
0.00
0.00
2.08
6.36
4.52
0.00
17.55
21.23
Table 19: Leaderboard of multimodal generalists (MLLMs) at level-4, where Comprehension and Generation.
Level Score
Model
Modality Paradigm
Ranking
of Image
of Video of Audio
of 3D
of Overall
Mini-Gemini
C+G
6.23
0.00
0.00
0.00
1.56
12
Vitron-V1
C+G
4.59
0.00
0.00
0.00
1.15
2
Emu2-37B
C+G
1.25
0.00
0.00
0.00
0.31
3
Task Supporting. In Figure 25, we present all skills (meta-tasks) supported by different MILMs across various modalities and within the scopes of comprehension and generation. Overall, MLLMs show relatively lower task support for 3D tasks and skills, compared with the status for other modalities. Also, the coverage of comprehension-relan should be generally higher than that of generation-related skills. This trend is consistent with the results observed in previous experiments. A significant trend we identified is that MILMS tend to support skills within only one (or few) task paradigms. This results in differentiated skill support across different MLLMs, with few models capable of supporting a wide range of skills across diverse tasks.
skills by MLLMs
Moreover, most MI.LMs are inclined to focus on basic skills or tasks with simpler and more straightforward definitions,
55

---

On Path to Multimodal Generalist: General-Level and General-Benu
Compeshessica
Generation
GPT-4V
Imagee
GPT-IV
GPT-40
65.1
GPT-5o
Gemini
654
Gemini-1.5-PP 1.5-Po
Emu2: 32
Video
NLP
EmI1--228
InternV L2.5-88
Mini-Giemai
Mini-Geminl
Qwen2-VL-722 VL-72B
Qwa2-VVL-728
36.4
SEED-LLaMA-1O0
SEED-LLaMA-IdB
Vitra-V1
NEXT-GPT-V1.5
Vitra-V1
B0.6
50.2
AnyGeT
NEXT-GPT-V1.5
Audio
3D
ModaV ModaVerse-7B-00
AmyGPT
56.2
81.5
Figure 20: Supporting modality.
ModaV
30
Figure 21: Supporting generation and comprehension.
while tasks requiring complex content output and advanced skills are supported by far fewer models. For instance, compared to gaars--grained visual understanding tasks (e.g., captioning and classification), tasks with more complex definitions-suchh as pixel-level object detection, image/3D segmentation, video tracking, and image generation-are supported by far fewer existing MLLMs. However, we observe that a few MLLMs stand out for their broader support of cross-modal skills, such as Vitron-V (Fei et al., 2024a). Thanks to their architectural designs, these models demonstrate wider range of task support compared others. We emphasize that supporting as many task paradigms as possible is critical requirement for developing more capable multimodal generalists.
Modality Supporting. The broader the range of supported modalities, the more general and versatile the model's capabilities are. Our benchmark emphasizes the evaluation of MLLMs' all-modality capabilities. As shown in the experimental results above, most MILMs support only a single modality (excluding the language modality, which is inherently supported by LLMs). To further illustrate this, Figure 20 compares the multimodal support capabilities several top-performing MLLMs. In general, there are very few MILMs capable of supporting multiple modalities simultaneously. In most cases, MLLMs support one non-language modality, e-g.. GPT-4V (OpenAI, 2022b), Emu2-32B (Sun et al.. 2024), Mini-Gemini (Li al., 2024c), Intern VL (Chen al., 2024c). Only few MILMs stand out with broader cross-modal or even all-modality support capabilities, encompassing language, image, video, and audio modalities. Examples include NEXT-GPT-VIS (Wu et al., 2024a), Unified-io-2-XXx (Lu et al., 2024a), and AnyGPT (Zhan et al., 2024), etc. Thanks to their architectural designs, these systems demonstrate a wider range of modality and task support compared to others.
Capabilities on Comprehension vs. Generation. Within a single modality, tasks and skills can be categorized into comprehension and generation types. Here, we explore the capabilities of different MILMS in supporting these two paradigms. We select several representative MLLMs that can or cannot support both comprehension and generation within various modalities for comparison. Figure 21 directly presents the statistics on these models' capabilities these two aspects. Overall, support for content comprehension significantly outweighs support for content generation. This phenomenon aligns with practical realities, as modeling tasks for comprehension typically involve expressing the understood content in language, which is relatively straightforward. In contrast, generating multimodal content requires additional efforts in model decoding and extra training, making it a more challenging capability to achieve. It is also evident that different models exhibit varying balances between comprehension and generation capabilities. Among them, Vitron-VI (Fei et al., 2024a) demonstrates the most comprehensive and well-roundes capabilities both comprehension and generation to date, i.e., supporting the largest number of both paradigms.
C.5 Analysis and Discussion on Synergy
Next, we conduct finer- finer-grained analysis of the synergy performance of various MILMMs
56

---

On Path to Multimodal Generalist: General-Level and General-Bene Benchh
36
Imige Comprr
21
GPT-4V
1.c-nil-.5-PProo
Eam2-32B
Owen2-VL-PP
DeepSock VL-7B
1ntemV L2-80
Image:
3 5
11 13 15 SEED-LLaMA-14R
3
5
9
11
13 15
3 5
11 13
7
11
13
1
7
11
13
15
Emm2-378
AnyGPT
LavIT-V2 (78)
NEXT-GPT-V1.5
Vitron-V
19
10
10 13 16 19
13 16
19
4
13
16
19
16 19
3
5
3
5
InternV L-2.5-268
Owen2-VL-728
DeepSeek- VL-2
Sa2VA-8B
VidAgent
LMLLV
Video Comprehension
Video G Gemeration
7
5 7
5
7
3
5
11
1
3
7 3D-VisTA
8
9
11
13
3
5
9
11
9
Qwwen-A wuuoo-oo-Chh
SALMONN- -18
Any-GPT
3D-LLM
MotionGPT-TE GPT
Audio Comprehene ehensionn
Audio Generation
3D Comprehension
3D Generation
Figure 22: Visualizations of synergy effects between all different skills of various MLLMs.
Language
Audio
3D
Vidao
Image
Image
Video
3D
Andio
Langaage
Imago
Video
3D
Andio
Langugge
Image
Video
3D
Andio
Language
Image
Video
3D
Andio
Language
Image
Video
3D
Audio
Langunge
Image
Vidco
3D
Audio
Langonge
AnyGPT
PandaG PT-13B
ImageBind-LLl -nd- LLM
ModaV bb
Unified-io-2-XXI
NEXT-GPT-V15
Figure Visualizations (symmetrised) of synergy effects between modalities of various MLLMMs.
Synergy Across Skills. First, we examine the different synergy effects exhibited by models across various skills. Skills are categorized based on different modalities and further divided into comprehension and generation categories. We explore the synergy effects displayed by top-performing MILLMs within these skill groups. Technically, we calculate the synergy score for tasks (skills) based on the level-3 score algorithm from General -evell ($ 3.2). Then we count the number of synergy tasks for each model and use scores exceeding SoTA specialists as weights. Figure 22 visualizes the results with heatmaps. It is shown that different models exhibit varying levels of cross-task synergy capabilities. Overall, models that achieve higher level-3 scores tend to display denser clusters of highlighted cells in the heatmap. Also, we observe that synergy effects are more likely to occur among closely related skills, as knowledge and information are more easily transfcrable between similar tasks. This trend is consistent across different modalities. Furthermore, generation tasks seem exhibit stronger synergy effects compared to tasks within the comprehension category.
57

---

On Path to Multimodal Generalist: General-Level and General-Bench
Synergy Across Modalities. Finally, we analyze whether different models have learned synergy effects across modalities. The approach is similar to the previous methods, where calculate instances where performance across modalities exceeds that of SoTA specialists. Figure 23 visualizes the performance of 6 representative strong-peee proong-performing MLLMs that cover as many modalities as possible. Several notable trends emerge. First, we observe that there is significant synergy occurs between the image and video modalities. This is reasonable, as static image information and dynamic video content are both fundamentally visual nature, allowing for substantial information sharing that enhances performance across tasks.
Most strikingly, although language appears to exhibit a synergy effect with various other modalities, this effect is in fact unidirectional, specifically from language + other modalities. Based on our previous results on NLP tasks, no synergy has been observed from any other modality to the language modality. While in theory, the audio modality should be closely related to language, and we would expect observe synergy between audio and language tasks, this is not reflected in current results. This limitation is likely due to the reliance of audio-based LLM architectures on the language intelligence of LLMs, which has not yet translated into performance improvements that exceed those of NLP SoTA specialists. We thus strongly urge the MLLM community to prioritize enhancing cross- coss-modalit synergy capabilities to advance the development of truly comprehensive multimodal generalists.
Synergy Across Comprehension and Generation. We further investi- gate the synergy across comprehension and generation, which represents broader type of synergy than the task level. Using similar approach, based the level-4 score algorithm from
6
calculate the synergy General Level. Specifically, count the frequency of synergy occur- rences for each model and use the performance increments exceeding SoTA specialists as weights. The total sum (after normalization) constitutes the synergy score across comprehension and generation. Figure 24 presents the comparisons for only models. As shown, Mini-Gemini (Li et al., 2024c) demonstrates the best synergy effects this level, securing the top rank General -Levell leaderboard. Also, observe that the coos-ccomprehension-ggnneee -enerratonnnn synergy is only pronounced in the image modality, as these 3 models all support only image modalities.
4.59
1.25
Mini-Gemini Figure 24: Synergy strength between com- prehension and generation.
Vitzon-V1 V1
Emu2-37B
58

---

On Path to Multimodal Generalist: General-Level and General-Benele
CPV- WV
Gccaiis 15-Pru
Exa2 308
QW-x2-VL-22 VL-FIB
Doepdeekl VL-7B
Vino- V1
NEXT GPE-LS
DuagBod-VLL VL-2
U.alfA-One Visee-n-----nnn 728
Vtallaal to 2XXL
SX2VA-288
VidAguai
Qwand aualo Eastrat
Any GPT
SDX 3D-VAITA
LLaKA- Mesh
Car
172 Trckk
Figh
Ecisso
RCCD Sen Eay
Laight
SaySip Trua
Nakccigg Ooc
Fatiel
134 Frams
In ipphlokki
chams
Image Comprehension
Image Gemeration
Video Comprehension
ArMACAY
AodiCe:
Dalng
Kaek
Cac.addy
Ea
VIRM
Aneel Nowei
Spen Thex
Generation
Audio Comprebension
Audio Generation
2C Cougtt
Dop
Cl
2B MalieAAn
Decaa
Sca
Ca
04
Salon
Desccie
Beach
XTI
Mes
New
10 Dintely
Techuw
IdCS
Ing Sm
Tep30 Oe
Comprehension
Generation
NLP
Figure 25: Visualization skill support in various multimodal generalists.
C.6 Results of Image-related Tasks
Image Comprehension Results. The complete results of all models image comprehension presented in Table 20 to Table 38. For the full list of results of all tasks, please visit the arXiv full version of the paper: abs/ 2205 04620.
59

---

On Path to Multimodal Generalist: General-Level and General-Bench
Table 20: Results on Image Comprehension Group, from #I-C-1 to #I-C-5. Scores over SoTA specialists are bolded.
#I-C-1
#I-C-2
#I-C-3
#I-C-
#I-C-5
Model
Rocog)
(Code Gen)
(Crock Den
(Disesse Dat)
(Disease Recog)
#1
ff2
#3
#1 1
#1
#2
#1
#1
#2
#3
#4
#5
#6
SoTA Specialist
21.70
82.30
49.80
53.32
63.08
21.00
22.30
40.52
40.50
35.90
38.42
16.40
62.40
GPT4V
57.00
87.97
63.30
58.64
79.08
0.00
0.00
56,60
99.75
90.92
38.01
49.40
62.40
GPT4o
58.70
90.10
72.80
63.42
86.46
0.00
0.00
69.25
98.64
94.97
50.89
53.00
62.60
GPT4o-mini
25.70
88.50
70.20
61.38
77.85
0.00
0.00
75.00
96.93
50.14
44.55
40.40
49.60
GPTT440-00%
24.90
90.00
89.40
0.00
89.54
0.00
0.00
74.43
99.75
99.88
63.56
52.00
62.40
ChatGPT-4o-latest
29.60
66.90
80.40
57.28
85.54
0.00
0.00
63.22
98.40
97.23
37.21
47.20
59.34
Claude-3.5-Sonnet
46.67
87.90
68.60
60.86
80.30
0.00
0.00
66.32
97.72
78.40
44.37
47.38
57.23
Claude-3.5-Opus
43.16
88.58
64.40
57.69
79.90
0.00
0.00
63.10
96.64
76.74
43.48
45.40
54.76
Emu2-32B
32.11
82,26
46.91
7.31
73.23
0.00
0.00
41.67
40.50
35.90
40.99
38.40
50.40
DetGPT
24.49
76.13
55.92
0.00
59.07
16.08
3.25
31.89
39.98
28.73
31.09
26.20
44.60
I22emBL
27.10
73.57
79.20
4.86
0.00
0.00
48.56
53.44
30.11
27.20
IntemV1.2.5-4B
32.30
63.77
39.80
4.97
34.15
0.00
0.00
60.06
46.31
21.65
36.63
25.20
62.20
IntemVL2..52B
26.80
76.73
61.80
4.64
66.22
0.00
0.00
37.36
90.79
23.04
39.41
31.20
60.40
Monkey-10B-chh
24.60
83.03
47.80
0.00
37.23
0.00
0.00
36.49
94.59
0.00
28.12
38.00
62.40
BeepSeck-VL-BB-hhi
20.00
79.70
60.20
4.27
67.57
0.00
0.00
37.64
100.00
19.57
43.56
28.20
62.40
Qwen2-VI--BB
20.00 85.13
89.40
3.61
58.78
0.00
0.00
72.70
48.86
73.72
42.38
38.40
56.20
Owen-VL-Chan
20.00 72.66
56.39
3,40
44.30
0.00
0.00
27.01
48.85
40.05
26.93
20.40
63.20
MoF-LLAAVA-Ph22--7B-----e
20.20 67.67
63.55
1,90
64.62
0.00
0.00
52.59
48.86
34.24
36.83
20.00
62.60
mPLUGG-WWI--LLMAA211
21.09 75.76
60.74
0,00
52,00
0.00
0.00
47.87
47.60
18.02
21.20
45.60
Phi-3.5-Vision-IInttuee
20.40 83.97
61.60
3.44
68.31
0.00
0.00
50.86
59.54
54.34
25.74
26.80
38.40
Cambrian-1-8B
11.10 75.73
48.29
0.00
55.08
0.00
0.00
38.51
0.42
43.00
33.66
19.60
5.60
MiniGPTA-LLMMA--77
28.80
58.70
27.52
0.40
65.23
0.00
0.00
51.44
82.65
77.18
97.32
10.74
55.03
Intem Vi-Cat---hat--1-5
45.80
86.90
81.40
0.00
79.60
0.00
0.00
64.30
83.60
56.00
34.60
32.60
49.60
Mini-IntemVVL-CHH -hateeeee---aa----VL--
34.40
81.10
53.40
0.00
72.60
0.00
64.60
65.70
58.20
27.00
56.00
MlemLLM-XComppser2-VV1
20.60
79.90
43.40
3.40
75.00
0.00
0.00
42.50
78.90
67.50
21.30
21.60
33.40
GPT4RoI
21.70
57.20
35.80
8.10
69.80
0.00
0.00
51.10
24.00
14.30
3.70
16.70
51.60
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
-eNN-eeTTT13B
42.60
80.73
66.80
8.67
73.85
0.00
0.00
52.87
51.97
47.21
42.18
35.60
44.80
LLaVA-NeXT-34B
52.60
83.90
70.40
10.32
76.00
0.00
0.00
60.35
56.39
43.44
39.80
37.40
55.20
Pixtral 12B
46.40
77.43
65.20
3.25
65.85
0.00
0.00
54.02
46.07
52.93
34.06
31.80
49.20
SEED-LLaMA-13B
28.30
57.93
53.80
0.00
63.69
0.00
0.00
34.48
46.68
55.87
30.69
28.20
47.60
BLIP2
33.90
71.47
44.60
51.20
0.00
0.00
46.26
30.10
25.00
42.80
MiniMonkey
4.90
34.77
49.40
0.00
12.62
0.00
0.00
0.00
0.00
0.00
0.20
4.08
69.20
DeepSeek Dee-Seek-VL-7B
23.30
79.53
57.80
0.00
67.69
0.00
0.00
42.66
95.45
37.50
32.28
26.20
64.60
LISA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Chat
46.35
85.57
64.40
12.32
84.62
0.00
0.00
54.31
58.48
52.51
42.38
36.80
54.20
ShareGPT4V-7B
40.76
78.20
54.20
8.45
72.92
0.00
0.00
38.79
49.51
48.32
30.50
31.40
48.40
ShareGPT4Vv-13B
44.19
80.03
57.60
10.37
75.69
0.00
0.00
49.43
53.69
51.96
35.45
35.60
52.60
BLIP-3 (XGen-MM)
43.28
82.63
61.80
8.54
82.15
0.00
0.00
52.87
59.71
55.45
37.43
34.20
50.60
AnyGPT
12.20
45.70
47.60
53.54
0.00
17.15
43.98
47.17
20.59
18.00
MiniCPM3-4B
47.30
79.13
68.80
4.68
74.77
0.00
0.00
57.18
56.76
49.16
39.80
36.80
49.20
LaVIT-V2 (7B)
36.90
55.17
55.40
0.00
62.15
0.00
0.00
34.48
49.88
52.23
32.87
27.40
48.60
51.10
72.20
70.20
17.92
81.85
0.00
0.00
63.79
53.44
56.28
44.16
39.80
54.80
Gemini-1.5-Pro
47.60
91.20
78.20
23.41
78.77
0.00
0.00
69.54
98.90
54.89
44.55
46.20
60.20
Gemini-1.5-Flash
42.10
87.30
71.60
25.79
75.69
0.00
0.00
67.24
98.03
52.65
40.20
42.20
56.40
OMG-LLaVA-ImtemLLNN
3.10
4.20
1.40
4.03
3.70
0.00
0.00
6,90
7.25
3.77
1.58
2.60
3.40
Idefies3-8B-Llamme
43.10
79.30
62.80
11.50
72.00
0.00
0.00
58.62
69.41
50.98
42.19
41.80
52.20
NEXT-GPT-VI.5
34.89
66.38
39.87
3.70
65.34
0.00
0.00
36,45
43.79
53.47
21.56
15.40
32.50
Vitron-V1
39.78
60.75
42.39
3.90
66.75
36.40
2.30
29.38
50.32
53.46
32.42
13.68
34.70
Otter
0.00
7.70
31.26
0.00
0.00
0.00
0.00
18.15
0.00
0.00
0.00
1.00
0.20
Show-o
20.00
0.18
4.21
0.00
0.00
0.00
0.00
31.89
51.14
54.33
20.19
2.21
37.40
NExT-Chat
19.92
45.76
63.54
0.00
68.12
0.00
0.00
26.82
42.61
47.31
11.74
22.41
16.64
Yi-vision-v2
24.60
79.40
82.96
2.13
76.31
0.00
0.00
47.13
52.95
15.78
40.79
30.20
60.00
Owen2-VL-72B
24.13
84.73
92.08
5.74
71.27
0.00
0.00
80.79
46.72
53.24
55.30
41.00
62.40
60

---

On Path to Multimodal Generalist: General-Level and General-Benel
Table 21: Results on Image Comprehension -ooup, part A.
#I-C-6
Model
(Doe VQA)
#1
#2
#3
#44
#5
#66
#7
#8
#9 1
#10
#11
SoTA Specialist
32.72
10.64
11.76
19.40
17.80
26.71
20.92
30.94
16.11
23.91
13.79
GPT4V
45.23
40.78
35.64
38.80
46.46
35.61
28.61
27.61
30.61
31.62
28.61
GPT4o
48.16
48.58
41.58
42.63
50.51
39.04
29.04
30.04
31.04
32.04
31.23
4PT4o-mini
41.18
38.30
40.59
38.80
45.45
30.83
30.83
31.83
32.83
33.83
30.23
GPT-4o-4096
49.63
47.52
46.53
46.45
51.18
45.21
43.51
46.41
48.59
42.81
46.29
CharGPT-4o-latest
44.49
45.39
44.55
40.44
47.81
32.88
39.75
43.65
30.27
30.28
45.85
Claude-3.5-Sonnet
44.38
42.54
39.07
39.97
46.59
34.58
28.74
28.91
31.14
32.49
29.86
Claude-3.5-Opus
42.65
42.22
38.18
36.83
44.33
34.28
26.74
26.22
30.67
27.91
26.64
Emu2-32B
32.72
29.79
28.67
30.42
26.27
25.25
19.06
21.40
18.26
19.45
20.55
DetGPT
30.88
18.44
31.86
34.55
29.40
27.42
19.47
22.87
18.84
19.64
20.93
V22.58B
28.47
43.99
41.89
35.77
30.83
35.06
36.67
41.67
25.49
35.23
IntemVV122.544B
31.54
26.59
36.50
37,34
30.33
27.89
32.84
35.68
38.71
25.03
33.90
V22.52
31.63
29.07
32.56
37.77
33.72
29.91
32.53
33.15
40.21
22.61
33,48
Monkey- 10B-ccat
32.42
27.54
34.96
29.52
29.71
29.47
26.77
30.89
35.36
24.40
30.57
DeepSeck-VL-7B-Che.
7.43
7.55
7.40
8.82
8.94
10.65
9.72
8.08
11.39
11.09
10.51
Qwen2-VLL-BB
45.83
52.00
46.70
62.66
50,00
46.66
52.94
68.18
51.06
36.73
58.62
Qwen-VL-Chal VL-Chat
18.75
16.66
23.76
19.67
15.48
9.58
15.48
17.12
17.39
15.02
15.72
MoF--LLAAA-Ph222277-444-32
11,40
9.57
12.87
8.19
9.76
12.32
11.72
10.77
9.72
10.99
8.73
mPLUG-OwI2-LLaMA2--11
13.24
9.57
11.48
10.96
10.04
11.60
12.28
13.41
7.86
Phi-3.5-Vision-Insstuut
34.00
42.00
40.51
40).47
40.00
38.23
40.11
52.00
35.35
35.35
35.00
Cambrian-1-8B
30.95
31.91
29.41
21.21
38.57
18.60
19.83
26.32
30.16
23.73
31.03
MinGGPT4--LaMA22778
14.09
17.45
15.00
17.45
18.79
26.21
20.81
16.11
20.81
11.41
16.78
Intern Vntem-VI-Chat-V1-5
18.00
20.20
13.80
21.70
60.00
33.30
22,20
15.10
57.10
18.90
15.70
Mini-IntemVLLCharr BaaaenV---Cha--B-VVVV
4B-V1-5
32.70
26.50
30.60
19.60
26.20
21.20
25.60
22.90
34.00
26.90
13.70
15.80
19.80
14.80
18.00
15.10
13.00
20.00
16.20
24.80
17.40
16,50
GPT4RoI
3.30
2.80
3.90
4.30
2.00
3.40
5,00
1.30
5.80
5.10
3,90
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
28.68
28.01
38.61
37.30
33.67
35.62
24.69
28.45
26.85
26.33
27.07
LLaVA-NeXT-34B
31.99
34.04
42.57
42.70
38.05
36.99
28.45
29.83
24.81
28.92
28.82
Pixtral 12B
33.46
35.46
34.65
40.00
35.02
29.45
22.59
25.69
25.58
24.07
24.02
SEED-LLaMA-13B
18.38
19.15
14.85
8.74
7.41
5.48
12.97
10.50
11.76
13.09
14.41
BLIP2
4.26
9.84
6.16
7.11
3.87
4.09
5.98
2.18
MiniMonkey
34,96
31.22
32.67
36.93
33.67
19.17
30.54
27.90
33.73
21.64
25.76
DeepSeck-VL VL-7B
31,99
25.89
33.66
28.96
30.64
22.60
25.10
30.11
36.06
21.00
26.20
LISA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Che
30.88
34.40
37.62
42.62
36.03
34.93
25.52
29.56
26.60
25.36
27.95
ShareGPT4V-7B
22.43
24.11
28.71
33.88
29.29
26.71
21.33
23.48
20.20
20.03
20.52
ShareGPTT4V-13B
25.37
25.53
32.67
37.16
31.99
29.45
22.59
25.41
21.99
21.65
22.71
BLIP-3 (XGen-MM)
28.68
29.79
33,66
39.34
32.66
32.19
24.27
28.18
24.81
23.10
25.33
AnyGPT
10.64
0.00
0.00
5.44
3.59
2.30
6.11
MiniCPM3-4B
34.19
36.88
41.58
47.54
36.61
35.62
27.20
30.11
29.16
21.81
23.14
LaVIT-V2 (7B)
16.18
18.79
12.87
12.02
9.29
0.00
11.28
10.50
11.00
12.12
15.28
GLM-VL-Chat
35.66
38.65
42.57
22.95
39.34
39.04
29.71
25.97
27.37
27.14
29.26
Gemini-1.5-Pro
39.71
31.21
42.57
34.97
34.01
33.56
33.89
30.39
33,50
28.27
29.26
1.5ini-1.5-Flash
35.29
26.24
35.64
26.23
29.29
28.77
26.78
31.49
29.16
27.14
25.76
OMG-LLaVA-IntcmL..
1.47
1.42
1.98
1.64
1.35
2.05
1.67
1.66
1.53
1.78
2.18
Idefics3-8B-LLamab
29.41
31.91
36.63
30.60
29.97
28.77
30.13
26.52
28.39
24.23
26.20
NEXT-GPT-VI.5
3.30
3,80
4.29
5.69
2.36
4.28
1.60
7.50
6.90
3.70
Vitron-V1
3.56
4.29
8.10
7.59
3.21
4.76
8.36
4.39
3.78
6.52
2.16
Otter
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
6.25
7.80
6.93
5.46
4.71
5.48
3.77
5.80
5.37
5.98
3,06
NExT-Chat
13.29
14.55
13.04
12.92
15.41
16.72
16.87
11.33
11.48
18.04
13.63
Yi-vision-v2
40.32
31.03
35.48
29.73
37.50
38.24
31.25
38.78
42.62
34.15
35.71
Owen2-VL-72B
29.86
37.06
18.37
54.63
30.36
43.27
33.74
55.22
54.27
42.94
36.17
61

---

On Path to Multimodal Generalist: General-Level and General-Bench
Table 22: Results on Image Comprehension #loup,#l-C-6, part B.
#I-C-6
Model
VQA)
#12
#13
#14
#15
#16
#17
#18
#19 1
#20 1
#21
SoTA Specialist
11.76
21.51
21.51
24.06
14.02
11.64
23.26
16.67
19.51
82.20
GPT4V
32.35
53.70
35.70
34.10
25,90
14.30
51.50
41.20
36.20
43.21
GPT4o
33.82
44.00
35.70
26.10
43.80
48.70
42.20
34.30
49.16
GPT4o-mini
29.42
40.40
42.00
30.90
33,30
25.00
39.60
40.20
31.60
40.08
GPT-4o-4096
38.24
48.11
44.37
45.94
42.58
44.60
53.49
46.67
47.76
95.90
CharGPT-4o-latest
30.88
47.03
40.40
40.94
16.13
40,30
51.62
42.38
35.26
48.28
Claude-3.5-Somec
31.41
48.16
39.76
32.75
27.96
27.67
46,05
40.66
33.43
43,90
Claude-3.5-Opus
28.32
44.09
37.23
30.11
24.71
24.46
46.05
39.79
30.15
42.83
Emu2-32B
19.82
24.25
8.96
21.35
16.85
19.97
24.98
23.45
21.22
14.03
DetGPT
20.46
26.42
5.82
21.88
16.75
20.36
26.60
24.99
22.42
12.14
Intern VL2.5-8B
38.78
27.03
40.85
23.50
26.54
45.26
41.02
38.20
38.41
34.27
25.32
34,40
37.95
20.95
25.62
49.59
38.43
35,43
31.93
Intern VL2.5-2B
32.44
23.02
34,67
40.45
23.62
24.74
47.28
39.11
31.78
33.33
Monkey-10B-chat
25.43
27.47
30.88
36.00
22.75
26.32
38.19
32.31
35.43
24.84
DeepSeck-VL-7B-Chhe
7.96
8.85
9.96
14.60
10.71
8.59
15.29
11.17
8.66
8.02
Owen2-V1L-77B
45.61
43.90
43.50
46.42
30.55
38.00
32.52
44.47
46.00
45.61
Owen-VL-Chal
18.37
15.23
17.50
16.17
13.54
14.35
18.60
14.28
16.32
15.06
11.76
11.89
8.61
12.81
9.03
10.03
11.63
6.19
7.76
42.80
mPPUUG-Ow22-LLMA22-77
11.35
5.88
10.65
11.42
13.95
11.02
38.94
Phi-3.5-Vision-Insstuut
47.90
44.23
43.00
41.30
38.25
33.70
55.11
50.00
47.36
49.00
Cambrian-1-8B
25.00
38.46
11.54
21.18
20.00
22.73
17.65
17.50
26.83
43.46
MiniGPT441.MM2-78
21.48
18.79
28.08
16.11
14.09
22.82
33.33
22.82
16.78
2.72
Intern Miern-V.-CCha-VI--
20.00
33,30
17.20
41.00
22.90
17.50
13.90
46,10
25,00
44.10
LindInntenV--Chhll
32.30
22.10
33.70
13.70
23.40
34.80
24.00
41.50
22.00
14,50
21.10
20.90
17,00
15.50
23.20
13.30
13.40
39.10
GPT4Rol
4.40
4,80
3.90
3.40
7.00
3.80
9.30
1.00
2.40
1.00
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXTIN
23.53
32.43
18.60
27.81
20.97
25.15
27.91
30.00
25.71
32.40
LLaVA-NeXT-34B
30.88
35.14
23.50
30.94
24.52
27.62
32.56
35.71
31.84
36.20
Pixtral 12B
20.59
28.65
12.40
25.94
23.55
19.29
20.93
32.38
28.57
30.80
SEED-LLaMA-13B
17.65
15.14
4.80
13.13
14.19
9.57
30.23
7.62
18.37
15.60
BLIP2
2.94
5.31
9.68
2.93
9.30
16.59
MiniMonkey
27.94
33.11
23.93
36.62
16.77
27.47
34.88
30.00
36.73
20.63
DeepSeek De--S
25.00
25.83
27.53
34.06
22.58
25.77
34.88
31.43
34.29
8.00
LISA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Chi Chat
25.00
34,59
21.19
27.81
22.90
27.16
37.21
30.48
27.35
31.50
ShureGPT4V-7B
17.65
27.57
13,91
22.19
18.39
22.07
23.26
24.76
20.82
24.94
ShareGPT4y-13B
20.59
29.19
17.22
23.75
19.68
24.23
27.91
26.67
23.27
26.19
BLIP-3 (XGen-MM)
26.47
31.35
19.20
25.31
21.61
25.31
32.56
29.05
26.12
28.47
AnyGPT
8.82
5.95
4.94
2.16
0.47
0.95
5.71
0.00
MiniCPM3-4B
29.41
37.84
11.26
30.31
24.84
16.05
32.56
34,29
32.65
35.64
LaVIT-V2 (7B)
16.18
20.55
9.27
11.25
12.58
9.10
20.93
6.67
17.55
14.77
VL-Chat
27.94
36.76
19.21
32.50
24.19
33.02
44.19
30.95
31.02
32.17
Gemini-1.5-Pro
32.35
42.16
35.76
31.25
29.03
25.31
39.60
38.57
34.29
40.60
Gemini-1.5-Flash
29.42
38.92
29.38
24.19
19.75
34.88
35.24
28.57
36.20
OMGG-LLaVA-ImemLLMM200
2.94
3.24
2.65
2,50
1.61
1.54
0.00
2.38
1.63
0.00
26.47
33.11
34.40
28.13
30.00
30.56
32.56
38.57
31.43
29.80
NEXT-GPT-V1.5
3.60
5.78
4.78
7.60
0.36
5.48
3.75
Vitron-VI
3.03
2.39
0.24
4.75
8.31
3.96
10.76
1.39
5.68
3.70
Otter
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
4.41
8.11
3.97
4.06
3.23
5.40
4.65
4.76
6.94
0.00
NExT-Chat
17.83
16.54
13.70
19.14
12.24
0.00
17.45
11.59
11.93
15.74
Yi-vision-v2
30.77
43.34
36.26
32.39
30.30
25.93
9.30
39.24
34.38
7.38
Qwen2-VL-72B VL-72B
44.61
25.64
56.50
48.99
40.27
57.94
37.07
25.15
28.00
50.43
62

---

On Path to Multimodal Generalist: General-Level and General-Bence
Table 23: Results on Image Comprehension Group, from #I-C-7 to #I-C-10.
#1-C-7
#I-C-8
#I-C-9
#I-C-10
Model
Den
(Grgh Cls)
(Hallacination Den
(ling Cap)
#1
#2
#3
#4 1
#1
#1
#2
#1
#2
#3
#4
#5
#6
SoTA Specialist
31.12
59.44
41.20
52.30
15.67
41.60
60.80
17.84
62.99
23.05
18.40
17.65
28.10
GPT4V
51.06
61.45
53.83
80.60
0.00
12.80
21.00
2.63
42.22
15.80
18.70
27.21
18.69
GPT4o
63.65
63.86
61.51
86.30
0.00
69.60
66.00
1.30
48.10
16.51
19.81
30.42
23.30
GPT4o-mini
61.38
56.62
64.57
84,30
0.00
40.80
78.20
4.00
43.10
18.64
15.05
25.62
18.60
GPT-4o-40%
65.63
64.66
66.39
88.70
0.00
68.20
70.80
1.00
45.45
17.36
20.68
28.86
15.72
CharGPT-4o-latest
55.02
62.25
59.41
81.11
0.00
50.00
64.20
1.74
41.23
15.82
15.85
21.03
13.53
Claude-3.5-Somee
58.31
59.89
59.92
83,26
0.00
40.40
54.25
1.96
44.25
16.80
17.63
26.82
19.57
3la-de-3.5-Opus
57.71
56.41
57.87
81.74
0.00
38.13
53.11
0.74
41.23
16.02
17.73
26.82
20.09
Emu2-32B
32.67
40.76
45.61
48.51
0.00
17.60
24.80
0.54
37.13
12.42
11.34
0.00
15.53
DetGPT
21.64
32,93
41.56
58.69
0.00
6.80
5.20
0.19
29.04
8.65
6.22
0.00
13.86
VL2.5-8B
45.54
47.14
72.41
4.29
9.06
16.99
20.42
13.79
7.49
7.02
Inter V1.2--4
47.10
61.45
45.75
100.00
0.00
13.13
32.62
8.24
16.43
20.53
13.37
14.59
4.61
LL2em2L
49.79
58.23
50.21
67.22
0.00
3.79
9.41
9.58
16.03
20.62
7.47
7.83
7.36
10B-chat
36.78
46.99
20.22
57.78
0.00
5.93
43.69
13.83
2.45
17.89
0.00
0.00
6.35
DeepSeek-VL-7B-CCIEE
47.67
62.65
48.50
78.52
0.00
3.48
3.18
3.10
12.75
17.77
17.84
19.03
3.64
Qwen2-V1.-7B
29.00
61.45
73.04
0.00
52.20
24.91
4.40
57.14
26.50
20.60
36.62
13.77
Qwen-VL-Chat
32.67
46.98
25.52
64.62
0.00
2.80
36.19
4.00
58.61
26.28
20.66
24.85
15.33
BEE-LILAAAA-Ph222.77-------
50.35
56.63
48.26
48.28
0.00
0.00
7.60
4.11
57.39
28.36
18.74
13.29
10.79
mPPLGG-OWI2-LLMA22-
42.29
40.17
0.00
52.69
26.30
20.00
11.91
10.32
Phi-3.5-Vision-Instuuet
19.31
60.64
46.72
78.70
0.00
0.00
0.00
4.63
54.51
26.84
24.50
20.20
15.40
Cambrian-1-8B
14.29
49.80
44.70
51.40
0.00
1.40
1.00
4.20
30.60
19.92
10.44
21.50
10.09
MiniGPTT4111M422-78
33.66
40.56
74.50
87.25
0.00
46.98
55.70
4.47
37.40
26.54
16.44
0.00
4.54
Inter V1-
49.30
55.80
49.20
76.20
0.00
38.20
46.40
4.30
20.40
14.20
7.60
4.50
7.90
Mini Wwwe-nnteWVL-Chhh-eee V1-5
41.30
65.70
25.80
40.60
4.60
19.90
13.50
3.70
2.70
1.90
InternI DeemLLM-XCoompssrr--Vi...
29.30
39.30
47.60
62.70
0.00
7.80
48.20
4.20
25.40
12.20
2.80
1.40
11.30
GPT4RoI
47.80
46.10
40.40
43.30
0.00
4.20
29.80
3.90
18.40
12.70
2.50
0.00
6.80
GLaMM
0.00
0.00
0.00
0.00
3.80
23.60
13.30
2.90
0.00
7.00
LLaVA-NeXT-131
36.21
48.59
34.73
62.41
0.00
15.20
18.40
1.18
32.48
22.05
14.32
0.00
18.12
LLaVA-NeXT-34B
45.69
55.42
47.00
71.48
0.00
18.80
25.60
1.04
36.12
23.45
15.96
0.00
17.49
Pixtral 12B
38.47
47.38
39.05
64.07
0.00
16.40
21.20
0.58
30.56
19.85
12.86
0.00
18.01
SEED-LLaMA-13B
26.73
35.74
34.03
43.89
0,00
5.80
8.60
1.12
24.12
9.84
7.56
0.00
11.92
BLIP2
43.37
51.11
18.20
20.40
6.74
15.24
17.26
11.30
8.30
MiniMonkey
12.16
21.69
11.16
58.63
0.00
4.60
29.80
0.31
29.98
18.91
0.00
0.00
4.42
DeepSeek VL-7B
26.59
58.78
44.21
73.26
0.00
5.80
6.20
2.02
18.74
19.04
12.60
0.00
4.06
LISA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Chan
44.55
48.59
45.19
69.63
0.00
17.40
23.60
2.56
38.74
22.04
16.47
0.00
23.15
ShareGPT4V-7B
30.13
39.36
30.26
56.85
0.00
14.80
19.20
1.48
33.92
24.46
11.59
0.00
16.72
ShareGPT4V-13B
31.26
45.78
31.24
60.37
0.00
15.60
21.60
1.85
37.30
25.08
13.95
0.00
18.38
BLIP-3 (XGen-MM)
42.57
42.17
43.79
63.33
0,00
16.20
22.40
2.17
29.77
24.45
14.36
0.00
21.79
AnyGPT
34.26
0.00
0.00
0.42
23.16
16.24
11.69
MiniCPM3-4B
48.37
52.21
53.56
64.26
0.00
18.40
22.40
1.24
39.13
20.04
14.91
28.93
22.67
LaVIT-V2 (7B)
37.06
44.18
34.59
46.11
0.00
4.40
14.20
0.91
29.28
12.08
10.08
0.00
16.45
GLM-VL-Chat
46.82
55.02
50.49
74.63
0.00
19.80
26.20
1.97
43.50
23.84
16.30
27.64
27.98
Gemini-1.5-PP
58.58
69.08
55.65
81.67
0.00
53.80
64.60
4.49
44.50
22.34
19.40
30.73
21.30
Gemini-1.5-FRp
52.86
76.67
52.00
61.00
4.51
39.72
23.58
18.95
28.58
17.80
OMM-LLLVA-Innee
MI-LLLLVA-IImemLLM220
6.36
13.65
3.49
3.89
0.00
2.20
1.60
3.24
22.37
18.28
11.37
0.00
7.50
Idefics3-8B-LLn
45.83
51.41
35.70
71.30
0.00
28.80
41.20
2.60
42.55
22.76
13.85
22.33
13.10
NEXT-GPT-V1.5
33.57
37.56
46.10
12.70
10.67
2.65
23.70
19.78
11.57
12.39
2.10
Vitron-VI
33.40
34.10
38.16
53.47
0.00
16,20
10.40
2.36
35.10
21.86
10.39
10.56
2.60
Otter
14.59
10.89
3.91
3.33
7.17
1.33
1.40
0.56
4.26
13.45
0.00
0.00
0.00
Show-o
10.34
12.61
17.15
17.22
0.00
0.00
0.00
0.00
38.07
11.50
0.00
0.00
0.00
NExT-Chat
25.43
70.63
0.00
27.39
0.00
0.00
0,00
3.49
56.19
12.28
15.13
0.00
0.00
Yi-vision-v2
21.07
53.41
16.60
35.48
0.00
66.93
52.79
4.24
22.80
15.02
12.47
0.00
13.49
Qwen2-VL-72BB VL-72B
28.22
61.88
26.47
78.57
0.00
54.31
32.04
4.79
57.37
27.02
16.48
29.30
16.98
63

---

On Path to Multimodal Generalist: General-Level and General-Bent
Table 24; Results on Image Comprehension Group, from #I-C-11 to #I-C-14.
#I-C-11
#I-C-12
#I-C-13
#I-C-14
Model
(lug Depth Est)
(Ing lest Seg)
(Iong OCR)
(Ing Recog)
#1 4
#1
#2
#1
#2
#3
#4
#1
#2 f
SoTA Specialist
36.40
66.50
63.80
45.03
95.14
10.23
24.70
29.80
88.00
GPT4V
0.00
0.00
89.71
35.68
60.45
18.30
32.50
94.53
GPT4o
0.00
0.00
94.07
97.36
74.01
19.46
27.83
95.24
GPT4o-mini
00
0.00
0.00
85.10
96.56
73.12
17.42
23.83
91.32
GPT-40-4096
0.00
0.00
94.22
96.67
74.45
0.00
34.00
32.20
ChatGPT-4o-latest
00
0.00
0.00
93.36
94.57
73.99
18.43
30.23
93.23
Claude-3.5-Someet
0.00
0.00
89.06
75.64
68.20
17.55
28.04
93.26
Claude-3.5-Opus
0.00
0.00
87.69
71.88
64.89
16.36
26.57
89.73
Emu2-32B
00
0.00
0.00
39.75
61.39
49.12
7.60
24.40
0.00
DetGPT
0.00
0.00
42.13
64.21
56,19
3.10
13.17
0.00
Intem VL2.5-8B
00
0.00
0.00
27.46
43.70
0.54
24.17
11.31
V1.emmV1.2.5444
0.00
0.00
24.31
41,53
29.27
0.00
14.83
12.62
VL2.522
0.00
0.00
27.39
31.06
43.47
0.00
12.83
9.34
Monkey-10B-cher
00
0.00
0.00
0.00
0.00
0.00
0.00
37.50
1.15
DeepSeck-VL-7B-Cha
0.00
0.00
22.98
45,90
14.11
0.27
33.33
4.38
Qwen2-V1--78
0.00
0.00
16.70
90.78
24.72
4.41
34.47
11.98
Ww-n-VL-Chhat
00
0.00
0.00
20.10
91.52
19.63
4.06
33.50
9.00
MVF-4LLAAVA--hh222277-----
0.00
0.00
22.28
93.10
20.15
0.40
33.50
6.24
mPLUG-OW22-LLMAA2-17
00
0.00
0.00
18.89
84.25
0.00
33.83
3.50
Phi-3.5-Vision-Innstuue
0.00
0.00
41.67
92.60
26.76
2.97
32.67
10.87
Cambrian-1-8B
0.00
0.00
16.77
84.19
20.80
0.00
14.00
3.81
MiniGPTA-JJ.MA2-78
00
0.00
0.00
22.50
83.35
16.54
0.31
39.50
4.37
Intern Intern VL.-Chat-V1-5
0.00
0.00
42.60
92.50
40.00
0.00
69.30
25.10
Mini Mini-IneemVL-Char-4PP
00
0.00
0.00
27.80
0.00
59.80
18.50
MicemL-M-CCompssr22-V1
00
0.00
0.00
22.10
12.90
31.70
0.00
31.00
15,50
GPT4RoI
0.00
0.00
0.00
0.00
0.00
0.00
22.60
14.10
GLaMM
00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-LE
0.00
0.00
62.38
72.34
62.57
6.40
27.33
7.30
LLaVA-NeXT-34B
0.00
0.00
69.89
78.05
70.08
8.20
28.67
14.40
Pixtral 12B
00
0.00
0.00
42.34
64.34
56.05
4.80
24.33
5.80
SHED-LLaMA-13B
0.00
0.00
22.03
32.57
47.08
0.00
16.00
0.00
BLIP2
0.00
0.00
23.81
44.38
46.92
3.60
38.00
0.00
MiniMonkey
00
0.00
0.00
0.00
0.00
0.00
0.00
38.00
2.84
DeepSeck-VLL VL-7B
0.00
0.00
61.82
47.09
32.15
0.35
37.50
5.67
LISA
00
42.09
92.75
0.00
0.00
0.00
0.00
CogVLM-Chi
0.00
0.00
72.69
86.53
70.89
8.35
27.00
38,20
ShareGPT4V-7B
0.00
0.00
54.21
69.44
56.47
6.72
21.50
27.40
ShareGPT4v-13B
0.00
0.00
59.24
72.31
60.36
7.55
22.83
32,80
BLIP-3 (XGen-MM)
0.00
0.00
68.57
82.47
63.52
7.24
25.67
35.60
AnyGPT
0.00
0.00
0.00
0.00
6.67
14.80
MiniCPM3-4B
0.00
0.00
71.96
77.08
71.85
9,80
29.50
73.60
LaVIT-V2 (7B)
0.00
0.00
43.16
57.98
47.36
0.00
15.17
32.40
GLM-VL-Chaa
00
0.00
0.00
74.28
61.42
64.27
8.40
25.17
69.00
Gemini-1.5-Pro
0.00
0.00
83.56
89.73
70.15
0.00
25.83
54.36
Gemini-L.5-Flash
0.00
0.00
73.48
82.81
64.58
0.00
23.67
42.17
OMG-LLaVA-IntemL...
00
39.50
68.22
18.91
14.63
8.92
0.00
2.00
8.95
Idefics3-8B-LLamaa3
0.00
0.00
43.02
62.25
59.32
0.00
20.50
16.57
NEXT-GPT-VI.5
00
0.00
0.00
22.56
$6.17
0.00
13.65
0.00
Vitron-V1
68.70
64.50
24.51
63.01
70.34
0.00
16.37
0.00
Otter
0.00
0.00
11.33
47.72
17.00
0.00
0.00
0.00
Show-o
00
0.00
0.00
0.00
0.00
0.00
0.00
25.17
0.00
NExT-Chat
0.00
0.00
76.75
71.32
61.04
0.00
17.41
0.00
Yi-vision-v2
0.00
0.00
83.57
90.28
55.23
0.00
35.67
16.83
Qwen2-VL-72B
0.00
0.00
26.58
95.27
60.79
0.00
45.16
13.72
64

---

On Path to Multimodal Generalist: General-Level and General-Bende
Table 25: Results on Image Comprehension Group, from #I-C-15 to #I-C-16.
#I-C-15
#I-C-16
Model
(long Sem Seg)
(ling
#1
#2
#3
#4
#5
#6
#7 f
#8
#1 1
#2
SoTA Specialist
19.80
50.40
88.00
98.70
83.65
59.36
47.74
62.15
84.76
90.91
GPT4V
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
67.69
74.11
GPT4o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
76.20
82.56
GPT4o-mini
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
57.18
59.44
GPT-46-4096
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
80.60
83.33
CharGPT-4o-latest
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
68.30
70.70
Claude-3.5-Somaa
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
66.50
71.66
Claude-3.5-Opus
0.00
0.00
0.00
0.00
0.00
0.00
0.00
62,40
70.73
Emu2-32B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DetGPT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
48.48
82.23
Intem VL2.5-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
V12.5-1
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
VL2.522
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Monkey-10B-aoe
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeck-VL-7B-CMl.
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
0.00
0.00
Qwen2-V1.-7B
0.00
0.00
0.00
0.00
0.00
0..00
0.00
0.00
0.00
Qwen- Owen-VL-Chaa VL-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
SEEELLLAVA--hhi--2.----
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
mPLUGG-WWD-LLLMA221
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Phi-3.5-Vision-Instmus
0.00
0.00
0.00
0.00
0.00
0.00
0..0
0.00
0.00
0.00
Cambrian-1-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MmmGPT74-1.aMA2-75
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ILeemV--Chat-V155
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Intern
NeomLM-XComppssrr--ll....
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4RoI
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GLaMM
42.10
33.90
83.60
0.00
0.00
57.40
64.20
0.00
72.70
LLaVA-NeXT-HE
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-34B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Pixtral 12B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
SEED-LLaMA-13B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
BLIP2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MiniMonkey
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeek VL-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LISA
9.35
90.28
44.45
22.45
88.74
88.65
39.46
0.00
0.00
0.00
CogVLM-Chir -hat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ShareGPT4V-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ShareGPT4V-13B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
BLIP-3 (XGen-MM)
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
AnyGPT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MiniCPM3-4B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LaVIT-V2 (7B)
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GLM-VL-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Gemini-1.5-Pro
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
Gemini-L.5-Flash
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
OMG-LLaVA, 1GG-LLaVAAImtemmLM22200
49.65
42.57
67.73
20.15
79.05
40.26
44.80
56.49
0.00
0.00
Idefics3-8B- 3iffes3-8B-Llama3
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NEXT-GPT-VI.5
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-V1
6.40
52.40
81.00
89,70
78.40
38.90
56.70
64.70
78.70
86.73
Otter
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NExT-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
Yi-vision-v2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Owen2-VL-722
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
65

---

On Path to Multimodal Generalist: General-Level and General-Bencls
Table 26: Results on Image Comprehension Group,#l-C-17, part A.
#I-C-117
Model
(Inn VQA)
#1 f
#2
#3 f
#4
#5
#6
#7
#8
f19 1
#10
So Specialist
14.36
36.40
17.32
43.12
89.38
17.50
34.30
80.75
92.72
78.30
GPT4V
16.20
26.54
52.84
41.04
80.38
45.74
37.00
57.33
87,54
43.33
GPT4o
16.00
23.82
50.31
31.41
85.75
37.00
53.00
96.10
51.00
GPT4o-mini
11.45
17.45
47.24
29.09
82.01
44.03
34.00
33.33
93.10
37.67
GPT-44-4096
24.40
22.73
53.78
43.16
89.02
74.15
40.00
52.00
96.63
48.67
CharGPT-4o-latest
16.82
19.64
49.20
51.64
83.88
69.32
36.98
63.33
84.81
41.67
Claude-3.5-Somne
13,98
21.73
49,57
33.44
81.97
52.38
35.48
47.78
91.38
43.34
Claude-3.5-Opus
13.40
20.90
47.42
28.99
77.92
48.74
34.46
43.76
90.51
42.52
Emu2-32B
10.18
34.55
33.25
38.54
65.42
35.51
31.67
43.64
65.32
33.67
DetGPT
9.36
29.09
21.19
28.90
55.84
28.41
25.00
34.55
63.97
28.33
Intem VB
3.45
47.27
31.54
39.31
55.74
60.00
70.67
83.64
98.65
43.67
10.64
37.82
34.16
40.66
27.02
46.57
38.33
86.36
97.31
44.33
IntemVL2.5-21
29.09
40.55
38.97
39.69
36.48
49.14
84.00
88.18
93.10
38.00
Monkey- 10B-chht
6.91
3.60
45.64
8.11
14.92
66.29
55.11
85.67
72.05
26.00
DeepSeck-VL-7B-CCK
3.45
100.00
0.22
0.00
33.78
6.57
66.67
53.10
58,94
24.44
Owen2-V1.-7B
54.91
65.45
54.32
37.34
26.87
0.00
34,00
61.82
54.04
29.30
Qwen- VL-c-a
$7.99
52,90
38.96
33.26
67.99
5.71
33.33
54.54
71.54
18.66
DOB-LLLAVA-hhi2222B--44-4
40.00
54,00
33.36
31.44
67.29
7.80
33.67
60.28
66.16
14.67
mPLUUG-OW1-LLMMA2-777
38.00
46.73
15.67
33.80
53.27
8.12
39.00
37.27
58.58
13.33
Phi-3.5-Vision-Instmeet
40.18
60.00
41,60
35.90
78.04
31.14
42,33
81.82
94.95
32.67
Cambrian-1-8B
5.82
1.45
29.00
28.70
32.94
0.00
2.00
28.18
26.26
0.33
MiniGPT44LJ.aMA2-77
34.23
38.26
46.83
30.06
55.03
32.43
28. 19
32.11
68.46
38.93
IntemVV--Chat-v1-5
11.20
23.90
6.40
47.90
62.90
22.40
34.60
58.30
68.40
26.60
Mini VL-emmt---hhht--
9.80
16.70
4.50
44.30
$9.20
16.32
11.33
51.70
62.90
20.00
Internl DeemLLM-XCommpssrr--Vi---
6.70
15.60
3.20
40.50
36.80
10.60
33.30
47.90
55.30
23.90
GPT4RoI
4.00
7.40
0.00
52.80
36.00
5.60
23.30
67.80
46.80
68.00
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-13B
14.55
33.30
36.18
40.46
80.14 14
41.76
30.33
34.55
83,83
37.33
LLaVA-NeXT-34B
15.09
36.80
40.25
48.17
78.84
52.27
33.67
40.00
87.88
43.67
Pixtral 12B
12.00
23,50
33.85
42.78
71.96
44.03
31.00
32.73
74.75
36.00
SEED-LLaMA-133
10.55
26.70
28.96
31.21
62.85
37.78
28.67
28.18
65.99
32.33
BLIP2
9.45
46.00
27.64
38.84
25
29.26
29.00
22.73
59.26
29.33
MiniMonkey
6.01
25.45
39.31
37.80
66.59
45,92
33.33
84.54
93.94
36.00
DeepSeek-VLL VL-7B
12.78
31.25
32.98
35.63
58.18
57.10
41.33
51.82
56.06
23.67
LISA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Chin
11.45
37.09
36.43
45.28
81.31
52.84
35,33
47.27
81.82
44.67
ShareGPT4V-7B
7.82
25.82
27.95
39.11
71.03
42.33
31.33
34.55
75.93
35.67
ShareGPT4V-13B
9.09
30,36
30.44
43.35
76.40
46.88
33.67
41.82
79.97
38.67
BLIP-3 (XGen-MM)
9.45
35.64
28.57
42.58
73.36
49.72
32.00
44.55
77.95
41.00
AnyGPT
0.00
17.26
29.11
40.19
12.73
35.52
3.67
MiniCPM3-4B
12.18
31.45
49.66
50.47
79.91
49.15
31.67
46.37
89.23
40.33
LaVIT-V2 (7B)
9.45
20.36
29.97
42.91
$1.87
37.50
15.00
28.19
64.98
32.67
GLM-hL-
12.73
37.64
41.20
51.98
84.11
57.67
34.30
50.91
86.03
44,33
Gemini-1.5-8
14.18
18.73
46.72
52.41
86.68
59.94
36.67
83.64
94.44
48.33
Gemini-1.5-Fi
13.27
17.27
43.80
42.77
85.98
54.83
34.00
82.73
90.91
44.67
OMG-LLa MaL_LaaVA-nttmLMM0000
0.00
2.55
22.16
2.12
4.91
1.42
1.67
2.73
2.19
2.67
Ideficx3-8B LB-mmaa3
12.91
16.73
36.46
34.30
67.99
44.03
28.67
57.27
82.15
38.33
NEXT-GPT-V1.5
3.40
28.90
31.54
30.50
47.60
34.96
13.40
24.63
63.45
29.40
Vitron-VI
5.60
31.50
34.78
27.30
$1.40
31.84
20.70
18.90
67.82
24.70
Otter
0.00
0.00
7.34
0.00
14.10
17.29
9.67
0.00
0.00
0.00
Show-o
14.72
39,24
33.47
0.00
0.00
0.85
34.33
1.82
0.51
0.33
NExT-Chat
7.41
34.83
41.67
0.00
38.86
11.63
43.29
41.67
40.37
37.93
Yi-vision-v2
14.00
41.64
53.94
0.00
83.41
45.63
37.00
83.64
96.13
35.67
Qwen2-VL-72B VL-72B
69.09
51.79
55.10
0.00
0.00
50.58
47.33
59.00
73.82
49.42
66

---

On Path to Multimodal Generalist: General-Level and General-Benel
Table 27: Results on Image Comprehension Cr-up, part B.
#I-C--17
Model
(Ing VQA)
#11
#12
#13 1
#14
#15
#16
#17
#18
#19
#20
SoTA Specialist
49.80
65.40
66.50
74.70
73.58
24.40
71.60
70.70
51.60
71.40
GPT4V
0.00
50.00
75.70
81.82
76.39
34.25
58.70
66.00
47.33
70.40
GPT4o
0.00
79.44
72.72
78.17
45.00
66.30
12.70
72.20
GPT4o-mini
0.00
56.90
71.96
77.27
69.27
28.00
56.52
69.67
12.60
63.20
GPT-4o-4096
0.00
64.66
83.18
81.82
79.29
45.25
54.35
68.33
50.33
75.60
CharGPT-4o-latest
0.00
62.07
79.44
77.27
75.72
45.00
57.61
70.33
47.00
75.20
Claude-3.5-Somec
0.00
57.81
75.57
76.59
74.35
35.54
60.07
67.93
24.13
50.60
Claude-3.5-Opus
0.00
54.04
72.74
72.86
71.10
32.88
59.55
64.81
19.68
53.73
Emu2-32B
53.67
47.41
65.42
54.55
66.82
27.50
55.43
54.00
29.67
32.80
DetGPT
34.33
37.07
56.07
31.82
54.34
25.50
43.48
47.33
27.33
34.40
Intern VL2.5-8B
72.00
13.79
69.16
72.73
87.31
41.50
36.96
43.66
50.67
53.85
47.67
42.24
75.70
77.27
95.09
27.37
71.20
67.72
41,22
86.04
Intern VL2.5-2B
99.33
16.09
60.75
77.27
83.96
62.50
41.30
45.77
0.00
87.04
Monkey-10B-chat
50.00
43.10
57.94
54.55
54.34
54.67
57.61
49.30
49.33
81.60
DeepSeck-VL-7B-Chee
50.00
47.90
41.55
52.27
56.94
25.00
51.83
38.25
99.33
83,43
Owen2-V1L-77B
54.28
64.35
37.38
22.73
30.73
34.00
84.78
21.75
54.32
72.00
Owen-VL-Chal
52.33
42.24
57.94
22.72
56.79
18.25
58.69
70.87
40.33
39.60
MoF:41LAAVA-Ph2-2227-------333
51.67
35.34
50.47
31.82
43.88
25.00
59.78
38.84
44.17
31,20
mPPLUG-Ow22-LLMM22-77
7.66
33.62
31.78
9,09
32.52
27.21
64.13
33.63
38.90
16.60
Phi-3.5-Vision-Insstuut
50.00
55.17
70.09
31.81
81,29
24.00
63.0M
53.33
0.00
45.80
Cambrian-1-8B
49.70
34.48
38.32
13.64
24.72
25.00
50.00
42.30
0.00
39.41
MiniGPT441LMM2-77
79.87
43.48
37.74
33.33
57.05
0.00
48.35
46.53
93.96
43.06
Intern Mieen-V.-Chaa-VI--
54.30
59.40
61.60
68.10
79.40
23,00
76.00
84,20
35.60
54.20
LindInntenVL-Chall
42.00
40.51
22.70
74.50
21.20
57.60
36.84
15.33
43.60
Internl DermLL--XCComppsr2--1--.1.
51.00
33.60
34.50
54.50
78.30
9.30
58.60
76.30
9.30
52.40
GPT4RoI
47.60
47.40
50.40
68.10
44.50
31.00
28.20
64.40
62.30
0.00
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXTIN
54.33
52.59
67.29
63.64
62.36
28.75
51.09
$4.67
9.33
45.00
LLaVA-NeXT-34B
48.67
65.52
73.83
72.73
70.82
32.50
61.96
62.33
12.67
50.60
Pixtral 12B
58.67
45.69
63.55
36.37
67.48
30.25
57.61
59.67
8.33
39.40
SEED-LLaMA-13B
30.33
41.38
42.06
45.45
57.46
18.75
28.26
$2.33
6.67
27.40
BLIP2
33.67
40.91
$2.78
19.00
27.17
$2.67
47.20
MiniMonkcy
50.00
56.03
61.68
59.09
77.28
29.75
75.00
76.32
65.00
69.20
DeepSeek De--Seck-VL-7B
48.33
46.55
38.32
50.00
56.79
24.50
52.17
37.33
57.82
75.60
LISA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Chi Chat
55.67
57.76
66.36
77.27
70.60
33.50
55.43
62.00
22.33
52.20
ShareGPT4V-7B
47.67
42.24
52.34
59.09
$7.24
28.50
46.74
53.67
15.33
44.80
ShareGPT4y-13B
45.67
49.14
57.94
72.72
61.47
32.00
51.09
55.67
17.33
46.20
BLIP-3 (XGen-MM)
46.00
54.31
58.88
63.64
67.04
31.75
57.61
61.33
19.00
50.60
AnyGPT
36.33
12.07
27.39
6.00
14.13
21.33
20.40
MiniCPM3-4B
52.33
56.03
57.94
63.64
73.72
33.75
66.30
64.00
9.67
54.00
LaVIT-V2 (7B)
36.60
35.34
36.45
40.91
$7.91
19,00
26.09
49.00
7.67
26.80
GL-Chtt
55,33
66.38
71.96
77.27
66.59
30.50
59.78
59.67
9.00
56.20
Gemini-1.5-Pro
60.33
65.52
76.64
77.27
70.16
40.00
61.96
67.00
41.00
70.40
Gemini-1.5-Flash
58.33
77.27
68.82
33.75
56.52
63.33
33.67
66.80
OMGG-LLaVA-ImemLLMM200
1.67
3.45
4.67
9,09
2.45
2.00
3.26
2.33
2.67
3.20
57.00
46.55
70.09
63.64
71.27
36.00
64.13
63.33
32.00
61.20
NEXT-GPT-V1.5
38.50
28.40
34.10
36.80
64.27
20.80
31.57
40.37
0.00
Vitron-VI
41.60
32.70
40.70
34.60
63.45
26.50
34.86
45.21
5.45
0.00
Otter
0.00
6.78
25.10
29.20
33.10
11.50
18.21
37.50
0.00
0.00
Show-o
50.00
1.72
1.87
0.00
2.00
0.25
28.26
62.99
50.00
1.20
NExT-Chat
39.81
44.79
35.29
50.00
38.46
21.69
59.78
43.64
11.48
37.45
Yi-vision-v2
31.33
68.10
71.96
22.73
89.53
28.75
35.87
61.54
45.33
50.23
Qwen2-VL-72B VL-72B
40.43
L.72
52.55
60.95
58.22
18.42
68.93
69.58
51.69
70.37
67

---

On Path to Multimodal Generalist: General-Level and General-Bencl
Table 28: Results on Image Comprehension (roup.#I-C-11, part C.
#I-C-117
Model
(Inn VQA)
#21
#22
#23
#24
#25
#26
#27 f
#28
#29
#30
So Specialist
70.60
59.30
61.90
61.40
62.80
62.10
70.20
83.80
82.40
68.60
GPT4V
68.60
22.81
20.40
24.10
22.64
22.02
18.01
93,00
91.40
71.20
GPT4o
74.00
37.30
40.30
43.45
35.47
39.87
29.60
93.80
94.30
65.00
GPT4o-mini
65.60
26.34
23.70
29.07
26,52
25.01
25,20
90.50
89.50
68.50
GPT-44-4096
77.00
27.95
49.51
52.05
43.15
48.03
43.31
94.40
95.60
69.60
CharGPT-4o-latest
76.00
28.61
40.01
42.66
35.43
37.96
36.81
89.40
92.20
70.60
Claude-3.5-Sonne
59.95
39.10
45,29
48.88
44.27
54.41
59.85
76.11
82.49
71.90
Claude-3.5-Opus
58.08
42.24
45.67
47.91
42.24
53.51
62.86
76.00
82.54
73.25
Emu2-32B
44.80
24.20
30.80
24.60
22.40
36.20
44.60
71.40
72.60
60.40
DetGPT
36.80
21.40
16.90
20.20
19.50
17.40
17.90
50.60
54.80
54.40
Intem VB
100.00
49.75
40.45
58.79
55.26
50.00
57.43
97.10
77.4 77.46
59.09
84.82
33.70
35.99
54.69
53.03
33.92
57.38
79.13
79.84
64.82
IntemVL2.5-21
83.54
27.33
22.83
31.13
60.27
50.00
63.84
94.11
68.42
68.90
Monkey- 10B-chht
80.35
39.10
42.29
65.94
57.99
42,91
47.34
85.53
80.15
44.84
DeepSeck-VL-7B-CCK
84.58
23,27
18.72
30.41
26.98
19.80
54.66
60.99
61.76
47.51
Owen2-V1.-7B
73.60
75.37
71,96
82.41
78.84
69.40
77.60
92.10
94.74
79.20
Qwen- VL-c-a
37.60
50.65
43.55
58.28
52.91
46.69
52.60
72.80
71.00
53,40
-OB-LLLAVA-hhi22227B--4--
30.60
62.43
58.87
72,02
67.55
57.35
32.63
42.80
43.00
41,80
mPLUUG-OWI-LLLMA2-777
14.39
44.41
43.98
65.46
63.62
45.78
23.60
28.99
Phi-3.5-Vision-Inntmeet
44.20
64.36
62.71
70.32
72.85
60.24
73.80
84.52
85.27
57.20
Cambrian-1-8B
28.73
49.50
50.61
58.83
$2.37
43.35
48.00
0.63
44.40
40.00
MiniGPT44LJ.aMA2--7
30.08
44.68
54.65
60.10
59.00
50.26
51.28
27.80
46.72
47.36
Intem-V--Chat-V1-5
52.40
30.30
26.80
33.30
36.90
28.20
73.40
87.70
83.30
70.60
Mini VL-eemtL--hhha--e
43.00
29.80
26.20
40.10
37.60
62.00
80.20
65.60
63.00
Internl DeemLM--XComppssrr--Vi---
52.80
46.75
48,90
69.80
65.40
51.00
52.20
54.20
52,60
74.00
GPT4RoI
0.00
20.80
23.80
31.70
23.90
23.20
37.20
38.20
34,80
28.20
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-13B
46.20
35.60
34,60
46.50
41.20
46.60
58.40
68.
72.40
60.40
LLaVA-NeXT-34B
52.40
37.80
41.20
46.20
37.80
48.80
61.20
74.60
78.60
65.40
Pixtral 12B
53.40
33.20
38.20
47.90
32,40
51.40
48.60
64.20
74,20
66.20
SEED-LLaMA-133
42.40
21.90
18.40
20.90
20.80
19.20
18.40
44.20
52.60
59.40
BLIP2
47.80
49.20
48.80
47.40
49.60
50.20
43,80
46.00
42.80
29.60
MiniMonkey
71.60
36.17
39.67
53.55
53.34
39.57
41.64
63.60
71.60
35,80
DeepSeek-VLL VL-7B
78.60
21.81
17.29
30.06
22.76
17.44
53.60
61.80
60.80
43,00
LISA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Chin
56.60
35.30
40.75
47.60
43.57
49.72
58.40
65.20
69.40
64,80
ShareGPT4V-7B
45.20
27.53
32.93
37.34
35.26
40.95
49.80
57.80
61.00
55.40
ShareGPT4V-13B
48.20
29.23
35.92
41.29
38.29
42.94
52.00
60.40
63.60
57.80
BLIP-3 (XGen-MM)
52.00
32.42
38.32
45.82
41.87
46.80
54.60
62.80
65.20
62.40
AnyGPT
27.60
19.90
21.70
26.40
28
8.60
26.00
MiniCPM3-4B
$7,80
36.70
38,90
50.70
48.70
48.60
63.20
62.80
67.60
62.80
LaVIT-V2 (7B)
43.20
29.40
24.50
33.70
36.70
31.90
35,20
42.00
46.80
48.40
GLM-hL-
55,20
38.80
44.70
51.30
49.60
52.40
59.60
69.80
69.80
67.40
Gemini-1.5-B
64.80
35,90
34.50
41.80
39.80
36.50
75.20
91.20
90.00
63.60
Gemini-1.5-Fi
66.00
30.80
28.60
38.70
41.40
33.70
68.40
89.60
86.80
58.40
OMG-LLaVA, Min-LaVVA-ntemLMM000
2.60
21.10
21.30
22.20
17.50
12.40
3,20
5,40
4.20
2.80
Ideficx3-8B LB-mmaa3
59.80
29.40
36.70
38.40
36.50
32.80
62.60
84.40
85.20
62.20
NEXT-GPT-V1.5
0.00
23.58
22.89
21.60
20.65
15.48
0.00
0.00
0.00
Vitron-VI
0.00
27.41
23.05
31.60
21.78
20.78
16.98
0.00
0.00
0.00
Otter
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
0.40
0.44
0.38
0.42
25.00
28,80
1.40
45.29
0.90
0.80
NExT-Chat
12.50
20.67
27.53
25.31
23.40
21.35
40.97
12.88
11.64
26.41
Yi-vision-v2
48.59
8.54
9.81
13.96
16.32
9.28
60.27
34.27
44.59
67.83
Qwen2-VL-P VL-72B
74.51
56.19
68.02
86.06
81.24
71.33
78.17
90.91
80.95
83,57
68

---

On Path to Multimodal Generalist: General-Level and Cennral-Bench
Table 29: Results on Image Comprehension Group.#l-C-17, part D.
#I-C-117
Model
(Inn VQA)
#31
#32
#33 1
#34
#35
#36
#37
#38
#39
#40
SoTA Specialist
75.00
67.20
68.40
66.80
48.80
56.60
75.80
31.00
20.71
29.20
GPT4V
77.10
70.40
73.90
82.80
89,40
98.20
14.80
19.60
16.83
18.30
GPT4o
74.20
71.80
72.10
83.70
92.40
91.80
27.20
30.20
18.04
19.20
GPT4o-mini
70.50
35,80
39.00
80.60
77.60
63.40
10.60
7.80
15.23
18.70
GPT-4o-4096
75.00
76.60
77.40
83.00
91.00
87.00
25.80
47.51
15.83
0.00
CharGPT-4o-latest
73.60
74.30
76.60
82.40
92.00
63.00
18.20
34.60
18.24
19.72
Claude-3.5-Somec
74.44
67.53
71.63
67.13
86,20
84.12
16.89
18.24
16.43
18.62
Claude-3.5-Opuu
74.34
68.98
70.97
66.84
82.41
84.20
16.78
18.21
16.40
15.56
Emu2-32B
66.80
59.00
67.20
62.60
71.40
74.60
22.78
14.60
15.43
14.85
DetGPT
56.40
52,00
38.20
44.60
54.60
60.66
18.64
10.20
8.22
13.83
Intern VL2.5-8B
36.11
52.05
50.74
33.71
15.54
18.54
12.40
3.17
69.37
40.74
40.60
35.92
49.00
30.41
49.21
6.45
1.80
3.39
Intern VL2.5-2B
68.93
49.99
46.86
64.26
3.18
1.74
8.25
5.74
3.55
4.59
Monkey-10B-chat
47.56
57.20
59.60
33.94
50.40
51.40
50.20
13.37
2.00
2.32
DeepSeck-VL-7BBChhe
50.05
51.09
56,04
28.06
3.50
6.50
11.57
7.93
1.93
4.11
Dwwn2-V1L-77B
81.80
13.33
41.91
93.33
91.06
86.67
45.80
48.60
45.36
20.89
Owen-VL-Chal
54.60
3.20
22.00
61.00
47.00
50.40
45.80
15,40
17.89
5.07
41.40
6.00
13.60
59.10
51,20
50.80
50.00
37.00
20.10
12.57
mPPLUG-Ow22-LLMA22-77
25.60
30.20
54.40
51.60
44.40
9.11
Phi-3.5-Vision-nnstuuut
61.44
18.40
40.67
44.90
48.97
62,46
49.03
20.00
32,06
18.80
Cambrian-1-8B
64.17
9.70
11.00
49.30
43.98
59.47
43.70
29.20
4.30
16.52
MiniGPT4-1LMMA-778
55.68
9.80
18.70
43.70
0.67
48.32
91.28
58.39
18.92
4.13
Intern Miern-V.-CCha-VI1-
69.20
55.00
55.20
71.00
74.00
76.40
60.20
27.60
0.00
3.00
Mini LindInntenV--Chhll
65.20
44.00
61.20
61.40
65.00
50.80
2.40
Internl.M-X LermLL--XCComppsr2--1--11.
73.40
64.60
62.60
72.20
66.60
49.00
48.80
20.00
1.60
5.80
GPT4Rol
31.60
10.60
8.20
28.00
53.60
50.40
50.00
11.80
0.00
14.60
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXTIN
64.20
58.60
59.60
52.40
62.40
64,40
30.20
12.20
12.02
16.42
LLaVA-NeXT-34B
70.60
62.80
65.60
60.40
70.60
72.20
42.40
16.40
15.63
17.93
Pixtral 12B
65
60.40
57.60
61.20
68.20
68.20
18.60
17.20
13.63
15.88
SEED-LLaMA-13B
50.20
44.40
35.80
40.80
49,20
58.80
11.40
11.60
8.62
11.73
BLIP2
31.20
11.40
40.40
52.60
61.80
59.40
2.10
MiniMonkcy
37.80
46.80
57.40
49.60
48.20
53.60
61.80
28.33
0.00
3.41
DeepSeek De--Seck-VL-7B
50.80
47.60
47.20
32.00
51.60
49,60
13.58
12.40
1.60
3.73
LISA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Cha
68.40
63.60
64.40
59.40
69.20
67.80
38.40
16,20
12.83
22.14
ShareGPT4V-7B
62.80
54.20
53,20
52.20
63.20
59.60
31.20
13.40
9.82
18.29
ShareGPT4V-13B
66.20
58.40
56.80
55.80
65,80
62.60
33.40
14.60
11.22
20.56
BLIP-3 (XGen-MM)
64.20
61.20
62.00
57.60
67.40
64.40
35.60
15.80
13.83
19.47
AnyGPT
42.20
14.60
24.80
0.00
34.00
16.40
5.81
0.00
MiniCPM3-4B
65,00
66,80
62.60
61.20
64.20
72.60
42.80
16.80
12.42
19.74
LaVIT-V2 (7B)
$4.80
42.60
46.80
40.60
50.80
$7.20
34.60
11.80
8.22
13.07
GL-Chtt
70.80
65.20
63.40
63.80
68.60
78.40
44.00
17.40
15.83
26.34
Gemini-1.5-Pro
65.20
62.00
67.80
74.40
82.60
89.60
74.00
27.20
17.23
18.10
Gemini-1.5-Flash
61.00
53.60
72.00
77.40
84,00
20.80
14.63
16.30
OMGGLLLaVA-ImemLLMM200
3.40
2.00
2.60
3,20
7.80
8.40
7.20
0.00
0.00
10.10
56.40
54.20
57.60
63.00
74.40
76.20
63.40
17.60
11.22
9.70
NEXT-GPT-V1.5
0.00
20.60
28.60
26.50
36.90
51.40
12.60
2.70
14.60
Vitron-VI
0.00
11.40
0.00
39.40
27.80
59.90
60.70
9.40
1.80
13.47
Otter
0.00
0.00
0.00
0.00
49.89
0.00
0.00
0.00
0.00
0.00
Show-o
0.80
0.90
7.20
0.00
5,40
5.80
0.00
0.00
0.00
0.00
NExT-Chat
27.38
18.52
17.44
36.80
47.44
51.32
58.42
49.85
47.33
11.71
Yi-vision-v2
67.36
7.22
17.04
41.31
46.00
65.80
48.60
19.70
2.35
Qwen2-VL-72B VL-72B
88.93
53.39
57.18
87.76
96.32
91.25
56.41
15.06
48.21
20.49
69

---

On Path to Multimodal Generalist: General-Level and General-Bence
Table 30: Results on Image Comprehension floup, part A.
#I-C-18
Model
(Ind-Anoeall Det)
#1
#2
#3
#44
#5 f
#6
#7
#8
#9
#10
SoTA Specialist
90.30
60.40
56.30
72.80
61.50
88.40
75.60
93.50
83.10
58.70
GPT4V
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4o-mini
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT-4o-4096
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CharGPT-4o-latest
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Claude-3.5-Somnc
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
Claude-3.5-Opus
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Emu2-32B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DetGPT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Intern VL2.5-8B
0.00
0.00
0.00
0.00
0.00
0.00
VntermV1.2.544B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Intern VL2.5-2B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Monkey-10B-chal
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeck-VL-77B-Che
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
Dwen2-V1L-7BB
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen-VL-Chal
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MoF:41LAAAA-hh2-2277------3
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
mPPUUG-OW22-LLMMA2277
0.00
0.00
0.00
0.00
0.00
0.00
Phi-3.5-Vision-nnstruut
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Cambrian-1-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MiniGPT441LMM2-78
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Intern Mtern-V.-Chha-V1-
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
Moni-IntemmLL-hha--ee
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4Rol
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-NA
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-34B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Pixtral 12B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
SHED-LLaMA-13B
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
BLIP2
0.00
0.00
0.00
0.00
MiniMonkey
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeek De--Seck-VL-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LISA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Chi Chat
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
ShareGPT4V-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ShareGPT4y-13B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
BLIP-3 (XGen-MM)
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
AnyGPT
0.00
0.00
0.00
0.00
MiniCPM3-4B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LaVIT-V2 (7B)
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
VL-Cha
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Gemini-1.5-Pro
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
Lemini-L.5-Flash
0.00
0.00
0.00
0.00
0.00
0.00
0.00
OMG-LLLVVAIImmmLLM220
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NEXT-GPT-V1.5
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-VI
60.50
26.40
13.50
36.50
25.30
10.70
24.00
33.50
15.70
13.40
Otter
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NExT-Chat
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
Yi-vision-v2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2-VL-72B VL-72B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
70

---

On Path to Multimodal Generalist: General-Level and General-Bende
Table 31: Results on Image Comprehension #I--p,,----1,,, part B.
#I-C-18
Model
(Ind-Anoeally Det)
#11
#12
#13
#14
#15
#16
#17
#18 f
#19
SoTA Specialist
90.00
75.20
64.80
81.90
85.20
62.10
76.20
23.10
73.60
GPT4V
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
GPT4o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4o-mini
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT-40-4096
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CharGPT-4o-latest
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Claude-3.5-Sunnct
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
Claude-3.5-Opus
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Emu2-32B
0.00
0.00
0.00
0.00
0.00
35.13
45,12
20.02
0.00
DetGPT
0.00
0.00
0.00
0.00
0.00
26.38
35.36
18.24
0.00
IntemVL2.558B
0.00
0.00
0.00
37.00
25.00
62.50
InternV1.2.5448
0.00
0.00
0.00
0.00
0.00
35.38
50.00
40.44
16.25
Lntem2VL2.5-2B
0.00
0.00
0.00
0.00
0.00
34.13
20.71
41.43
45.00
Monkey-1OB-chat
0.00
0.00
0.00
0.00
0.00
34.50
71.43
40.03
0.00
DeepSeck-VL-7B-Chr.
0.00
0.00
0.00
0.00
0.00
36.75
28.57
45.02
41.43
Qwen2-VIL-7B
0.00
0.00
0.00
0.00
41,12
60.71
68.33
35.00
Qwen-VL-CHie
0.00
0.00
0.00
0.00
0.00
26.75
71.42
58.16
5.00
MofF-1L-AVA-hh2-2227-------
0.00
0.00
0.00
0.00
0.00
25.00
71.07
42.63
25.00
mPLUG--OWD-LLaMA2--7
0.00
0.00
0.00
28.49
56.79
24.30
68.75
Phi-3.5-Vision-Innsrret
0.00
0.00
0.00
0.00
0.00
34.75
73.93
60.96
56.25
Cambrian-1-8B
0.00
0.00
0.00
0.00
0.00
22.75
60.71
10.96
37.50
MiniGPTA-LJaMA2-778
0.00
0.00
0.00
0.00
0.00
15.63
70.00
51.00
57.50
InternVl-Chaa-v1-5
0.00
0.00
0.00
0.00
0.00
65.10
79.60
76.40
0.00
Mini-IntemVLLChatee
0.00
0.00
0.00
39.80
65.70
Internl Incem_M-XCCompser--l-1115
0.00
0.00
0.00
0.00
0.00
36.50
71.40
56.90
0.00
GPT4RoI
0.00
0.00
0.00
0.00
0.00
70.50
70.30
41.60
0.00
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
41,00
39.29
39.84
0.00
LLaVA-NeXT-34B
0.00
0.00
0.00
0.00
0.00
38.89
49.29
44.22
0.00
Pixtral 12B
0.00
0.00
0.00
0.00
0.00
34.63
45.36
46.22
0.00
SEED-LLaMA-13B
0.00
0.00
0.00
0.00
0.00
26.13
37.14
27.09
0.00
BLIP2
0.00
0.00
0.00
17.23
47.14
33.27
MiniMonkey
0.00
0.00
0.00
0.00
0.00
26.41
7.86
23.71
0.00
DeepSeek-VL.3
0.00
0.00
0.00
0.00
0.00
31.57
69.31
47.26
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM- Chat
0.00
0.00
0.00
0.00
0.00
43.25
55.71
48.41
0.00
ShareGPT4V-7B
0.00
0.00
0.00
0.00
0.00
31.13
43.93
39.44
0.00
ShareGPTT4V-13B
0.00
0.00
0.00
0.00
0.00
33.25
48.57
41.24
0.00
BLIP-3 (XGen-MM)
0.00
0.00
0.00
0.00
0.00
36.88
53.93
41.04
0.00
AsyGPT
0.00
0.00
17.75
37.50
MiniCPMB-4B
0.00
0.00
0.00
0.00
0.00
36.75
54.64
37.25
0.00
LaVIT-V2 (7B)
0.00
0.00
0.00
0.00
0.00
25.38
43.57
25.30
0.00
GLM-VLLChat
0.00
0.00
0.00
0.00
0.00
40.25
57.50
45.42
0.00
Gemini-1.55Pro
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
G.5ini-1.5-Flash
0.00
0.00
0.00
0.00
0.00
OMG-LLaVA-IntemLL2220
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Idefics3-8B-LJamaa
0.00
0.00
0.00
0.00
0.00
38.75
51.79
40.04
0.00
NEXT-GPT-VI.5
0.00
0.00
23.67
22.30
Vitron-VI
14.80
7,80
3.40
23.70
19.60
24.89
42.69
26.10
0.00
Otter
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NExT-Chat
0.00
0.00
0.00
0.00
0.00
22.72
32.56
39.25
3.85
Yi-vision-v2
0.00
0.00
0.00
0.00
0.00
26.63
53.93
53,98
0.00
Qwen2-VL- VL-72B
0.00
0.00
0.00
0.00
0.00
40.77
71.43
49.46
45.16

---

On Path to Multimodal Generalist: General-Level and General-Benu
Table 32: Results on Image Comprehension Group, from #I-C-19 to #I-C-20.
#I-C-19
#I-C-20
Model
(Mod Seg?
(Keypoint Det)
#1
#2
#3
#4
#5
#6
#1 f
SoTA Specialist
50.10
16.30
25.85
23.55
53.40
37.88
95.70
GPT4V
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4o-mini
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT-40-4096
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CharGPT-4o-latet
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Claude-3.5Soonee
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Claude-3.5-Opus
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Emu2-32B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DetGPT
0.00
0.00
0.00
0.00
0.00
35.26
0.00
VL2.55L
0.00
0.00
0.00
0.00
0.00
0.00
0.00
IntemV1.2.5-4B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
VL2.522
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Moncey-10B-chat
0.00
0.00
0.00
0.00
0.00
0..00
0.00
DeepSeck-VL-7B-Che.
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2-VL-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Owen-VL-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
BBBFLLL-AAA-Ph2222288-------
0.00
0.00
0.00
0.00
0.00
0.00
0.00
mPLUGG-OW2-LLMMA2-11
0.00
0.00
0.00
0.00
0.00
0.00
Phi-3.5-Vision-Innstiet
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Cambrian-1-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MiniGPTA-LJaMA2--78
0.00
0.00
0.00
0.00
0.00
0.000
0.00
Inter
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Min--neem/VL-Cha---B-Vi.
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4RoI
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GLaMM
0.00
17.70
0.00
0.00
2.10
41.50
0.00
LLaVA-NeXT-13B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-34B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Pixtral 12B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
SHED-LLaMA-13B
0.00
0.00
0.00
0.00
0.00
0.00
0,00
BLIP2
0.00
0.00
0.00
0.00
0.00
0.00
MiniMonkey
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeck-VL.
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LISA
0.00
0.00
0.00
0.00
0.00
30.22
0.00
Cog VLLMChai
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ShareGPT4V-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ShareGPT4V-13B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
BLIP-3 (XGen-MM)
0.00
0.00
0.00
0.00
0.00
0.00
0,00
AnyGPT
0.00
0.00
0.00
0.00
0.00
0.00
MiniCPM3-4B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LaVIT-V2 (7B)
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GLM-VL-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Gemini-1.5-Pro
0.00
0.00
0.00
0.00
0.00
0.00
0.00
G.
0.00
0.00
0.00
0.00
0.00
0.00
0.00
OMG-LLaVA-ImteLLMMP
1.68
14.53
19.96
17.75
1.02
0.00
0.00
Ldefics3-8B-LLama3
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NEXT-GPT-VI.5
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-V1
2,00
10.70
35.60
10.40
12.30
16.78
0.00
Otter
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NExT-Chut
0.00
0.00
0.00
0.00
0.00
0.00
0,00
Yi-vision-v2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2-VL-72B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
72

---

On Path to Multimodal Generalist: General-Level and General-Bence
Table 33: Results on Image Comprehension Group, from #I-C-21 to #I-C-22.
#I-C-21
#I-C-22
Model
(Multi-ing VQA)
(MM Diolog)
#1
#2
#3
#4 f
#5
#6 1
#1
#2
#3
#4
#5
#6
SoTA Specialist
88.20
87.60
75.00
49.60
59.80
59.80
51.90
51.60
53,40
53.10
51.60
40.80
GPT4V
46.20
47.20
82.40
94.00
84,80
76.80
23.45
55.06
28.00
52.38
37.05
26.80
GPT4o
61.60
59.80
95.60
98.20
85.00
87.60
21.84
62.09
28.13
53.13
38.09
34.40
GPT4o-mini
26.80
28.40
96.60
97.80
80.80
66.20
22.79
47.74
24.94
42.82
32.13
11.60
GPT-40-4096
51,40
50.00
96.00
98.80
91.00
78.20
20.88
59.21
30.22
50.20
40.12
2.40
CharGPT-4o-latest
25.60
27.00
95.80
97.60
76.20
79.20
22.11
54.44
28.35
49.62
38.09
17.25
Claude-3.5-Somme
44.28
45.04
90.56
96.56
83.38
75.94
30.71
49.91
35.13
45,96
39.53
38.68
Claude-3.5-Opus
41,96
40.65
89.46
94.03
81.52
74.70
33.72
49.87
35.39
44.50
42.15
41.51
Emu2-32B
48.40
$3.60
62.60
66.80
52.40
54.20
28.13
35.60
27.87
27.40
23.40
34.20
DetGPT
39.00
31.00
42.60
29.20
28.40
38.40
22.96
25.60
19.67
19.70
20.70
22.40
Intem VL2.588
50.00
49.43
21.26
18.94
17.21
17.52
30.35
27.96
35.52
29.30
29.54
69.13
68.27
49.40
50.84
47.48
50.00
15.29
19.40
20.64
17.25
17.52
27.47
IntemVL2.5522
46.79
43.02
7.18
7.30
2.62
53,00
17.31
28.54
28.46
18.99
20.63
28.09
Monkey-10B-chat
43.80
39.73
49.40 40
47.60
50.80
50.00
10.09
11.47
12.55
17.38
10.93
37.07
DeepSeek-VL-7B-Chan
59.73
56.20
8.92
7.86
8.58
4.02
14.36
17.99
33.17
19.04
28.30
27.59
Qwen2-VI.-7B
78.83
77.52
87.60
96.39
61.80
83.60
20.50
47.01
33.01
53,50
42.38
31.20
Qwen- -w-n-VL-Chat
35.60
33.60
64.80
73.00
64.80
45,80
18.26
30.91
25.80
39.69
27.22
21.09
10B-:JJAVA-Phi2-22B----4-
55.60
53.20
49.60
49.40
50.40
48.60
17.05
21.81
23.38
23.71
22.26
17.32
mPLUGG-WWD-LLaMA2-77
55.40
52.40
53.00
51.80
47.20
49.80
18.38
30,96
25.06
31.02
27.30
8.90
Phi-3.5-Vision-Instie
67.26
58.51
70.34
88.00
62,80
58.45
21,42
31.49
29.88
42.70
39.67
28.73
Cambrian-1-8B
48.44
50.31
59.12
64.07
45.46
23.97
8.53
26.70
26.29
18.90
9.37
22.80
MiniGPT44-1.aMA22-75
95.97
85.91
12.08
8.05
45.64
1.34
18.96
18.93
26.05
23.41
18.97
17.49
Intern InemmV--Chat-Vi-5
63.40
84.20
53.00
65.20
55.60
45.60
11.60
17.20
17.20
19.10
16.80
42.60
Mini Lee-nneee-L-hhaa-B V1-5
80.20
82.40
52.40
59.60
51.60
42.80
11.90
16.80
17.20
33.20
MeemLMM-XComposer22-W
75.20
68.80
50.00
49.20
46.90
38.50
20.50
23.40
22.00
26.10
24.10
34.70
GPT4RoI
54.60
53.60
8.20
25.80
50.00
35.20
15.10
28.00
27.40
39.90
24.50
0.00
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-BE
54.60
34.20
52.60
53.40
41.40
57.60
22.16
38.40
23.04
37.40
37.40
29,80
LLaVA-NeXT-34B
62.40
51.20
66.40
65.60
58.20
69.80
27.24
42.40
25.18
40.20
36.40
32.20
Pixtral 12B
52,40
46.40
56.20
57.80
50.20
36.40
24.14
43.80
27.24
41.80
34.20
31.00
SEED-LLaMA-13B
41.80
28.40
44.40
28.60
25.60
62.40
17.25
34.40
16.85
24.20
20.80
21.60
BLIP2
57.20
58.40
44.40
53.00
$8.80
35.40
36.00
32.80
33.60
35.00
39.60
MiniMonkey
69.00
43.40
59.40
63.20
33.60
67.20
15.63
13.52
16.89
15.26
10.19
33.20
DeepSeek VL-7B
56.40
58.80
47.20
39.80
57.60
61.40
11.86
14.69
29.66
18.97
17.21
23.40
LISA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Chal
53,40
47.20
64.60
62.40
56,80
67.20
21.03
41.90
23.47
41.60
39.36
30.40
ShareGPT4V-7B
45.80
41.60
56.20
56.20
49.80
61.60
16.22
34.19
18.39
34.27
35.84
22.00
ShareGPT4V-13B
48 60
43.20
58.40
57.40
52.60
62.40
17.97
36.58
19.45
36.89
38.50
24.60
BLIP-3 (XGen-MM)
51.20
45.80
62.20
60.80
54.20
65.80
19.58
38.67
21.48
39.23
40.56
26.80
AnyGPT
0.00
0.00
27.60
12.40
14.20
11.34
18.40
17.10
17.40
MiniCPM3-4B
49,40
34.60
78.00
64.00
56.60
57.40
20.62
45.90
24.07
47.30
44.60
29.60
LaVIT-V2 (7B)
43.20
0.00
45.80
26.80
21.40
27.40
15.53
31.70
15.31
25.80
21.50
19.40
GLM-VL-Chat
54,20
50.40
67.40
56.80
42.80
64.60
22.47
42.40
27.39
46.40
43.10
31.40
Gemini-1.5-Pro
88.80
86.20
82.20
84.60
82.60
83,00
25.60
33.40
30.30
32.60
28.40
39.00
G.5-nih
84.20
81.40
77.60
83.00
78.20
79.40
30.30
27.40
34.00
25.70
33.60
AMG-LLaVA-Intermn...
6.20
8.80
7.60
8.40
8.00
7.00
16.60
15.90
14.60
19.10
13.90
1,80
(lefies3-BB-Llamee
68.60
61.40
71.20
59.80
64,80
63.40
22.50
26.10
24.60
32.80
22.70
36.20
NEXT-GPT-VI.5
$7.60
57.30
47.90
45.80
51.60
16.40
18.60
28.40
28.70
31.76
22.36
36.81
Vitron-VI
59,20
60.30
48.60
47.40
55.80
28.70
18.70
29.60
25.60
35.29
18.65
41.00
Otter
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
50.60
49.80
48.20
43.40
46.80
49.60
0.20
0.10
0.10
0.10
0.10
0.00
NExT-Chat
33.79
28.95
52,40
64.29
61.55
13.79
16.32
17.81
19.15
20.38
20.05
21.37
Vi-vision-v2
50.40
49.80
61.80
69.40
66.40
66.80
12.12
18.30
25.74
20.58
25.82
7.75
Swen2-VL-722
80.41
80.15
89.60
98.43
75.21
67.34
24.95
48.33
33.34
50.70
44.23
30.01
73

---

On Path to Multimodal Generalist: General-Level and General-Bencl
Table 34: Results on Image Comprehension Group, #1-C-23.
#I-C-23
Model
(MM Resson)
#1
#2
#3
#14
#5
#16
#7
#8
#9
#10 f
#11
#12
#13
#14
SoTA Specialist
68.80
69.80
77.20
50.00
49.80
64.60
80.20
70.60
80.60
47.20
66.20
67.60
74.20
56.80
GPT4V
0,00
0.00
72.40
22.80
17.80
91.00
91.20
79.00
81.80
23.80
56.00
56.60
80.90
30.90
GPT4o
0.00
0.00
62.20
10.40
8.80
97.40
97.40
75.80
77.40
37.00
50.40
51.40
81.40
31.20
GPT4o-mini
0.00
0.00
76.40
18.80
19.20
92.40
92.60
71.60
66.80
17.80
56.40
53.20
73.20
34.10
GPT-40-4096
0.00
40.60
75.00
7.80
6.60
96.00
96.60
77.80
79.20
31.60
51.00
52.00
82.20
39.00
CharGPT-4o-latest
0.00
0.00
69.72
13.00
12.80
93.28
93.80
76.20
73.00
25.60
53.00
54.00
82.00
37.80
Claude-3.5-Sunce
53,91
53.14
65.54
17.24
25.84
40.84
81.56
74.82
78.57
28.13
65.19
67.70
73.57
36.80
Claude-3.5-Opus
55.67
53.32
68.41
18.09
40.69
84.83
72.19
80.09
27.42
67.96
64.57
71.38
37.88
Emu2-32B
40.20
36.40
54.60
10.40
18.80
29.60
76.40
66.40
71.20
21.40
58.40
59.60
62.40
30.60
DetGPT
36.40
35.80
33.20
3.20
11.60
5.60
62.40
49.60
46.80
14.20
48.80
40.20
40.00
20.60
Intem
33.33
45.06
78.66
34.83
34.18
62.71
58.45
$7.98
47.62
37.98
37.84
42.41
39.46
VntemmV1.2.544B
44.61
44.28
65.00
33.06
31.97
28.81
46.20
52.54
51.95
31.81
43.42
44.95
45.29
28.93
Intern VL2.5-2B
11.63
45.45
60.20
33.50
33.68
32.72
70.83
65.98
43.25
34.02
36.14
29.62
46.42
40.53
Monkey-1OB-chat
45.40
32.20
45.60
32.94
33.04
32.31
45.00
55.60
36.42
33.79
52.99
52.60
41.76
37.00
DeepSeck-VL-7B-COK
46.52
68.60
56.20
39.84
39.64
41.70
55.40
31.82
39.97
37.03
38.72
39.45
42.71
56.60
Qwen2-VIL-7B
52.00
51.40
81.41
43.40
44.00
43.40
57.71
72.40
42.20
53.20
59.35
59.49
68.20
13.33
Qwen-VL-Chat VL-Chat
37.60
40.00
62.20
49.20
49.80
31.20
45.80
45.50
14.20
52.80
20.00
20.70
58.19
40.20
MMB-LLLAVA-Ph22-2278-------
4.80
4.40
55.91
54.00
52.60
45.60
1.00
22.60
9.40
34.40
22.00
20.00
47.40
3.80
mPPLGG-WWD-LLaMA2-T1
53.00
62.40
55.71
28.00
27.80
28.20
56.30
29.20
39.40
41.20
$7.20
Phi-3.5-Visson-IInstiet
59.72
66.40
67.40
51.06
48.00
40.76
54.20
37.40
38.50
54.00
55.20
48.77
60.05
39.65
Cambrian-1-8B
36.54
49.62
56.20
44.36
39.81
29.74
46.00
39.10
19.82
46.60
40.27
19.20
37.60
25.78
Min(GPTA-1JaMA2--78
39.83
44.34
48.90
39.46
35.90
33.26
39.83
38.69
20.00
48.96
41.28
22.17
27.61
28.00
IntemVL-Chaat-v--5
46.40
47.60
61.80
68.40
75.60
68.20
56.40
61.60
61.00
74.80
53.00
53.20
68.20
43.00
Mini-meemVV--Chat4BBBWW
44.20
46.40
57.60
53.40
69.80
63.90
61.20
33.40
55.20
47.20
46.80
69.20
46.00
Intem_M-XCooppsee22-11
48.20
45.40
60.60
41,90
56.90
53.70
45.40
53.20
48.60
73.40
54.00
53.00
54.80
37.00
GPT4Rol
39.20
22.60
38.60
0.00
0.00
0.00
27.20
50.20
19.60
0.00
47.00
47.40
28.40
36.40
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-13B
49.20
47.00
54.60
7.20
14.40
28.80
68.40
63.40
68,20
16.60
54.20
49.20
60.60
26.80
LLaVA-NeXT-34B
51.00
53.40
60.20
10.80
17.40
37.60
75.60
68.40
70.40
20.20
56.80
56.20
65.20
28.60
Pixtral 12B
42.80
43.20
56.00
8.60
15.60
35.40
74.80
67.80
74.20
17.80
53.40
51.20
66.40
30.20
SEED-LLaMA-13B
43.20
39.80
46.80
2.80
4.20
8.40
64.80
37.80
42.40
10.20
48.40
44.60
46.60
17.40
BLIP2
43.00
44.40
53.40
2.00
0.20
58.80
30.20
46.20
43.40
37.80
MiniMonkey
42.40
29.80
42.88
29.80
18.80
29.40
40.40
52.40
31,20
28.00
48.60
44.20
36.40
37.00
DeepSeek- DeepSeek-VL-7B
42.20
63.80
51.80
33.60
37.60
37.60
55.20
29.60
34.80
36.80
38.80
40.60
41.80
53.40
LISA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Chal
50.40
54.20
62.80
10.40
15.80
31.60
77.40
66.40
76.00
14.20
59.40
57.80
65.80
29.40
ShareGPT4V-7B
42.00
46.60
54.40
8.20
13.60
26.80
69.40
60.60
64.20
11.40
$1.20
48.80
61.60
24.80
ShareGPT4V-133
44.20
48.40
56.80
9.60
15.20
28.20
71.80
62.40
68.60
12.80
54.80
51.40
58.20
26.40
BLIP-3 (XGen-MM)
48.60
52.80
60.20
8.40
14.00
29.40
74.60
64.80
72.20
13.60
$7.20
54.40
63.20
27.20
AnyGPT
26.80
31.20
24.80
0.00
2.80
26.40
10.60
12.40
21.40
8.60
MiniCPM3-4B
52.20
53.80
66.40
10.60
18.60
27.60
76.20
69.40
74.60
11.60
62.80
60.80
61.40
29.40
LaVIT-V2 (7B)
43.00
38.80
49.60
3.40
3.60
7.60
66.60
34.00
40.60
9.60
48.40
51.00
48.20
14.20
OLM-VL-Cha
50.60
57.40
64.80
12.20
14.60
32.40
79.80
70.20
79.80
15.40
61.00
65.20
66.80
28.80
Gemini-1.5-Pro
46.20
40.60
77.40
37.80
37.20
91.60
93.20
76.40
74.80
40.60
$6.20
55.40
72.60
52.20
Gemini-1.5-Fias
44.80
38.20
69.20
33.00
33.60
89.20
90.40
71.80
73.00
37.80
48.40
52.20
68.40 40
46.80
OMG-LLaVA-Intemme
2.60
2.20
3.80
1.80
2.20
2.80
4.00
2.20
3.00
2,20
2.60
2.40
2.40
2.60
Idefics3-8B-LLamae
43.80
41.20
58.40
23.20
28.80
45.60
78.80
64.20
72.20
38.40
43.20
48.80
48.20
43.20
NEXT-GPT-VI.5
19.60
18:40
18.60
10.70
24.78
28.96
10.36
0.00
37.98
13.40
3.40
10.30
34.60
Vitron-VI
38.41
20.57
21.50
9.45
22.16
31.20
14.89
0.00
28.64
52.38
17.50
4.80
14.60
35.80
Otter
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
42.80
45.60
25.00
7.20
0.00
9.00
45.00
33.20
15.20
0.00
39.80
41.00
11.60
37.00
NExT-Chat
37.08
61.76
57.80
13.71
13.36
25.66
33,93
45.37
25.61
35.98
15.73
29.35
23.68
44,30
Yi-vision-v2
45.09
37.80
70.03
26.40
14.40
36.62
61.23
49.79
21.90
$8.08
57.06
35.16
39.49
Qwen2-Ve VL-B2B
53.06
55,56
90.86
46.67
37.13
49.32
52.23
81.24
54.42
57.17
64.46
52.34
79.93
51.51
74

---

On Path to Multimodal Generalist: General-Level and General-
Table 35: Results on Image Comprehension Group, from #I-C-24 to #I-C-25.
#I-C-24
#I-C-25
Model
(Object Count)
(NMT)
#1
#2
#3
#5
#6
#7
#B
#9
#1
SoTA Specialist
5.95
9.00
26.84
8.13
16.41
17.91
28.16
10.00
27.00
78.00
GPT4V
8.41
3.20
31,29
9.79
26.91
10.72
10.86
32.21
11.14
72.20
GPT4o
2.30
31.17
11.85
10.76
5.29
23.01
15.70
93.00
GPT4o-mini
10.06
2.80
30.98
13.62
43.87
12.87
10.56
26.30
14.66
83.00
GPT-4o-4096
7.47
23.66
31.11
14.65
21.75
9.93
4.69
16.26
16.00
88,40
CharGPT--4o-laest
10.41
2.60
29.23
9.66
21.44
10.41
5.86
14.04
22.35
76.52
Claude-3.5-Someet
9.21
1.89
30.73
11.01
31.93
11.08
8.24
26.30
13,42
82.23
Claude-3.5-Opus
8.50
0.24
28.21
9.29
27.59
7.29
8.40
23.33
11.99
77.80
Emu2-32B
10.57
1.85
33.37
26.23
39.15
14.24
17.58
37.24
12.83
64.20
DetGPT
11.08
1.54
31.34
31.65
42.41
13.58
16.28
42.12
12.94
38.40
Intern VL2.5-8B
9.11
0.37
35.40
10.84
27.17
9.81
21.69
23.68
14.48
7.82
Intern V1.2541
12.58
0.59
32.88
16.09
39.62
15.38
27.88
24.67
14,41
33.27
Intern VL2.5-2B
14.11
1.07
37.27
37.15
69.25
20.03
24.69
28.67
21.02
20.00
Monkey-10B-eee 10B-chat
8.35
1.04
37.09
63.28
73.40
13.91
32.71
24.52
14.68
52.20
DeepSeek-VL-T85 -L-TB-hhat
25.84
1.63
36.38
69.02
100.00
24.81
34.30
24.52
14.58
5.34
Qwen2-V1.-7B
12.49
0.27
30.30
46.49
12.50
27.79
16.00
12.57
97.20
Qwen-VL-Chat VL-Chat
7.41
1.40
25.89
36.60
67.19
7.42
21.29
16.45
11.09
94.80
MoF-4JJAAVA-Phi--2777--44
26.49
1.05
33.91
69.43
102.00
19.28
30.47
21.66
14.33
80.80
mPLUGG-OWD2LLaMMA227
14.28
2.30
35.22
55.12
69.30
13.37
20.19
21.57
12.35
86.80
Phi-3.5-Vision-Instruet
19.31
0.80
26.80
47.26
52.80
10.12
22.80
22.40
11.03
90.10
Cambrian-1-SB
14.08
2.73
39.15
50.61
95.19
9.85
26.40
22.06
10.11
71.40
MiniGPT4LL1MAA--78
9.60
2.10
41.20
65.30
4,40
20.03
28.49
20.58
10.92
76.32
Intern BieeVVi-Chat-V11
14.30
0.60
34,60
41.60
69.60
10.70
16.90
23.90
17.30
85,20
Mini- InternVL-Chha-teetaa V1-5
18.20
1.94
36.80
40.10
68.30
14.10
22.10
25.20
17.25
80.40
InternI Maemml--XCoompsser--Vi...
13.30
2.59
39.00
54.80
62.30
12,40
24.70
26.50
16.70
79.30
GPT4RoI
3.29
0.00
GLaMM
00
00
20
bo
oc
00
00
00
0.00
LLaVA-NeXT-13B
15.38
3.81
34.40
17.94
43.35
13.38
22.44
34.08
15.24
56,80
LLaVA-NeXT-34B
13.86
3.60
32.42
18.02
38.71
14.46
20.06
30.04
14.53
68.40
Pixtral 12B
12.07
4.52
34.65
18.52
45.04
12.98
17.02
36. 14
16.29
60.40
SEED-LLaMA-13B
14.50
5.40
38.60
28.06
47.52
15.02
16.42
38.64
20.49
32.20
BLIP2
19.20
2.67
37.50
33.22
37.21
21.10
39.20
28.70
20.60
23.60
MiniMonkey
13.98
3.78
38.72
53.10
74.57
15,80
23.64
26.64
23.62
24.20
DeepSeek -L-7e VL-7B
27.29
3.06
38.61
62.59
68.83
22.63
33.49
21.59
18.97
5.80
LISA
00
00
20
OC
00
OC
00
00
00
0.00
CogVLM-Char
11.61
3.02
33.17
15.68
35.92
11.63
18.34
32.87
19.35
65.60
ShareGPT4V-7B
12.47
4.15
38.94
25.19
42.41
13.19
22.68
26.38
21.94
56.20
ShareGPT4V-13B
12.62
2.84
38.20
26.77
40.75
12.22
21.54
28.49
23.58
59.60
BLIP-3 (XGen-MM)
14.47
3.54
34.17
13.45
38.34
14.21
23.58
34.36
17.62
63.20
AnyGPT
13.92
7.43
40.82
62.53
14.25
30.94
36.07
26.84
14.60
MiniCPM3-4B
10.36
3.24
35.25
14.45
44.38
10.30
19.64
28.07
15.03
73.40
LaVIT-V2 (7B)
10.98
4.62
35.93
24.16
60.17
10.41
15.64
34.88
18.21
45.00
GLM-VL-Chat
9.09
4.21
33.44
17.28
34,36
9.67
9.99
27.88
11.14
68.80
Gemini-1.5-Pro
7.74
0.32
32.19
10.78
28.85
9.90
8.28
26.73
12.03
86.40
Gemini-1.5-Flast
11.23
1.78
33.03
12.36
33.17
10.75
10.05
25.00
11.80
82.60
OMG-LLaVA-ImmmmLLMDOO
22.85
2.89
38.65
$7.65
71.08
21.00
38.74
39.44
28.58
3.40
Idefics3-8B-LLA
12.51
3.71
35.69
22.25
41.75
12.25
13.68
27.50
16.84
63.40
NEXT-GPT-V1.5
11.58
36.77
31.25
67.40
13.59
16.47
30.58
23.31
0.00
Vitron-VI
12.34
3.69
35.91
27.59
56,70
11.50
15.46
28.15
20.36
0.00
Otter
22.04
1.12
38.34
60.17
91.58
27.75
36.11
27.67
9.33
0.00
Show-o
71.14
89.42
30
OC
00
72.13
00
73.30
0.00
NExT-Chat
29.56
10.03
36,84
69.78
89.27
21.55
14.38
28.68
25.24
43.52
Yi-vision-v2
13.63
0.44
29.83
24.51
44.48
14,09
20.14
13.23
15.67
8.77
Qwen2-VL-77
9.64
0.30
27.13
9.81
42.55
9.47
21.86
10.40
14.34
97.43
75

---

On Path to Multimodal Generalist: General-Level and General-Bench
Table 36: Results on Image Comprehension Group,#1-C-26, part A.
#I-C-26
Model
(Chjject Den
#1 f
#22
#3
#14
#5
#6
#7
#8 f
#9
#10
#11
#12
#13
SoTA Specialist
87.20
87.80
32.60
41.10
73.80
29.00
34.00
62.40
93.20
26.20
37.60
18.10
61.60
GPT4V
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4o-mini
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT-40-4096
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CharGPT-4o-latest
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Claude-3.5-Sonna
0,00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Claude-3.5-Opus
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Emu2-32B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DetGPT
0.00
85.99
30.69
39.93
67.60
25.20
26.38
56.40
88.80
3.00
33.40
9.00
55.60
Inter VL2.5-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
IntemW112.5-4B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Intern VL2.5-2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
10nkey-10B-chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeek-VL-7B-Chl.
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
Qwen2-V1.-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen- WWen-V
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
10B-IJJAVA-Phii22278-------
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
mPLUG-OWW2-LLMM22-7
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Phi-a.--Viioon-nnsstret
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0..00
0.00
0.00
0.00
0.00
0.00
Cambrian-1-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MiniGPT4411.MA22-788
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Intern IntemVL-Chat-Vi-5
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Mini Mini-InermWLL-hatt-4e
0.00
0.00
0.00
0.00
0.00
0.00
0.00
InternL.M-X MacmLM--CComonsrr--111183
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4RoI
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GLaMM
37.10
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-11
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-34B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Pixtral 12B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
SEED-LLaMA-133
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
BLIP2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MiniMonkey
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeek-VL.. VL-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LISA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Che
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ShareGPT4V-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ShareGPT4V-13B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
BLIP-3 (XGen-MM)
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
AnyGPT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MiniCPM3-4B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LaVIT-V2 (7B)
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GLM-VL-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Gemini-1.5-Pru
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
G.5
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NEXT-GPT-VI.5
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-VI
12.34
37,84
50,36
43,50
82.10
16.40
38,00
56.80
75.80
32.40
31.70
46.70
60.34
Otter
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NExT-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Yi-vision-v2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2-VL-72B VL-72B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
76

---

On Path to Multimodal Generalist: General-Level and General-Bench
Table 37: Results on Image Comprehension Group,#I-C-20 part B.
#I-C-26
Model
(Chject Den
#14
#15
#16 1
#17
#18
#19
#20
#21
#22 f
#23
#24
#25
#26
SoTA Specialist
27.20
68.30
81.10
86.40
18.20
17.20
31.60
78.90
28.30
48.10
3.50
39.40
22.40
GPT4V
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4o-mini
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT-40-4096
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CharGPT-4oolatest
0.00
0.00
0.00
0.00
0.00
0.00
000000000
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Claude-3.5-Somee
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
Claude-3.5-Opus
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Emu2-32B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DetGPT
21.40
66.20
80.60
83.60
10.40
9.40
21.80
69.80
19.00
42.80
2.60
0.00
0.40
Intern VL2.5-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Intern V1.2.5-BB
0.00
0.00
0.00
0.00
0.00
0.00
00000000
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Intern VL2.5-2B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Monkey-10B-chan
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.000
0.00
0.00
0.000
DeepSeck-VL-7B-COE
0,00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
Qwen2-V1.-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen-VL- VL-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MOF-4LLAAVA-hhi222278---66
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0..00
mPLUG-OOWI-LLLMA2227
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Phi-3.5-Vision-nnstrret
0.00
0.00
0.00
0.00
0.00
0.00
0.000
0.00
00000
0..00
0.00
0.00
0..00
Cambrian-1-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MiniGPT4-L1LMA--78
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0..00
ViernnL--Chat-Vi1
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Piii-InteeVVLLLL Www.-nneemn---Cha---------
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
InternI.M-Xee WMM.---Xooompssr2-1111.1
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4RoI
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
-LaVA-NeXT-I33
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-34B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Pixtral 12B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
SHED-LLaMA-13B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
BLIP2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MiniMonkey
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeek Dee-Seek-VL-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LISA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Chi
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ShureGPT4V-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ShareGPT4V-13B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
BLIP-3 (XGen-MM)
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
AnyGPT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MiniCPM3-4B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LaVIT-V2 (7B)
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GL-Cht-
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Gemini-1.5-Pry
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Gemini-1.5-Flash
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DMG-LLaVAAIInemLLMMMN
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Idefics3-883 8B--laa3
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NEXT-GPT-VI.5
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-VI
21.47
56.70
64.20
87.36
23.40
36.50
42.70
53.70
34,60
56.10
10.40
3.60
1.80
Otter
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NExT-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Yi-vision-v2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2-VL-72 VL-72B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
00000
0.00
77

---

On Path to Multimodal Generalist: General-Level and General-Bench
Table 38: Results on Image Comprehension Group.#l-C-26, part C.
#I-C-26
Model
(Object Det)
#27
#28
#29
#30
#31
#32
#33 f
#34
#35
#36
#37
SoTA Specialist
67.60
63.40
10.90
48.60
38.00
97.10
61.10
97.50
25.90
58.20
64,20
GPT4V
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
0.00
GPT4o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
4PT4o-mini
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT-40-4096
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CharGPT-4o-latet
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Claude-3.5-Sonnet
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
0,00
0,00
0.00
Claude-3.5-Opus
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Emu2-32B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DetGPT
26.80
58.98
9.39
47.68
30.55
93.93
52.23
92.80
22.83
57.04
59.71
VL2.55B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
IntemV1.22.54B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
VL2.52L2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Monkey-10B-chh
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeck-VL-7B-CTK
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
0,00
0,00
0.00
Qwen2-V1L-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen-VL- WL-Ca-
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MAB--LL-AVA-Phi222.78------
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
mPLUG-OWw2-LLMMA2-70
0.00
0.00
0.00
0.00
0.00
0.00
Phi-3.5-Vision-Innsriit
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Cambrian-1-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MiniGPTA-LlaMA2-778
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Intern
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Mini- Jini-ntemMLL-Cha--BB-VLL
0.00
0.00
0.00
0.00
IntemnLM-X Mecmll--XCComosser-----....
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4RoI
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-13B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-34B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Pixtral 12B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
SEED-LLaMA-13B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
BLIP2
0.00
0.00
0.00
0.00
MiniMonkey
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeek-VL. VL-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LISA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Cog CogVLM-CHE Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ShareGPT4V-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ShareGPT4V-13B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
BLIP-3 (XGen-MM)
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
AnyGPT
0.00
0.00
0.00
0.00
MiniCPM3-4B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LaVIT-V2 (7B)
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GLM-VL-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Gemini-1.5-Pro
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
L.5
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DMG-LLaVA-ImtmmLLMP
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Idefics3-8B-Llama3
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NEXT-GPT-VI.5
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-V1
37.50
74.10
45.70
63.30
26.40
85,70
56.40
87.60
35.60
47,80
26.70
Otter
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NExT-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
0.00
Yi-vision-v2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
7.24
0.00
0.00
Qwen2-VL-72B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
78

---

On Path to Multimodal Generalist: General-Level and General-Bench
Image Generation Results.
The complete results of all models image generation presented in Table 39 Table 42,
Table 39: Results on Image Generation Group, from #I-G-1 #I-G-3.
#1-G-1
#1-G-2
#1-G-3
Model
(Edge2ing Gen)
(BBC22ing Gen)
(ling Danose)
#11
#11
#11
#21
#31
#41
#51
H6]
#71
SoTA Specialist
18.70
45.40
34.56
34.55
31.45
33.92
34.60
42.05
25.28
Emu2-32B
93.52
0.00
15.77
19.91
20.31
20.57
23.46
143,93
0.00
SEED-LLaMA-13B
127.10
0.00
15.29
18.45
18.72
19.71
170.65
0.00
AnyGPT
158.21
0.00
17.69
19.46
20.04
20.45
189.73
0.00
LaVIT-V2 (7B)
79.79
0.00
18.76
20.87
21.59
23.20
24.02
111.03
0.00
NeXI-GPT-VL.5
49.71
0.00
10.45
11.36
13.62
6.38
0.00
0.00
0.00
Vitron-V1
19.78
0.00
14.59
12.68
16.82
7.56
19.55
76.86
0.00
Show-o
8
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Table 40: Results Image Generation Group, from #I-G-4 #I-G-8.
#I-G-4
#I-G-5
#I-G-6
#I-G-7
#I-G-8
Model
(Img Enhance)
(lmng Espain))
(lmg-Sttyl Thans)
(Ing2Mask Gen)
(Img25kerch Gis)
#11
#21
#31
#4
#51
#61
#11
#11
#21
#11
#11
SoTA Specialist
3.29
27.35
24.49
0.54
22.16
19.97
4.86
16.73
31.26
99.29
15.06
Emu2-32B
19.21
14.37
0.00
17.58
0.00
101.80
147.95
15.95
0.00
00
SEED-LLaMA-I3B
15.77
13.17
15.09
0.00
127.42
179.87
16.79
0.00
00
AnyGPT
15.92
14.67
0.00
13.40
15.81
117.21
214.33
17.49
0.00
LaVIT-V2 (7B)
19.67
16.50
0.00
14.79
17.25
149.78
98.58
19.88
0.00
NeXI-GPT-VL.5
20.45
2.63
0.00
0.00
0.00
0.00
75.71
65.89
16.51
0.00
47.30
Vitron-VV VI
15.89
3.78
17.86
0.15
6.90
0.00
32.15
51.48
19.17
86.53
23.47
Show-o
0.00
0.00
0.00
0.00
0.00
0C)
Oc
0.00
0.00
00
79

---

On Path to Multimodal Generalist: General-Leve and General-Bench
Table 41: Results on Image Generation Group, from #I-G-9 to #I-G-14.
#I-G-9
#1-G-10
#1-G-11
#I-G-12
#1-G-13
#1-G-14
Model
(lnn kddi))
(Layoun2lag Gen)
(Maak2lmg Gen)
(Sketch2ling (iea)
(Sound2lng Gen)
(Text-based Ing Editt)
#1t
#2
#31
#11
#It
#1
#24
#34
#41
#11
#1
#21
#31
#44
SoTA Specialist
19.27 70.90 69.30
16.47
25.33
15.78
64.50
67.32
28.10
20.35
31.38
65.80
70.10
102.46
Emu2-32B
22.72 50.17 48.63
118.55
15.43
162.90
201 .994
232.23
19.96
0.00
168.23
0.00
60.13
SEED-LLaMA-13B
14.24 37.01 39.28
87.90
14.58
159.50
238.79
289.85
13.17
0.00
160.10
0.00
47.17
AnyGPT
16.49 37.17 32.98
108.06
14.91
176.46
275.20
305.26
16.65
0.00
177.20
0.00
34.87
LaVIT-V2
17.81 60.78 60.61
89.78
15.79
123.75
246.68
256.77
18.96
0.00
144.22
0.00
56.51
00
NEXT-GPT-VI.5
2.60 45.38 36.59
86.45
6.53
32.67
75.86
12.45
76.59
38.65
60.69
Vitron-VI
3.70 56.49 53.44
24.89
17.95
13.76
36.45
57.59
16.34
0.00
89.62
43.78
59.78
Show-o
0.00 0.00
0.00
0.00
0.00
0.00
0.00
0.00
Table 42: Results on Image Generation Group, from #I-G-15.
#I-G-15
Model
(Tx22mmg Gen)
#1.
#21
#3.
#4.
#5.
#6.
#71
#81
#91
#101
#1
SoTA Specialist
19.86
32.57
10.58
15.66
177.30
9.71
31.20
27.86
29.91
12.52
30.07
Emu2-32B
87.59
22.97
71.44
00
167.91
97.15
18.26
15.79
14.59
126.80
17.40
SEED-LJaMA-13B
78.91
26.53
84.25
00
224.63
74.42
16.92
16.37
15.77
133.85
16.91
AnyGPT
94.82
17.64
88.13
00
243.63
66.10
17.97
15.55
16.78
127.49
18.21
LaVIT-V2 (7B)
66.93
21.05
65.39
181.58
57.33
21.22
16.48
17.44
101.92
19.12
VEXT-GPT-VI.5
77.38
26.75
70.14
102.24
234.65
86.78
24.65
16.58
14.26
125.86
20.58
Vitron-V VI
49.58
21.38
46.56
95.68
203.86
73.64
19.87
13.64
10.97
93.78
18.63
Show-o
120.36
9.21
119.19
220.46
290.44
199.99
10.44
7.55
8.2
181.22
9.73
80

---

On Path to Multimodal Generalist: General-Level and General-
C.7 Results of Video-related Tasks
Video Comprehension Results.
The complete results of all models video comprehension are presented from Table 43
to Table 55.
Table 43: Results on Video Comprehension Group, from #V-C-1 (a).
#V-C-1
Model
(Video QA
#11
#21
#31
4-1
#51
#61
#71
#81
#91
SoTA Specialist
27.70
9.80
17.80
20.50
24.80
25.70
26.40
25.20
21.20
IntemVV.-22.-88B
21.10
7.50
13.10
12.40
16.20
24.50
21.70
18.50
14.90
InterVL--2-- ---2-66B
14.10
31.90
34.50
16.10
22.60
23.50
Intern 2eeemVL-2.5-2000
25.60
8.20
15.10
19.50
22.30
26.40
25.10
23.70
17.80
Qwen2- VL-7B
27.70
7.40
16.40
19.80
22.30
25.10
25.20
16.80
Qwen2-VL- ---22BB
27.70
9.80
17.80
20.50
25.80
27.20
26.10
24.50
18.90
DeepSeek- VL-2-small
7.30
2.60
6.10
10.40
7.60
8.40
10.70
DeepSeek-VI.. V1.-2
8.50
6,50
5.00
5.80
11.30
10.50
8.20
10.10
8.60
I.L.V -One- Vision-7B
18.70
10.50
6.40
12.60
11.50
10.20
9.40
12.20
LLaVA-OL Va----n-- Laa--Onn---iiie Vision-72B
18.50
9.40
3.10
12.30
17.10
15.80
15.10
17.60
16.60
Sa2VA-8B
24.40
10.30
25.30
21.20
12.60
13.40
18.40
14.80
19.60
Sa2VA-26B
25.30
10.90
26.10
22.60
15.20
14.90
19.60
17.80
20.10
CoLVA-2B
17.10
6,90
11.60
11.50
15.30
21.80
19.80
17.20
13.90
CoLVA-4B
7.30
13.00
12.20
16.10
24.40
21.50
18.20
14.80
Long-LLaVA-98
41,30
14.50
33.20
39.10
16.50
13.60
19.60
18.60
18.60
NEXT-GPT-VI.5
4.60
1.50
5.60
15.70
5.60
8.70
13.40
13.40
Vitron-V1
7.20
4,80
3.70
5.80
12.40
6.30
8.90
14.10
15,20
IntemVL-2-8B
7.30
12.90
13.20
15.30
23.50
19.30
VidAgent
24.80
10.80
25.90
22.20
13.60
15.40
20.40
16.70
20.30
Table 44: Results Video Comprehension Group, from #V-C-1 (b).
#V-C-1
Model
#101
#111
#121
#131
#141
#15t
#16t
#171
SoTA Specialist
100.00
100.00
85.50
20.40
32.10
18.20
63.10
17.90
Intem
100.00
100.00
80.20
18.50
27.40
14.00
58.20
15.40
Intern VL-221--222BB
100.00
100.00
82,70
13.70
14.50
12.40
79.30
16.20
ILtemmV-22.52206
100.00
100.00
83.30
31.00
19.20
69.20
Qwen2-VLL VL-7B
100.00
100.00
85,50
20.40
32.10
18.20
64.40
18.30
Qwen2-VL-727
100.00
100.00
86.90
22.30
32.30
19.40
70.20
20.40
DeepSeet-VL--2smaall
74.30
100.00
64.00
7.40
2.10
13.60
49.30
8.40
DeepSeek-VL->
53.50
100.00
72.50
4.80
1,90
1.50
46.70
10.40
100.00
100.00
64.60
4.50
11.60
6.80
69.00
10.10
LLAVA-One- VOV---O---iiion One-V-sion
100.00
100.00
70.00
6,20
13.60
13.50
83.70
17.40
Sa2VA-8B
100.00
100.00
80.80
12.70
13.30
7.70
77.70
12.10
Sa2VA-26B
100.00
100.00
83.10
16.80
20.40
14.10
79.60
14.10
CoL-V-2B
100.00
100.00
76.00
17.20
16.60
8.60
54.20
12,60
CoLVA-4B
100.00
100.00
80.00
18.10
26.50
13.10
56.30
14.50
Long- LLaVA-9B
100.00
100.00
76.20
12.90
10.10
72.30
NEXT-GPT-VI.5
83.50
100,00
64.50
3.20
8.90
1.20
51.70
4,30
Vitron-V
100.00
98.00
73.80
7.10
3.50
56.70
5.20
Inter VL-2-8B
100.00
99.00
78.6
17.50
27.90
14.70
58.70
15.70
VidAgent
100.00
100.00
82.80
14.20
24.30
10.70
75.70
12.30
81

---

On Path to Multimodal Generalist: General-Level and General-Bence
Table 45: Results Video Comprehension Group, from #V-C-2 (a).
#V-C-2
Model
Recog)
#11
#21
#31
#41
#51
#61
#71
#81
#91
SoTA Specialist
11.10
21.00
29.10
4.50
65.90
10.50
66.70
64.10
76.60
Intern VL-2.558B
8.50
21.50
22.10
3.90
56.20
8.10
62.20
60.80
77.90
Intern VL-2-26
17.10
24.50
2.60
8,40
88,90
15.00
78.90
75.00
82,20
Intern 2.e5-22-2.5-26B
13.20
25.10
25.50
65.20
13.20
75.30
70.20
Qwen2-VL VL-7B
13.10
20.30
29.10
4.50
65,90
10.50
61.30
63,30
63,20
Qwen2-VL-729
15.80
24.80
12.20
73.20
74.50
70.50
DeepSeek-VL-2-smile
12.00
10.70
0.00
4.80
18.60
8,70
50.00
50.00
44.40
DeepSeck-VI.2
9.00
8.70
2.90
6.50
30.00
8.50
56.70
40.00
50.00
LaaV--One-Vision-7
13.80
22.30
11,50
5.00
46.50
5.00
51.10
53.00
52.20
M..VA--Ome-iision-222
16.70
24.20
17.40
50.50
6.90
80.00
80.00
77.80
Sa2VA-8B
10.00
9.20
0.00
5.30
0.00
12.70
80.00
76.00
80.00
Sa2VA-26B
12.80
12.50
0.00
6.40
0.00
13.20
82.40
78.50
80.20
CoLVA-2B
7.20
17.70
17.40
3.90
50.20
6.50
56.00
54.40
71.60
CoLVA-4B
8.10
20.10
20.00
3.70
55.40
7.10
60.00
58.80
75.00
Long-LLaVA-99
16.60
29.30
1.50
9.40
4.00
14.00
72.20
67.00
71.10
NEXT-GPT-VI.5
4.60
6.10
0.00
1,20
12,10
5.30
64.80
53.10
68.40
Vitron-VI
3.80
7.20
14.00
6.10
70.10
68.30
77.10
Intern VL-2-8B
7.80
20.50
20.80
4.20
55.80
7.10
60.70
60.60
75.60
VidAgent
11.30
13.30
14.20
6.30
53.20
6.50
63.20
74.30
80.00
Table 46: Results on Video Comprehension Group, from #V-C-2 (b).
#V-C-2
Model
(Vid-Otj Recog)
#101
#111
#121
#131
#141
#150
#167
#171
SoTA Specialist
60.10
57.70
23.00
24.70
92.80
91.00
71.30
73.80
IntemVL-22.588
55.30
50.70
18.60
22.40
0.00
0.00
Intern InterVL-22-268 VL-2-26B
65.70
59.40
22.00
24.60
0.00
0.00
0.00
0.00
InternV1.-2.5-2666
65.70
59.40
22.00
24.60
0.00
0.00
0.00
0.00
Qwen2-V1.-1
58.10
$2.10
17.90
21.30
0.00
0.00
0.00
0.00
Qwen2-VL-72B VL-72B
67.20
62.00
22,90
24.50
0.00
0.00
0.00
0.00
DeepSeek-
DeepSeek-VL-2ssmall VL-2-small
36.70
46.70
5.40
8.90
0.00
0.00
0.00
0.00
DeepSeek-VL.
42.20
42.20
10,50
14.90
0.00
0.00
0.00
0.00
-LaVA-One-Vision--
58.90
66.70
7.40
6.40
0.00
0.00
0.00
0.00
LLaVA-Onee-Vision-728
73.30
74.40
12,50
9,80
0.00
0.00
0.00
0.00
Sa2VA-8B
63.30
80.00
6.00
4.40
0.00
0.00
0.00
Sa2VA-26B
65.60
82,20
8.00
5.80
0.00
0.00
0.00
0.00
CoLVA-2B
50.40
48.20
14.40
19.80
0.00
0.00
CoLVA-4B
53.30
50.10
17.60
20.50
0.00
0.00
0.00
0,00
InternVL.-2-2BB
72.20
80.00
3.70
4.80
0.00
0.00
0.00
0.00
Long-LLaVA-99
66.70
65.60
15.00
13.80
0.00
0.00
0.00
0.00
NEXT-GPT-V1.5
57.20
64.70
10.70
9.60
0.00
0.00
0.00
Vitron-Vi VI
60.80
65.20
11.90
11.30
0.00
0.00
0.00
0.00
VL
55.60
50.90
19.60
21.40
0.00
0.00
0.00
0.00
VidAgent
63.30
80.00
6.50
14.40
0.00
0.00
0.00
0.00
82

---

On Path to Multimodal Generalist: General-Level and General-Bent
Table 47; Results Video Comprchension Group, from #V-C-3 (a).
#V-C-3
Model
Recog)
#11
#21
#47
#51
#61
#71
SoTA Specialist
17.60
14.60
20.10
17.40
21.30
13.50
17.70
26.00
Intern
13.20
10.50
14.20
15.10
17.40
6.80
14.90
22.60
IntemVL-2-266
19,40
13.70
19.80
15.80
20.80
12.40
17.30
24.10
2-22.-266B
19.40
13.70
15.80
12.40
17.30
24.10
Owen2-V1.-7B
14.10
9.80
13.90
14.80
18.40
9.00
15,20
13.50
Owen2-VL.-72B
20.80
12.90
20.20
16,90
23.00
17.80
18.90
DeepSeek- VL-2-small
7.40
4.30
7.90
14.10
8.50
6.30
14.30
13.70
DeepSeek-VLL VV--2
14.60
7.90
14.00
17.70
10.80
15.50
21.00
19.60
LLaVA-One SOWNA-OO--Viiio-772
Vision-7B
10.10
8.60
11.20
10.60
11.10
10.80
9.90
13.70
LLaVA-One- LLava-Ome-Vissun7222
20.60
16,20
22.10
17.10
23,00
15.80
17.40
21.50
Sa2VA-8B
17.80
22.40
22.50
22.40
21.90
19.80
25.30
27.50
Sa2VA-26B
18.80
22.60
24.10
24.60
23.10
20.40
26.40
28.20
VA--B
11.30
9.20
10.90
13.90
14.40
5.30
12.20
20.00
4oLVA-4B
12.40
9,90
13.00
14.30
16.60
6.10
13.80
21.10
Intern V--2-266
20.20
20.50
24.40
27.30
29.10
23.30
Long-LLaVA-9B
21.90
18.90
21.00
20.00
18.70
22.40
23.10
21.60
NEXT-GPT-VI.5
16.50
14.90
10.70
17.60
8.70
12.60
Vitron-VI
13.60
16.20
17.40
13,50
17.20
12.00
9.10
10.80
VL-2-88
12,20
10.60
14.20
15.10
17.30
6.80
13.80
21.50
VidAgent
12.30
14,40
12.30
22.30
20.80
19.80
22.30
23.50
Table 48; Results Video Comprehension Group, from #V-C-3 (b).
#V-C-3
Model
Reoog)
#91
#101
#111
#12f
#131
#141
#151
#161
SoTA Specialist
18.00
24.90
20.40
61.30
17.20
6.50
19.60
24.80
InternV1.-2.5-4
15.70
22.00
15.80
0.00
16.80
14.20
14.30
18.70
VnterVLL-2-24
17.40
25.40
21.20
0.00
24.30
18.90
23.40
25.40
Intern VL-2.5-26B
17.40
25.40
0.00
24.30
18.90
23.40
25.40
Qwen2- VL-7B
14.90
22.80
14.80
0.00
18.40
13.60
13.80
19.20
Qwen2-VL-
19.20
26.20
0.00
25.90
21.10
28.70
DeepSeek- VL-2-small
8.70
13.20
7.40
0.00
15.50
14.20
16.40
0.80
DeepSeek-V V1.-2
14.80
17.10
12.20
0.00
11.50
4.90
5.30
6.40
SiVV--On--Viiio-11
10.90
12.70
10.90
0.10
15.00
3.20
12.70
8.00
LLaVA-One- Vision-22
18.40
22,50
20.70
0.20
26.90
12.50
19.60
30.40
Sa2VA-8B
23.20
26.50
20.00
0.00
11.30
7.40
0.00
0.00
Sa2VA-26B
23.60
26.80
20.80
0.00
13.10
8.80
0.00
0.00
CoLVA-2B
12.80
18.90
13.20
0.00
14.20
11.90
13.10
15.30
CoLVA-4B
13.60
19,80
15.40
0.00
16.20
13.70
14,00
16.90
Intern VLerr-VL-2-266
21.00
21.80
19.40
0.00
13.60
7.60
19.10
37.60
Long-LLaVA-99
21.10
22.20
21.00
0.40
12.20
9.70
0.00
0.00
NEXT-GPT-VL.5
14.20
6.30
0.00
8.50
4.10
0.00
Vitron-VI
16.50
7.40
10.50
0.00
7.80
4.70
0.00
0.00
Intern VL.-2-B
14.90
22.30
16.80
0.00
15.80
14.00
17.70
VidAgent
20.30
24.50
20.40
0.00
12.30
8.80
13.00
15.30
83

---

On Path to Multimodal Generalist: General-Level and General- -enchh
Table 49; Results on Video Comprehension Group, from #V-C-4 1O-C4Ato #V-C-5.
#V-C-4
#V-C-5
Model
(Vid Understond)
(TW VOS)
#11
#21
#31
#41
#51
#61
#71
#81
#91
#11
#21
#31
#41
SoTA Specialist
23,80
21.50
26.70
23.00
25.90
14.50
16.70
25.80
29.60
78.30
82.90
86.60
79.60
In-e5VVL
22.60
18.70
21.20
19.30
22.90
8.40
14.20
19.80
22.40
0.00
0.00
0.00
0.00
Mmem/VL-2.52200
24.90
19.60
24.20
22.10
24.10
10.30
15.80
24.20
28.90
0.00
0.00
0.00
0.00
Qwen2-VL-22 VL-7B
19.70
17.90
24.80
20.10
21.90
9.70
13.90
16.80
25.80
0.00
0.00
0.00
0.00
Qwen2-VL-728 VL-72B
22.20
24.80
25.10
21.80
26.70
11.70
14.00
25.30
32.70
0.00
0.00
0.00
0.00
DeepSeek-VL-225 VL-2-small
15.30
5.00
9.10
6.40
9.10
4.30
5.40
10.50
5.80
0.00
0.00
0.00
0.00
DeepSeck- VL-2
16.20
6.70
13:40
9.20
13.20
9.20
14,40
10.10
16.60
0.00
0.00
0.00
0.00
LLaVA-Ome-Vision-77
7.50
12.10
12.30
6.30
2.00
3.40
9.10
6.10
0.00
0.00
0.00
LaaVA-Ome-Vision-222
8.70
5.80
24.40
19.30
8.30
3.80
5.80
11.60
7.40
0.00
0.00
0.00
0.00
Sa2VA-8B
4.30
30.00
14.90
4.00
4.30
5.30
5.00
0.00
Sa2VA-26B
6.60
6.90
32.20
16.90
6.20
5.60
5,80
6.90
6.40
0.00
0.00
0.00
0.00
CoLVA-2B
18.20
13.70
19.30
17.30
18.20
5.40
12.40
16.90
19.70
0.00
0.00
0.00
CoLVA-4B
20.10
17.70
21.00
18.70
20.40
7.90
12.90
18.70
21.20
0.00
0.00
0.00
0.00
Long-LLaVA-98
13.90
14.70
14.00
14.70
16.20
22.40
14.30
12.80
0.00
NEXT-GPT-V1.5
1.40
5.40
13.40
7.80
0.00
3.40
5.00
14.50
3.50
0.00
0.00
0.00
0.00
Vitron-VI
2.30
5.70
14.10
6.20
0.00
3.70
5.70
13.60
3.90
59.80
60.20
54.20
36.40
IntemVL-22-8 VL-2-8B
22.30
17.60
20.20
19.30
20.30
8.40
14.20
18.80
17.40
0.00
0.00
0.00
0.00
VidAgent
20.30
14.60
32.20
15.40
13.60
4.30
14.30
15.30
12.30
0.00
0.00
0.00
0.00
Table 50: Results Video Comprehension Group, from #V-C-6 #V-C-9.
#V-C-6
#V-C-7
#V-C-8
#V-C-9
Model
(Ginena vOS)
VOS)
(ReVOS)
#21
#31
#41
#11
#2*
#31
#11
#21
#31
#11
#21
SoTA Specialist
92.60
88.60
89.40
71.10
64.70
50.50
48.40
69.70
72.40
52.40
40.70
40.60
IntemVL-2.5-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Intern 5--2.5-66
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2-V1.-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2-V1.-72
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeek- VL-2-small
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeek-Ve VL-2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-One-Vision--
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-One-Visionn->
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Sa2VA-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
80.40
73.60
59.10
55.00
46.90
Sa2VA-26B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CoLVA-2B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CoLVA-4B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Interm VL-2-66
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Long-LLaVVA-98
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NEXT-GPT-V1.5
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-VI
74.20
64.10
45,30
63.10
12.70
20.40
6.80
68.40
63.80
50.30
45.20
Intern VL-2-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
VidAgent
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
84

---

On Path to Multimodal Generalist: General-Level and General- BBrrcl
Table 51: Results Video Comprehension Group, from #V-C-10 to-CC-11
#V-C-13.
#V-C-10
#V-C-11
#V-C-12
#V-C-13
Model
(Temp Act Det
(Vid Groundi
Depth Est)
#11
#21
#1 1
#21
#31
#41
#51
#11
#21
#31
#1.
#2.
#31
#4.
SoTA Specialist
26.00
35.60
40.20
73.00
35.10
38.70
42.20
31.30
16.30
21.60
0.27
0.12
0.07
0.10
Intern BL-2.5-BB
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Intern -
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2-VL-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2-VL-72B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeopSeck-VI-2-small
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeek-VLL VL-2
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-One- SWWV--On--iiii--77 Vision-7B
0.00
0.00
0.00
0.00
LLaVA-One-Vision--2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Sa2VA-8B
0.00
58.40
41.20
56.70
0.00
0.00
Sa2VA-26B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.000
0.00
0.00
Col.VA-2B
0.00
0.00
0.00
0.00
0.00
0.00
0..00
0.00
0.00
0c
00
VoI.VA-4B
0.00
0.00
0.000
0.00
0.000
0.00
0.00
01.00
0.00
0.00
InternVL-2-22
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
8
Long-LLaVA-99
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NEXT-GPT-V1.5
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
00
Vitron-V V1
6.70
3.60
51.70
81.50
33.60
36.90
53.90
36.40
15.70
10.80
InternVL-2-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
VidAgent
0.00
0.00
0.00
0.00
0.00
0.00
Table 52: Results Video Comprehension Group, from #V-C-14.
#V-C-14
Model
(Okj Masch)
#11
#21
#31
#4*
#51
#61
#71
#81
SoTA Specialist
40.90
31.00
47.60
76.60
50.60
51.20
39.80
46.80
IntemVV--2.5-55
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
VnterVL--2.5--01
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2 Dwen2-VL-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2-VL-72B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeck- VL-2-small
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeek-VLL VL-2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
I.LAAA-One- -n--One-iision-7
0.00
0.00
0.00
0.00
0.00
0.00
0.00
One-Visionn-22B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Sa2VA-8B
20.70
22.40
24.70
18.50
Sa2VA-26B
24.80
14.30
33.60
43.80
16.80
31.20
31.80
31.00
CoLVA-2B
31.00
47.70
68.80
50.60
49.60
33.50
38.40
CoLVA-4B
38.30
31.00
41.10
76.60
41.70
51.20
39,80
46.80
InternVL-2-26B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Long-LLaVA-99
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NEXT-GPT-V1.5
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-V1
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Intem-VL-2--B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
VidAgent
0.00
0.00
0.00
0.00
0.00
0.00
85

---

On Path to Multimodal Generalist: General-Level and General- Benchh
Table 53: Results Video Comprehension Group, from #V-C-15 #V-C-16.
#V-C-15
#V-C-16
(Obj Track)
(Long Vid Track)
Model
#11
#21
#31
#41
#51
#61
#71
#11
#21
#31
#41
#51
SoTA Specialist
66.60
81.80
71.10
69.50
72.20
61.10
60.40
76.30
18.30
82.90
80.20
60.40
Inter L
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LntemVL-2.55226
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2- VL-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2-V1.-722
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeck-VI-2-small
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeek- VL-2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-Oae RAV--One--t Vision-7B
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-One- Vision-2i
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Sa2VA-8B
22.60
77.80
56.70
49.30
38.20
14.30
61.20
4.60
71.80
25.00
Sa2VA-26B
24. 10
78.30
59.10
49.60
38. 10
15.60
7.60
63.30
5.10
71.60
69.60
25.90
VA--2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
VA--VA-4B
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
V
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Long- -LV-LL
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NEXT-GPT-VI.5
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-VI
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
InternVL-2-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
VidAgent
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Table 54: Results Video Comprehension Group, from #V-C-17 #V-C-18.
#V-C-17
#V-C-18
Model
(UAV Track)
(UW Track)
#11
#21
#31
#41
#5*
#11
#21
#31
#41
#51
SoTA Specialist
80.60
78.50
59.90
84.20
81.90
78.60
78.90
77.10
76.90
63.90
IntemVL-22.5.8
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
VnterVL-22.5444
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2- VL-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2-VL-720
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeck-Ve
-L--Scck-VL-2ssmall VL-2-small
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DeepSeek-VI. VL--2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
I.LVAA-On- MAVA-OOe-Visions Vision-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-One- LLVVA-One-iision-2D
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Sa2VA-8B
56.80
47.70
25.20
23.80
59.40
44,50
58.50
64.80
43.20
Sa2VA-26B
57.80
48.10
25.40
24.70
59.60
44.70
58.60
65.30
43.70
29.80
CoLVA-2B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CoLVV-4B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Intern VL-2-26BB
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Long-LLaVA-99
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NEXT-GPT-V1.5
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-VI
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
InternV1-2-8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
01.00
VidAgent
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
86

---

On Path to Multimodal Generalist; General-Level and Ceneral-Bencl
Table 55: Results on Video Comprehension Group, from #V-C-19 #V-C-20.
#V-C-19
#V-C-20
Model
(UAV Track)
(UW Track)
#1
#2.
#3.
#11
#21
#31
#41
SoTA Specialist
49.20
39.40
23.00
22.70
71.80
48.51
32.98
IntemVLL-2.5-8B
19.40
0.00
0.00
0.00
IntemVL-22.-222
00
00
00
21.20
0.00
0.00
0.00
Qwen2-VI-7B
19.90
0.00
0.00
0.00
Qwen2-V1-72B
00
00
OC
22.80
0.00
0.00
0.00
6.10
0.00
0.00
0.00
DeepSeck-VL.22
00
00
0c
12.90
0.00
0.00
AlaVA-One-Vision-77
5.10
0.00
0.00
0.00
LLaVA-One- Vision--28
00
00
00
6.80
0.00
0.00
0.00
Sa2VA-8B
5.90
0.00
0.00
0.00
Sa2VA-26B
00
00
00
6.80
0.00
0.00
0.00
CoLVA-2B
14.40
0.00
0.00
0.00
CoLVA-4B
00
00
00
16.90
0.00
0.00
0.00
IntemVL-22200
3.80
0.00
0.00
0.00
Long-I.LaVA-99
00
00
06
16.80
0.00
0.00
0.00
NEXT-GPT-VI.5
0.20
0.00
0.00
0.00
Vitron-Vi VI
00
00
00
0.10
0.00
0.00
0.00
IntemVL-228D
19.4
0.00
0.00
0.00
VidAgent
0
0
15.5
0.00
0.00
0.00
87

---

On Path to Multimodal Generalist: General-Level and General-Bench
Video Generation Results. The complete results of all models on video generation are presented from Table 56 to Table 59.
Table 56: Results on Video Generation Group, from #V-G-1.
#V-G-1
Model
(Ta22id Gen)
#11
#2
#31
#41
#51
#61
#71
#81
#91
#101
#111
#12f
#13,
SoTA Specialist
96.88
97.03
99.72
64.20
60.92
69.13
83.62
44
47.82
48.20
55.08
24.86
71.27
VidAgent
94,58
95.79
97.75
38.40
54.40
53.53
68.26
48.23
0.00
0.00
34,91
22.38
73.25
LM4LV
0.00
0.00
0.00
0.00
0.00
0.00
NEX--GPT-V15
43.20
36.40
55.10
12.00
13.40
30.70
36.10
10.70
0.00
0.00
6.90
6.10
97.56
Vitron-V V1
46.70
68.40
15.70
15.60
47.80
58.90
13.80
0.00
28.40
12.50
106.30
Table 57: Results on Video Generation Group, from #V-G-2 and #V-G-3.
#V-G-2
#V-G-3
Model
(Cond Vid Gos)
(Acr Tuz2Ni Gien)
#11
#2
#31
#11
#21
#34
#4
#51.
SoTA Specialist
25.40
90.75
51.20
95.61
72.74
105.62
95.36
75.36
VidAgent
22.36
83.43
37.40
53.20
87.38
111.22
102.62
89.40
LM4LV
0.00
0.00
0.00
0.00
OC
Oc:
NEXT-GPT-V1.5
3.45
2,40
14.30
2.60
126.74
140.70
254.12
126.94
Vitron-VI
6.67
33.80
17.50
44.90
98.50
135.90
186.47
115.78
Table 58: Results on Video Generation Group, from #V-G-4.
#V-G-4
(lmg2Vid Gen)
Model
#11
#21
#31
#41
#51
#61
#71
#81
#91
#101
11
SoTA Specialist
59.46
61.55
60.79
62.89
67.37
63.17
59.21
61.97
62.18
65.45
67.89
VidAgent
60.45
62.31
61.95
62.89
67.19
62.94
60.13
65.14
63.35
64.19
69.20
LM4LV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NEXT-GPT-VI.5
10.53
18.76
20.91
17.53
16.79
14.63
18.52
13.84
14.52
16.51
13.74
Vitron-V1
36.10
15.20
36.40
47.80
26.10
23.70
18.50
17.90
16.57
14.37
Table 59: Results on Video Generation Group, from #V-G-5 and #V-G-6.
#V-G-5
#V-G-6
Model
(Vid Enhance)
(Vid Edio
#11
#21
#31
#41
#51
#61
#71
#81
#97
#11
#21
#31
#41
#51
SoTA Specialist
27.53
38.15
36.60
37.87
25.32
25.75
33.86
56.71
58.29
34.27
34.16
36.75
97.50
$4.60
VidAgent
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LM4LV
24.17
33.05
0.00
0.00
22.28
22.46
28.69
50.12
52.33
0.00
0.00
29.64
0.00
0.00
NEXT-GPT-VIA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-Vi
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
88

---

On Path to Multimodal Generalist: General-Level and General-Bende
C.8 Results of Audio-related Tasks
Audio Comprehension Results. and Table 61.
The complete results of all models on audio comprehension are presented in Table 60
Table 60: Results on Audio Comprehension Group, from #A-C-1 to #A-C-4.
#A-C-1
#A-C-2
#A-C-3
#A-C-4
Model
Analy)
Amaly)
1Specchiimo Analy)
Analy)
#21
#31
#41
#11
#21
#31
#11
#11
#21
#31
#41
SoTA Specialist
75.20
85.60
90.33
97.95
98.76
95.29
43.20
70.62
83.50
69.3
74.00
89.20
Owen-Audio-Chat
56.70
75.00
55.40
40.60
93.00
76.80
36.50
76.80
61.20
55.20
33.00
1.40
Owen2-Audio-Inttruc
45.80
99.20
53.20
92.40
94.40
82.80
47.20
61.40
75.20
47.80
19.40
4.80
GAMA
23.40
86.50
32.40
85.70
80.90
77.50
34,20
68.00
63.50
63.90
85,40
0.00
Pengi
21.40
78.90
35.00
76.20
78.00
65.70
36.50
56.70
61.90
39.20
46,00
0.00
SALMONN-7B
24.70
80,50
85.00
65.10
65.40
58.30
24.10
45.56
60.50
16.20
33.70
0.00
SALMONN-I3B
26.04
84.20
93.50
67.80
68.70 70
68.90
31.40
67.80
63.80
19.40
34.60
0.00
WavI.LM
37.45
70.15
90.00
60.20
35.60
63.10
24.50
71,20
56.80
49.80
12.50
1.20
NEXT-GPT-VI.5
12,50
55.40
64.50
30.50
20.10
65.80
43.70
0.80
PandaGPT (13B)
6.50
58.30
57.30
45.10
24.50
26.40
9.80
45.20
32.40
45.60
5.40
0.50
ImageBind-LLA.
10.50
65.00
25.40
50.10
40.70
38.50
18.50
56.80
42.30
25.70
10.40
0.00
ModaVerse-
10.50
50.40
35.20
40.30
18.70
20.10
10.30
32.80
24.00
32.60
4.20
0.00
Any-GPT
34.60
62.10
15.00
66.30
37.40
46.10
12.90
63.40
68.40
78.90
45.00
0.00
Unffied-io-2-XXL
22.50
45.90
13.50
38.70
34.50
32.50
15.80
56.10
56.70
36.80
20.10
0.70
Table 61: Results Audio Comprchension Group, from #A-C- to #A-C-9.
#A-C-5
#A-C-6
#A-C-7
#A-C-8
#A-C-9
Model
(Aud Tech Analy)
(Aud Analy)
(Aud QA)
(Animal Sound Det
(Gnvir Sound Det)
#11
#21
#31
#1
#21
#11
#21
#11
#21
#11
#21
#31
SoTA Specialist
69.40
80.30
65.90
70.50
55.30
38.90
78.50
77.60
78.20
76.40
86.70
71.10
Wven-AAudo-Chae
63.40
58.40
21.34
19.32
20.25
31.27
81.60
86.30
84.00
86.
92.60
56.80
Lren2-AAdd--Instruue
65.70
61.40
10.36
16.82
10.08
34.56
88.80
87.50
70.40
71.58
86.80
45.60
GAMA
23.50
25.40
6.40
28.50
23.60
74.10
89.60
81,50
72.30
80.50
32.60
Pengi
38.40
15.70
5.20
16.80
22.30
15.40
70.50
83.60
71.20
68 .0
78.40
36.70
SALMONN-7B
31.70
17.90
5.10
15.40
15.60
17.80
60.40
75.80
65.00
57.80
66.40
24.60
SALMONN-13B
32.30
35.40
6.30
21.00
17.72
19.00
68.90
79.60
73.50
60.60
75.90
33.50
WavLLM
53.60
31.40
8.90
29.70
23.40
13.50
78.00
86.40
36.40
74.30
77.80
41.60
NEXT-GPT-VI.S
24.60
16.50
2,30
16.80
34.50
25.60
70.30
76.90
63.50
69.50
80.90
57.90
PandaGPT-13B
21.50
10,50
30.50
15.30
69.20
52.90
56.70
61.50
80.10
55.90
ImageBind- LLM
26.70
14.30
22.50
67.30
63.40
36.50
72.60
57.10
VodaVerse-77bv0
14.20
4.10
1.50
2.50
15.30
13.80
$6.30
46.80
$1.60
58.70
75.70
46.00
Any-GPT
28.40
10.80
9.60
36.10 10
36.70
28.90
76.40
62.10
73.80
45.20
52.30
36.40
Unified-io22-XXI
33.10
5.40
7.90
41.30
35.40
12.30
65.10
57.30
69.70
56.80
79.40
45.70
89

---

On Path to Multimodal Generalist: General-Level and General-Bench
Audio Generation Results.
The complete results of all models audio generation are presented in Table 62 and Table 63.
Table 62: Results on Audio Generation Group, from #A-G-1 to #A-G-7.
#A-G-1
#A-G-2
#A-G-3
#A-G-4
#A-G-5
#A-G-6
#A-G-7
Model
(Andio Edio
(Dinlog Gen)
(HimoSpeech Gen)
(TTS)
(Txz2Au)
(1mgZAud Gen)
(V2A)
#11
#11
#11
#21
#11
#21
M11
#21
#11
#11
SoTA Specialist
31,50
3.82
4.12
3.15
5.60
3.76
47.04
36.03
51.40
11.52
LLaMA-Ommi
0.00
3.01
5.61
2.36
20.15
0.00
0.00
0.00
0.00
00
Unified-io
18.36
2.03
7.86
2.35
78.50
2.54
21.78
11.03
24.31
16.97
Any-GPT
23.50
3.24
6.98
2.15
65.80
1.35
16.52
10.24
14.05
27.49
NeXI-GPT-V1.5
13.60
1.15
6.78
1.35
100.00
1.02
53.68
15.34
1.35
12.36
AudioGPT
0.50
1.32
5.32
3.89
45.20
48.63
10.32
0.00
00
SpeechGPT
0.10
2.79
5.74
3.14
63.70
0.00
0.00
0.00
0.00
00
ModaVerse
12,30
1.15
7.52
1.05
100.00
1.00
50.33
7.65
1.05
16.45
Table 63: Results Audio Generation Group, from #A-G-8 to #A-G-111
#A-G-8
#A-G-9
#A-G-10
#A-G-11
Model
Trans)
(Speech Trans)
(Music
Trans)
#14
# #1
#2
#3.
#11 1
#21
#31
#4.
#11
#21
SoTA Specialist
7.10
7.70
10.20
25.80
59.01
2.85
3.87
28.16
I.AMA-Ommi
45.30
89.36
93.56
100.00
0.00
0.00
0.00
0.00
0.00
Unified-io 2
86.23
93.21
100.00
90.36
0.00
0.00
00
3.15
Any-GPT
45.36
56.89
95.45
99.34
0.00
0.00
0.00
0c
1.28
3.65
Next-GPT
96.70
99.30
98.40
100.00
0.00
0.00
0.00
0c
8.76
6.78
AudioGPT
46.30
45.68
94.25
100.00
0.00
0.00
0.00
0.00
0.00
SpeechGPT
30.24
57.96
98.67
100.00
0.00
0.00
0.00
0.00
0.00
Moda Verse
100.00
100.00
100.00
100,00
0.00
0.00
3.59
4.75
90

---

On Path to Multimodal Generalist: General-Level and General-Bende
C.9
Results of 3D-related Tasks
3D Comprehension Results. Table 66.
The complete results of all models 3D comprehension are presented in Table 64 to
Table 64: Results on 3D Comprehension Group, from #D-C-1 to #D-C-.
#D-C-1
#D-C-2
#D-C-3
#D-C-4 (Indoor Scene Seg!
(3D Human Cls)
(3D Serrt Cls)
(Tech Cls)
Model
#11
#21
#31
#41
#51
#11
#21
#11
#21
#11
SoTA Specialist
91.18
96.25
94.29
99.50
100.00
99.20
97.50
95.56
100.00
78.50
3D-VisTA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MM-77
6.36
59.28
45.55
48.12
71.50
0.00
15.00
65.71
80.00
0.00
Poi3LLLM-13B
5.90
56.42
52.77
49.87
79.00
5,00
15.00
69.28
87.00
0.00
3D-LLM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
AvatarGPT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Table 65: Results 3D Comprehension Group, from #D-C-5 to #D-C-9.
#D-C-5
#D-C-6
#D-C77
#D-C-8
#D--C-9
Model
(Ouidocr
(lndoor lnst
tist)
(Part Seg)
(30 Track)
#11
#11
#1
#11
#21
#31
#41
#51
#61
#11
SoTA Specialist
70.02
81,20
55.00
87.31
93.40
88.62
89.38
81,44
89.52
75.20
3D-VisTA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
PointI.LM-77
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Pom3
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
3D-LLM
0.00
0.00
00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
AvatarGPT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Table 66: Results on 3D Comprehension Group, from #D-C-10 #D-C-13.
#D-C-10
#D-C-11
#D-C-112
#D-C-13
Model
(313GGo Analy)
Den
(3D
(3D-Motion Analy)
#11
#1t
#11
#21
#11
#21
#11
#21
#11
#11
SoTA Specialist
9.96
68.52
12,40
35.60
67.20
48.50
71.40
49.10
45.80
22.30
3D-VisTA
00
34.80
63.30
45.40
69.80
47.20
48.10
0.00
PointLLLM-BB
00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
PoinILLM-113B
00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
3D-LLM
0
0.00
12.00
36.50
65.60
47.20
68.80
48.00
46.30
0.00
AvatarGPT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
12.70
91

---

On Path to Multimodal Generalist: General-Level and General-Bench
3D Generation Results.
The complete results of all models 3D generation are presented in Table 67 and Table 68.
Table 67: Results on 3D Generation Group, from #D-G-1 to #D-G-4.
#D-G-1
#D-G-2
#D-G-3
#D-GG4
Model
Comply
(PC2M Recon)
(Tx12P)
(TxL2M
#11
#11
#2
#11
#21
#31
#41
#11
#21
#31
#47
SoTA Specialist
0.22
9.32B-05
4.93E-05
24.94
25.10
24.07
23.56
26.42
25.22
25.93
25.18
MotionGPT-1
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MotionGPT-2
00
00
00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaMA-Mesh
0.00
0.00
0.00
0.00
20.06
14.43
18.06
17.65
Table 68: Results on 3D Generation Group, from #D-GG5 to #D-G-9.
#D-G-5
#D-G-6
#D-G-7
#D&G-8
#D-G-9
Model
(Img2PC
(1mg2M Gon)
(RGBD2PC
Recea)
#11
#21
#31
#41
#11
#21
#31
#4
#11
SoTA Specialist
77.06
78.27
78.87
78.04
83.30
82.88
84.43
83.96
6540.02
6540.02
0.23
MotionGPT-1
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.51
MotionGPT-2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
00
0.60
ILaMA-Mesh
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
00
92

---

On Path to Multimodal Generalist: General-Level and General-Benehs
C.10
Results of NLP Tasks
All the results of all generalists on NLP tasks are shown in Table 69, Table 71, Table 73, Table 75, Table 77, Table 79, Table 81, Table 83, Table 85, Table 87, and Table 89.
Table 69: Results on NLP Group, #L-1.
#L-1
Model
(Cog QA)
#11
#21
#31
#41
#51
#61
#71
#81
#91
SoTA Specialist
87.33
95.00
31.47
28.42
80.00
33.80
40.75
86.45
80.39
74.90
82.40
13.11
5.18
65.80
6.89
16.26
43.00
50.20
Qwen2.5-7B 1ann2.5-7BB1e In-tructtt
74.71
82,00
16.36
4,46
67.80
6.30
19.38
53.65
37,99
1omma-2--9b..
80.59
78.60
20.79
8.08
75.60
9.28
17.19
29.19
59.65
ChatGLM-6b
52.75
67.40
8.27
2.69
43.80
13.03
5.57
43.00
24.21
7
44.51
63.80
0.35
1.01
46.40
0.47
0.08
44.18
22.24
InterLLM-Chat-77
78.24
60.20
13.40
2.53
49.20
9.96
6.79
43.20
25.20
GPT-J-6B
20.39
47.00
0,50
0.23
38.60
0.32
0.52
51.28
27.95
Falcon3-7B-Insiruc
74.31
78.60
18.71
4.24
51.20
7.27
11.83
57.79
27.17
Baichuan2-P7 2B-Beas
20.98
47.80
0.07
1.11
40.80
0.12
1.29
54.64
25.20
Biisstrll-8--nssuuct--222
71.76
80.20
22.14
19.35
65.00
21.33
20.19
37.48
38.19
Yi-Ligntning
79.22
84.60
13.31
6.30
74.20
7.42
22.78
54.44
33,27
GPT-3.5-turbo
20.78
47.40
12.83
3.32
41.00
3.11
23.15
55.42
26.18
GPT-4v
20.78
47.40
14.62
2.26
41.00
2.87
37.43
55.42
26.18
GPT-4o
20.78
47.40
2.92
41.00
3.20
23.34
55.42
26.18
GPT4o-mini
20.78
47.40
14.04
2.04
41.00
2.72
18.71
55.42
26.18
GPT-4o-4096
20.78
47.40
17.68
3.77
41.00
3.89
23.28
55.42
26.18
CharGPT-4o-latest
20.78
47.40
15.33
3.57
41.00
3.35
21.56
55.42
26.18
Claude-3.5-Sonect
64.71
73.20
22.28
7.37
56.20
22.85
20.34
54.24
43,90
Claude-3.5-Opus
60.78
75.60
17.11
5.91
50.00
15.25
25.56
44.18 18
38.58
Emu2-32B
56.27
64.80
14.94
3.58
47.20
11.24
17.72
44.18
36.22
DetGPT
51.18
62.80
12.78
0.09
42.40
4.18
10.66
38.46
31.10
InternV1.2.5-88
78.43
77.60
21.57
8.00
65.40
15.71
20.55
44. 18
54.92
IntemVL2.5-4B
81.37
78.60
20.75
6.41
61.60
20.25
19.66
34.71
50.00
NEXT-GPT-VI.5
43.26
56.84
0.35
0.76
40.58
0.39
0.08
43.65
0.00
IntemVL2.5522
72.35
76.00
16.36
2.49
43.00
11.23
14.74
25.83
33.07
Monkey-10B-char
26.27
$2.20
2.46
15.99
37.80
3.05
5.97
41.42
24.01
DeepSeek-VL-7B
72.54
11.05
3.70
14.20
7.64
14.20
35.89
29.92
Qwen2-V1--7B
36.00
60.00
11.63
3.97
33.00
6.63
12.00
19.00
33.00
Qwen-VL-Chat
59.80
63.80
0.29
0.92
47.40
0.42
0.64
36.88
26.97
Qwen-Audio-CT!
59.80
56.40
9.89
3.04
48.60
6.08
6.05
41.42
23.62
Owen2-Audio-Instrue
85.00
82.00
4.77
2.87
0.00
5.58
19.43
60.00
0,00
MoFF-LLAVA-Ph2222278--888M
64.71
71.80
13.75
4.24
$3.80
8.76
9.78
35.50
26.77
mPLUG-
BELUG-OWI2-LLaMA2-1
61.96
73.40
11.24
2.26
58.00
13.03
6.52
41.42
25.59
1..35--Vision-Instrue
74.71
82.00
19.05
5.06
62.40
19.57
21.83
40.83
29.92
Cambrian-1-89
21.76
49.20
4.95
2.57
41.00
6.13
8.56
55.42
26.38
MiniGPTA-LLLMA2
72.55
47.40
0.35
1.22
63.60
0.44
0.00
37.28
30.71
MMernVL-Chat-VI-
73.13
68.20
14.61
5.80
59.40
7.43
7.46
39.84
35.43
Mini-Intem Caaiteee---hhataae -LC-----B----
60,39
75.60
15.56
5.50
53.80
12.21
14.69
40.03
27.75
65.88
76.40
15.13
2.33
46.60
6.33
6.98
31.75
22.04
GPT4Rol
21.56
2.77
41.20
2.50
0.72
26.18
GLaMM
20.78
47.40
0.00
0.00
41.00
1.28
4.77
55.42
26.18
LLaVA-NeXT-13B
37.25
62.60
9.47
3.64
43.60
6.05
6.63
42.80
27.56
LLaVA-NeXT-34B
43.92
64.60
17.93
5.88
47.20
7.71
8.66
40.83
28.74
Pixtral-12B
46.67
72.40
18.95
6.25
47,80
6.43
7.10
42.21
29.13
SEED-LLaMA-13B
15.29
46.60
6.67
1.03
19.20
2.76
5.43
41.42
24.61
BLIP2
43.13
48.60
11.41
1.08
3,40
4.62
13.49
39.05
32.09
MiniMonkey
70.78
76.20
17.02
2.31
51.00
7.10
17.49
37.48
27.76
DeepSeek VL-7B
5.00
44.20
6.93
46.35
26.18
LISA-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
93

---

On Path to Multimodal Generalist: General-Level and General-Bene Bennch
#L-1
Model
(Cog
#11
#21
#31
#41
#51
#67
#71
#81
#91
CogVLM-Chat
55.88
69.20
14.16
6.53
53.00
13.59
19.73
44.38
35.43
ShareGPT4V-7B
47.45
62.00
10.76
4.26
42.00
8.30
17.69
41.62
27.76
ShareGPT4V-13BB
54.51
67.00
15.50
5.37
50.40
9.17
18.78
39.64
35.24
BLIP-3 (XGen-MM)
51.57
62.80
16.86
5.79
49.20
10.97
17.94
40.24
36.61
AnyGPT
21.96
6.33
4.84
17.00
6.16
17.91
MinCPPM3-4B
52.35
72.80
18,91
6.17
55.80
16.15
20.25
47.93
37.01
LaVIT-V2 (7B)
39.41
60.20
12.92
11.42
45,20
14.59
14.48
37.28
28.94
GLM-VL-Chat
55.69
71.40
5.44
55.60
14.07
20.71
49.51
39.96
Gemini-1.5 1,5-Pro
81.76
83,40
20.86
7.25
62.80
21.03
22.85
54.24
53.15
Gemini-L.5-Flaa
74.51
76.60
19.27
6.63
56.00
18.49
17.74
44.18
46.26
Boc-LLLaVA-nntemLMMMe
49.02
47.20
13.64
2.83
41.20
9.21
13.07
36.65
27.59
Idefiex3-BB-Llamu3
78.04
76.40
16.71
6.39
57.80
16.36
18.20
43.00
41.54
Yi- Yi-Vision-v2
20.78
16.18
6.34
41.00
2.82
22.19
55.42
26.18
Qwen2-VL-72B
41.33
63.95
20.84
8.31
45.77
9.65
25.04
56.93
27.34
Otter
30.00
49.10
1.78
0.76
40.80
0.76
1.01
54.83
26.57
Show-o
18.00
45.00
1.17
8.62
20.00
7.19
12.64
29.10
31.00
NExT-Chat
20.78
47.40
2.88
1.19
41.00
1.04
3.52
55.42
26.18
Intern VL2-266
81.70
68.00
18.40
6.50
63.80
10.70
19.20
42.90
33.80
Qwen2- Dwen2-VL--2BB
81.50
86.60
24.80
14.10
76.00
29.50
22.20
50.60
73.40
DeepSeek-Vi V---2ekkVV--2-small
74.30
80.20
8.90
4.20
57.20
6.20
12.00
36.70
28.00
DeepSeek-VL-2
77.80
79.00
3.70
65.60
6.90
16.70
28.70
LLaVA-One-Viste
76.50
79.00
19.90
5.60
64.60
15.30
18.80
42.80
46.30
LLaVA-One-Visiont Visionnii
85.90
87.40
27.80
6.90
76.80
25.90
26.00
53,50
64.00
Sa2VA-8B
84.90
82.20
25.60
14.60
71.40
18.60
22.50
49. 10
59.10
Su2VA-26B
76.50
79.80
20.60
7.30
75.40
22.10
13.40
45.40
68.90
Col.A--2
49.60
68.80
11.00
2.90
42.20
9.40
12.10
27.60
20.90
CoLVA-4B
84.90
84.80
16.70
5.00
69.40
11.20
16.40
39.80
50.40
Long-I.LaVA
26.90
49.60
13.30
2.50
41.20
10.90
13.30
55.40
25.40
LM4LV
0.00
0.00
0.00
0.00
0.00
Vitron-VI
40.32
56.84
3.50
1.67
35.42
2.56
1.09
40.37
0.00
PandaGPT (13B)
15.78
67.54
0.14
0.90
0.00
0.45
0.00
47.32
0.00
AnyGPT
21.96
24.00
6.33
4.84
17.00
6.16
7.87
19.92
17.91
GAMA
34.60
$7.80
4.75
2.29
0.00
4.85
0.35
76.50
0.00
Pengi
20.40
40.80
2.56
1.89
3.45
3.24
0.23
56.80
0.00
SALMONN-7B
23.53
70,90
1.84
1.03
0.00
1.48
0.61
68.23
0.00
SALMONN-13B
24.31
71.20
1.78
0.90
23.40
2.13
0.65
69.60
0.00
WavLLM
32.50
$7.46
2.01
0.00
4.32
83.10
ImageBind -LM LLM
28.70
56.71
3.45
1.78
0.00
2.56
0.10
58.30
0,00
Unified-io-2-XXI.
21.56
47.60
1.27
2.77
41.20
2.50
0.72
55.42
26.18
ModaVerse- 7b-v0
15.78
37.23
3.10
0.50
0.00
0.34
0.00
43.24
0.00
AudioGPT-GPT4
21.45
42.34
10.23
2.04
39.80
2.68
38.56
54.98
25.38
SpeechGPT-7B-com
54.23
4.23
0.00
4.09
45.70
0.00
LLaMA-Omni
56.67
63.50
13.11
5.18
15.60
6.89
16.26
43.00
0.00
3D-LLM
20.78
47.40
17.55
8.61
41.00
21.31
7.92
55.42
26.18
PM-7
20.78
47.40
0.35
1.15
41.00
0.48
0.87
55.42
26.18
PoinLLM-133
20.20
47.40
0.35
1.20
40.80
0.47
0.88
55.42
26.77
3D-VisTA
20.78
47.40
2.48
0.28
41.00
0.56
3.65
55.42
26.18
MotionGPT-TS
20.78
47.40
0.27
0.11
41.00
0.04
5.52
55.42
26.18
MotionGPT-LLaMA.
23.14
47.40
0.98
1.14
36.00
0.81
0.90
50.10
23.43
AvatarGPT
20.78
0.30
41.00
55.42
26.18
LLaMA-Mesh
65.69
72.40
0.35
1.64
57.20
0.48
0.85
39.45
25.98
94

---

On Path to Multimodal Generalist: General-Level and General-Bene
Table 71: Results NLP Group, #L-2.
#1-2
Model
(batics NIPI
#11
#21
#31
#41
#51
#61
#71
#81
491
SoTA Specialist
95.62
45.39
75.00
79.80
94.20 20
95.40
99.88 88
95.80
95.00
Meta-Llama-3.I-9FF
83.40
24.08
48.80
67.00
56.00
81.00
45.40
94.20
11.00
Qwen2.5-7B. --nnst--t-
84.80
32.24
36.60
46.80
80.40
66.40
91.40
55.60
62.80
Gemma-2-9h-it
89.60
35.31
35,00
57.40
86.60
25.40
95.00
58.60
54.80
CharGLM-6b
68.40
31.80
0.00
1.60
6.60
67.20
42.80
Vicuma-7b-vl.5
50.40
20.44
29.80
0.00
0.00
0.00
0.00
0.00
0.00
InternLM-Chat-71
75.60
26.83
36.00
43.80
9.40
25.80
27.80
29.00
6.00
GPT-J-6B
72.00
3.76
21.00
0.00
0.00
0.00
0.00
0.00
Malcon3-7B-oo IB-sruct
87.20
31.23
44,20
49.20
82,60
68.00
40.40
54.80
67.60
Baichuan2-7B-BB
82.40
4.46
20.20
0.00
0.00
0.00
0.00
0.00
0.00
Ministral-80 Berrall----mmmee 3B-Instruct-244
87.40
21.66
42.80
48.20
79.60
40.40
88.40
41.80
37.60
Yi-Ligntning
91.40
34.43
53,40
55.60
74.60
42.60
96.40
45.20
51.20
GPT-3.5-ttubo
84.40
26.35
20.20
56.40
78.40
47.00
69.00
48.80 80
76.20
GPT-4v
84.40
29.63
20.20
54.00
82.60
77.40
99.60
64.00
49.80
GPT-4o
84.40
30.29
20.20
54.20
80.40
75.60
99.00
63.60
55,40
GPT4o-mini
84.40
29.33
20.20
55.00
80.20
63.60
98.80
59.60
59.00
GPT-40-4006
84,40
30.83
20.20
55.20
84,20
55.80
99.20
65.40
66.20
ChatGPT-4o-latst
84.40
27.61
20.20
53.40
82.00
71.00
99.20
61.80
68.00
Claude-3.5-
85.40
39.21
38.80
39.75
55.21
45, 19
59.26
57.57
60.07
Claude-3.5-Opus
86.40
34.19
35.40
43.75
43.99
38.95
55.59
57.13
49.76
Emu2-32B
79.60
29.15
34.44
46.40
37.46
48.13
53.04
44.61
DetGPT
73.40
18.91
28.80
29.53
36.38
31,68
45.03
45.16
44,91
VL2.58L
85.20
24.83
45.60
22.99
23.17
33.21
35.42
80.80
78.58
InternVL.2.544B
78.80
18.69
39.00
20.49
20.76
33.75
32.58
80.80
78.98
NEXT-GPT-VI.5
50.78
20.44
30.74
23.10
18.60
10.98
21.43
25.70
0.00
InternVL.2.522B
70.60
21.88
17.97
31.47
34.11
64,80
65.25
Monkey-1OB-chat
69.00
5.13
23.20
5.35
5.97
12.17
8.54
43.20
43.63
DeepSeck-VI-775
85.00
29.48
35.80
18.12
26.04
33.82
33.59
68.00
69.69
Qwen2-VLL VL-7B
27.00
29.00
46.00
34.00
1.00
26.00
12.00
Qwen- VI-Chat
82.00
9.65
32.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen-Audio-Chnn
49.00
32.49
31.40
46.80
43.60
68.30
60.40
31.50
7.80
Qwen2-Audio-Insten
66.85
33.24
0.00
47.00
49.60
68.20
61.60
30.00
6.20
MoE-LLLVAA-Phi2-2.77-444
79.20
30.48
28.40
52.20
80.20
13.00
39.60
47.00
63,40
mPP.UGG-OW/2-LJAMM22..
82.40
31.59
35.20
53.60
78.40
46.40
76.40
48.60
29.80
Mhi-3.5-Vision-Instee
83.40
30.88
41.40
50.60
83.20
91.20
89.20
60.60
$5.20
Cambrian-I-8B
84.40
22.09
20.40
44.20
1.80
3.80
0.20
0.00
0.80
MiniGPT4-LJ.MMA2
81.60
16.20
43.80
0.00
0.00
0.00
0.00
0.00
0.00
InterVV-CCha-Vi
87.80
27.53
39.20
22.83
26.50
34.19
32.93
75.00
69.09
Mini-IntemVVl-Char-BB-WWW
71.80
22.41
35.80
19.28
23.91
32.03
31.11
74.20
74.34
InemLM--CCompser--WL1115
75.20
29.08
32.20
18.52
8.52
32.73
26.87
59.60
68.88
GPT4Rol
84.20
19.25
20.40
24.20
20.29
12.94
21.43
47.20
33.13
GLaMM
84.40
20.20
8.33
9.58
12.72
12.72
33.13
LLaVA-NeXT-133
75.60
27.81
24.80
53.20
56.80
36.20
64.40
49.60
53.20
LLaVA-NeXT-34B
78.60
36.41
28.20
49.20
62.80
44.80
71.20
51.20
58.40
Pixtral-12B
80.20
38.20
34.40
52.40
61.40
40,20
74.20
52.00
59.60
SEED-LLaMA-13B
44.20
14.32
14.60
38.80
47.60
17.40
45.60
39.80
30.60
BLIP2
76.80
43.59
40.60
49.00
60.80
38.60
62.80
43.60
$7.20
MiniMonkey
72.40
35.01
34.00
$4.80
56.60
13.60
71.20
48.40
78.00
DeepSeck-VL-7B
76.40
31.40
22.80
22.28
22.42
28.05
0.00
52.40
47.68
LISA-7B
0.00
0.00
0.00
0.00
CogVLM-Chat
83.20
24.30
33.60
34.64
44.71
41.14
47.47
48.75
44.84
ShareGPT4V-7B
72.60
21.24
30.80
28.76
36.28
30.66
44.28
47.46
40.08
ShareGPT4V-133B
78.00
25.67
34.00
36.74
45.58
33.09
43.25
51.01
42.50
95

---

On Path to Multimodal Generalist: General-Level and General-Bende
#L-2
Model
(Pihicx
#11
#21
#31
#41
#51
#67
#71
#81
#91
BLIP-3 (XGen-MM)
78.80
25.63
36.00
35.55
41.23
38.53
45.28
48.25
47.43
AnyGPT
29.20
9.42
12.80
14.54
16.12
15.31
17.22
19.70
15.66
MiniCPM3-4B
86.60
26.23
32.20
33.08
46.01
33.86
48.91
58.84
52.47
LaVIT-V2 (7B)
64.40
21.72
27.80
26.72
33.86
26.62
36.15
44.13
43.77
GLM-VL-Che
86.80
27.19
30.40
32.97
43.97
32.48
57.70
51.78
Gemini-1.5-PM
87.00
34.33
44.20
51.60
69.80
69.00
81,20
62.60
78.40
Gemini-1..5Flast
83.40
29.98
40.60
47.00
60.80
63.60
77.40
58.00
75.20
OMG-LLaVA-IntemL...
59.80
12.26
21.20
19.20
17.00
9.40
24.20
20.20
32.60
Idefics3-8B-Llama3
82.20
27.71
39.60
48.20
63.60
61.20
76.40
59.80
73.20
Yi-Vision-v2
84.40
20.63
20.20
53.60
67.60
41.99
96.20
58.80
48.60
Qwen2-VL- VL-72B
89.20
25.73
20.27
46.67
74.67
80.02
96.00
47.33
62.67
Otter
69.00
10.33
20.00
0.00
0.00
0.00
0.00
0,00
0,00
Show-o
79.00
6.89
23.00
26.73
17.75
39.24
31.10
36.87
21.54
NExT-Chat
20.78
7.15
20.20
0.00
0.00
0.00
0.00
0.00
0,00
InternVL2-226B
90.00
29.10
43.40
51,20
81.50
41,40
91.40
53.70
64.00
Mwen2-VL-72B
91.80
32.80
52.40
73.40
77.40
93.40
62.20
DeepSeck-VL. ----eek-VLL-2smalll
83.80
34,20
39.20
53.60
65.60
24.80
68.40
26.40
36.20
DeepSeek-VL-2
86.80
35.10
42.20
56.60
81.20
65.80
78.80
28.40
69.00
LLaVA-One- -n----nn--iis
89.20
25.30
49.00
54.80
82.80
58.80
90.20
56.80
61.60
JLaVA-One-Vision-2
92.40
24.70
58.00
0.20
0.00
44,80
99.40
54.60
3.80
Sa2VA-8B
89.20
30.40
48.80
48.60
84,40
38.20
76.20
62.80
Sa2VA-26B
91.60
28.90
52.60
51.00
85.40
41.60
98.80
54.80
71.20
CoLVA-2B
67.40
29.00
30.20
51.20
77.40
1.80
6.20
55.80
34.40
Col
81.40
30.50
44.40
54.00
71.60
28.60
93.80
36.40
58.80
VA
87.20
30.80
22.00
36.60
45.20
64.40
66.80
60.20
32.20
LM4LY
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-VV VI
57.83
10.32
23.45
13.50
38.40
10.98
23.45
27.50
0.05
PandaGPT (13B)
60.40
19.32
0.00
1.20
2.10
0.00
0.00
0.00
0.00
AnyGPT
29.20
9.42
14.54
16.12
15.31
17.22
GAMA
35.87
17.47
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Pengi
68.51
14.35
0.00
0.00
0.00
0.00
0.00
0.00
0.00
SALMONN-7B
1.00
21.59
0.00
1.60
4.40
1.60
1.00
1.00
SALMONN-13B
1.20
22.67
0,00
1.50
4.70
1.40
1.10
0.00
0,00
WMv
25.83
14.56
0.00
1.40
4.30
2.40
1.30
0.00
0.00
ImageBind- BiL
56.40
13.60
0.00
1.10
4.50
1.70
1.10
0.00
0.00
Seltied-io-2-XX-
56.80
17.56
17.80
16.50
18.50
10.30
11.40
4.30
23.60
ModaVerse-7b--e
49.39
1.02
1.90
0.00
0.00
AudioGPT-GPT4
84.30
30.40
20.10
54.10
81.90
77.40
98.70
63.60
49.70
SpecehGPT-7B-com
45.60
13.56
0.00
1.20
4.10
0.00
0.00
0.00
0.00
LLaMA-Omai
44.50
24.08
0.00
45,30
34.20
45,30
23.10
14.50
11.00
3D-LLM
84.40
9.29
20.20
1.20
18.00
0.00
0.00
0.00
0.20
PoinLLLM-7B
84.40
20.20
0.00
0.00
0.00
LM-133
84.60
19.69
23.60
0.00
0.00
0.00
0.00
0.00
0.00
3D-VisTA
84.40
0.64
20.20
0.00
0.00
0.00
0.00
0.00
0.00
MotionGPT-TS
84.40
3.56
20.20
0.00
0.00
0.00
0.00
0.00
MotionGPT-LLANE -LLMA
71.20
5,30
28.40
0.00
0.00
0.00
0.00
0.00
0.00
AvatarGPT
84.40
11.48
20.20
0.00
0.00
0.00
0.00
0.00
0.00
LaMM--Msh
87.20
17.93
45.20
0.00
0.00
0.00
0.00
0.00
0.00
96

---

On Path to Multimodal Generalist: General-Level and General-
Table 73:
Results on NLP Group, from #L-3 #L.4.
#1--3
#1.-4
Model
(Dumain QA)
(Social QA)
#11
#31
#41
#51
#16
#71
#11
#21
#31
#41
#51
#61
SoTA Specialist
82.62
48.79
73.55
53.31
94.50
96.67
88.00
48.73 73
82.00
49.21
97.30
91.00
57.74
Meta-Llama 1.lmm-..1-2
3.1-8B-Instruue
29.20
22.89
36.90
41.42
83.00
84.24
81.80
42.66
63.80
29.39
94.40
93.00
39.89
Qwen2.5-7B-Imss
20.56
27.48
41.47
27.69
86,80
83.43
77.20
35.58
63.00
20.48
94.20
93.80
22.69
Gemma-2-9b-it
15.78
28.87
37.97
15.70
89.80
80.40
77.20
23.13
62.60
14.85
93.40
93.80
13.42
ChatGL M-6b
19.60
30.54
27.28
52.60
55.76
51.60
31.83
57.60
22.98
68.60
71.00
Vicuma- 7b-v1.
15.63
23.22
15.62
14.90
59.00
48.08
$7.60
18.83
55.40
19.06
67.40
71.00
15.46
Intern. Inteml.M--hat-7 M-Chat-7b
14.20
15.81
29.55
20.55
67.20
73.13
70.80
23.91
67.80
17.15
84.60
81.60
15.51
GPT-J-6B
12.50
12.93
6.11
10.87
43.40
32.12
36.60
48.20
10.40
36.20
31.60
12.42
Falcon3-7B-Instmen
24.60
26.11
38.93
38.45
81.20
65.45
74.60
41.65
65.20
27.99
87.80
87.80
30.33
Baichuan2-7B-BB
12.63
14.51
8.38
12.63
48.20
33.13
39.20
9.04
51.40
9.78
35.40
30.00
13.65
Ministral-8B-IIstree--2
16.39
27.41
38.60
21.27
82.60
83.84
76.60
28.74
66.60
14.68
92.20
92.40
16.87
Yi-Ligntuing
27.08
26.62
41.18
37.58
89.60
85.45
80.20
42.34
68.20
28.95
95.00
94,00
34.55
GPT-3.5-turbo
14.22
29.65
36.26
24.38
33.13
40.00
28.
51.20
14.95
35.60
30.00
13.58
GPT-4v
22.56
25.12
36.75
36.22
48.20
33.13
40.00
37.08
51.20
25.99
25.60
30.00
25.40
GPT-4o
19.95
26.31
43.04
27.21
48.20
33.13
40.00
33.32
51.20
19.45
35.60
30.00
19.43
GPT4o-mini
21.42
41.72
33.83
48.20
33.13
40.00
38.67
51.20
23.10
35.60
30.00
24.83
GPT-4o-4096
19.36
27.90
42.29
27.30
48.20
33.13
40.00
33.38
51.20
20.36
35.60
30.00
19.12
ChatGPT-4o-latest
19.67
28.22
43.00
26.80
48.20
33.13
40.00
32.25
51.20
18.64
35.60
30.00
17.50
Claude-3.5-Somne
47.83
37.75
53.34
37.66
69.60
76.20
54.40
41.68
60.20
30.02
58.60
61.00
42.93
Claude-3.5-Opp
46.29
33.32
43.88
28.87
69.40
71.20
49.60
34.60
57.00
24.78
59.60
51.00
33.36
Emu2-32B
40.15
29.51
46.14
26.70
67.60)
69.40
49.80
30.88
48.40
23.51
51.80
49.20
33.57
DetGPT
36.27
21.60
40.13
21.79
61.00
62.80
42.00
28.70
46.00
18.50
43.40
43.00
27.02
IntemVL2..58B
74.80
37.58
65.80
24.73
92.20
90.60
31.10
24.89
25.92
33.72
26.24
31.72
44.54
V12.5mV
78.40
34.30
65.00
22.55
91.80
90.60
26.21
26.87
33.41
19.32
38.71
36.48
36.36
NEXT-GPT-VI.5
15,46
34.87
10.56
12.50
54.74
46.50
53.20
18.56
53.40
18.40
63.50
69.50
13.68
IntemVV.2.5-2B
65.80
35.48
61.20
25.94
75.40
28.97
22.94
33.01
12.06
28.83
31.40
25.34
Konkey-10B-chat
41,80
8.41
47.60
6.09
37.40
43.00
7.65
8.59
11.82
14.87
6.21
11.28
18.78
DeepSeck-VL-778
69.60
37.17
58.80
25.01
84,00
85.20
29.07
24.30
21.08
18.84
19.38
19.90
18.63
Qwen2- VL-7B
27.22
19.56
24.37
35.59
37.00
67.00
$3.00
32.20
42.00
67.00
69.00
38.75
Owen-VL--Chal
18.43
14.25
7.97
15.07
56,20
72.20
60.00
11.93
60.60
16.52
72.20
61.60
17.60
Qwen-Audio- Chat
23.20
27.51
35.14
32.37
54.20
58.98
59.00
37.24
51.00
25.99
81.20
74.00
28.59
Qwen2-Auuio-Insstruct
22.04
26.94
35.01
31.89
65.33
86.60
78.58
37.84
78.65
24.22
90.11
0.00
BE--LLAVA-Phi2-2.1011
4e-3844-4
24.06
28.98
34.26
32.01
60.00
71.00
55.40
36.03
59.60
26.12
71.00
66.60
28.89
mPLUGG-OWD-LLMMA2--
25.13
23.25
34.04
35.23
71.80
83.60
75.40
39.15
66.40
28.12
83.60
83.00
32.44
Phi-3.5-Vision-IInstue
24.27
26.39
38.03
37.09
79.40
91.00
78.20
40.94
66.60
26.73
91.00
88.00
34.85
Cambrian-1-SB
26.77
17.90
26.20
37.79
48.40
37.00
41.20
28.41
51.20
24.05
37.00
32.20
36.13
MiniGPT4-LLaM22
26.85
14.42
19.58
84.60
73.20
63.40
84.60
66.40
32.67
Intern Micem-L--Chtt-i15
68.80
36.29
61.20
25.20
84.60
86.00
28.85
24.68
19.83
16.15
18.50
18.16
25.45
Wonn-lnteem-L-Cha---BBBBB
72.00
33.38
61.60
21.77
87.80
86.80
25.55
27.77
26.71
12.11
22.99
25.42
30.81
Intern Wow.LL--CCoooose2---We VL-
57.80
34.44
57.20
22.75
75.80
76.80
25.04
26.40
18.82
8.95
17.57
21.57
16.96
GPT4Rol
40.60
17.49
51.40
23.36
36.40
29.60
26.25
3.25
5.47
6.68
0.90
0.60
0.00
GLaMM
40.00
51.20
35.60
30.00
20.24
9.04
9.16
9.55
LLaVA-NeXT-113
27.85
23.33
36.21
20.22
46.40
55.60
46.60
32.13
60.40
19.74
62.60
$1.60
27.55
LLaVA-NeXT-34B
23.78
26.84
33.25
31.89
56.60
58.80
50.20
35.24
57.20
23.19
67.40
61.20
26.64
Pixtral-12B
24.67
36.94
32.92
62.20
62.00
55.60
40.17
62.40
21.02
69.80
66.80
29.71
SEED-LLaMA-13B
12.05
13.95
24.67
15.48
29.20
47.60
42.80
19.07
38.40
15.16
33.60
24.40
20.49
BLIP2
29.27
23.32
41.83
37.69
64.20
63.00
59.20
46.29
68.40
41.84
79.60
88.40
52.61
MiniMonkey
17.43
15.90
31.21
25.90
62.80
65.86
64.40
32,76
58.00
23.26
80.40
78.40
23.68
DeepSeck-VL-91
50.80
33.74
56.20
0.00
61.80
58.00
0.00
24.40
15.43
12.02
14.56
17.63
1.53
LISA-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Chan
45.61
25.47
46.78
34.10
63.40
67.80
53.20
29.83
49.60
27.09
49.60
48.20
35.70
ShareGPT4v-77
33.20
24.84
41.18
22.86
62.60
64.20
42.20
27.93
46.40
20.43
46.00
45.60
27.66
ShareGPT4V-13B
36.19
23.32
40.73
26.17
65.40
64.40
47.80
30.45
44.60
22.43
50.40
49.40
27.20
97

---

On Path to Multimodal Generalist: General-Level and General- Bench
#L-3
#L-4
Model
(Domain QA)
(Social
#11
#21
#31
#41
#51
#61
#71
#1
#21
#31
#41
#51
#61
BLIP-3 (XGen-MM)
41.17
26.99
44.56
25.79
65.40
68.40
47.60
33.70
46.00
24.40
48.80
45.60
30.89
AnyGPT
13.45
10.84
16.73
10.64
24.80
24.20
18.00
0.00
16.00
7.96
18.60
18.20
10.80
MiniCPM3-4B
51.94
34.47
52.23
31.14
69.80
73.00
45.20
33.55
46.00
25.14
56.40
54.00
37.87
LaVIT-V2 (7B)
40.36
23.12
39.05
25.22
61.60
58.60
36.40
25.57
37.80
23.52
45.20
39.00
26.88
GLM-VL-Chal Vh-Chat
50.08
52.19
31.23
71.00
43.60
29.72
43.20
25.78
56.40
$3.80
37.28
Gemini-1.55PM
29.14
34.32
41.77
36.41
89.60
90.20
84.20
42.15
81.40
27.23
86.60
88.40
40.45
Gemini-1 1.5-FFash
26.93
30.08
37.04
33.83
86.20
84.40
78.20
38.47
72.60
23.78
80.20
82.60
36.83
BLGGLLLaVA-ImtmmLMMME
16.07
14.54
26.76
12.95
53.80
51.20
29.80
15.42
14.80
12.91
14.80
16,20
Seffiss-8B--lannas
28.54
29.32
35.93
31.89
87.80
88.20
73.60
39.30
74.40
22.05
78.00
79.60
36.13
Yi- Vision-v22
15.03
23.04
37.95
30.74
48.20
33.13
40.00
35.67
51.20
16.25
35.60
30.00
21.46
Qwen2- Owen2-VL-72B VL-72B
16.47
25.99
34.25
31.83
69.32
46.24
66.31
34.96
66.34
21.97
47.88
61.92
22.47
Otter
19.58
18.72
9.86
21.66
44.60
38.38
46.00
15.12
47.40
18.77
40.60
34,20
20.50
Show-o
14.13
13.04
23.23
38.00
46.00
19.57
38.00
32.00
28.00
NExT-Chat
11.62
12.13
6.66
14.17
48.20
33.13
40.00
12.88
51.20
16.85
35.60
30.00
19.15
VL2
19.30
19.70
33.40
27.10
81.60
75.90
71.40
32.70
65.00
21.70
86.20
86.40
24.70
Qwen2-VL--2 VL-72B
22.80
27.80
38.50
36.50
91.20
83.80
79.20
39.50
70.10
25.00
95.40
96.20
DeepSeek- DeepSeek-VL--2smaall V1-2--mmall
21.40
29.20
40.50
29.70
77.00
80.20
81.40
37.40
65.80
26.50
92.80
92.60
28.20
DeepSeek-VL.2
21.90
28.10
42.40
31.80
80.60
82.80
80.80
40.30
67.60
24.70
92.60
95.40
26.80
LLaVA-One Vision-7B
20.40
21.90
35.30
32.80
82.60
78.20
73.80
33.80
64.80
23.60
88.20
93.00
28.70
LLaVA-One- One-Vision-72B
23.50
25.00
36.60
39.30
91.20
85.30
81.00
41.10
70.00
28.10
96.40
96.00
35.20
Sa2VA-8B
23.30
38.20
37.60
84.60
82.60
78.80
40.50
66.80
26.80
92.80
91.40
Sa2VA-26B
26.20
27.10
38.40
40.00
87.20
77.40
76.20
41.70
68.40
28.40
89.00
89.00
34.80
CoLVA-2B
19.90
20.20
33.50
30.70
53.20
54.30
51.40
34.20
49.80
24.20
66.40
60.60
27.30
VA-BB
21.20
27.80
38.30
30.90
83.60
79.80
80.80
36.20
68.60
23.60
93.00
89.40
26.80
Long-LLaVA
23.10
25.70
30.90
34.80
50.20
36.40
42.60
34.40
54,00
28.10
38.40
46.60
36.20
LM4IV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-V V1
13.47
14.58
15.60
13.20
54.74
45.60
$2.30
20.56
49.82
13.40
60.34
62.34
16.83
PandaGPT (13B)
9.78
5.34
17.61
14.90
45.10
22.08
0.00
15.89
10.38
11.15
56.90
24.00)
15.46
AnyGPT
13.45
10.84
16.73
10.64
24.80
24.20
18.00
16.00
18.20
GAMA
22.58
35.84
24.29
28.91
0.00
63.83
65.50
27.30
0.00
21.57
0.00
47.21
25.57
Pengi
14.35
30.46
20.67
26.63
0.00
67.00
45.60
20.40
0.00
20.89
0.00
45.30
15.67
SALMONN-7B
11.63
7.06
28.09
18.58
66.36
32.00
0.00
21.90
20.00
14.68
68.44
28.00
19.19
SALMONN-13B
11.87
7.31
29.78
17.91
76.53
32.13
0.00
22.50
22.10
13.02
77.80
29.70
17.40
WavLLMM
21.45
25.43
29.01
0.00
66.55
57.35
28.09
0.00
22.12
0.00
68.85
26.23
BrngeeBind-LLLM
23.50
45.60
24.30
28.70
0.00
78.00
56.00
29.80
0.00
12.40
0.00
48.75
15.40
Unified-in-2-XXI.
16.51
14.35
15.67
12.40
21.34
24.67
20.56
2.50
5.03
6.78
0.80
0.60
0.00
ModaVerse-
9.45
12.40
10.34
20.45
0.00
14.50
9.68
10.32
46.78
23.60
AudioGPT-GPT4
22.76
25.05
36.89
36.48
43.10
32.18
39.78
37.09
51.10
25.34
25.70
29.80
25.30
SpecchGPT-7B-com
20.34
34.23
24.34
28.35
0.00
65.40
45.60
13.09
0.00
20.40
0.00
67.50
16.34
LLaMA-Ommi
27.60
22.89
36.90
41.42
83.00
74.56
80.20
43.30
63.20
28.37
84.40
65.30
19.78
3D-LLM
4.24
1.03
17.38
8.69
48.20
33.13
40.00
12.15
51.20
6.40
35.60
30.00
5.32
LointLLM-7B
16.65
16.02
17.27
48.20
33.13
40.00
20.15
51.20
35.60
30.00
18.32
PointLLLM-13B
17.24
23.20
16.45
16.99
45.80
32.93
32.20
20.01
51.20
19.91
34.80
30.60
18.39
3D-VisTA
0.20
0.00
0.31
0.05
48.20
33.13
40.00
0.23
51.20
0.10
35.60
30.00
0.05
MotionGPT-TS
1.27
5.00
0.91
48.20
33.13
40.00
1.82
51.20
1.13
35.60
30.00
PotionGPT-LLLMA
18.53
19.87
8.41
6.14
43.40
33.54
41.80
11.64
46.00
14.46
39.40
39.20
21.27
AvatarGPT
7.99
2.38
13.69
7.64
48.20
33.13
40.00
11.84
51.20
7.48
35.60
30.00
6.09
LLaMA-Mal Messh
28.67
22.44
23.21
34.27
71.60
76.16
73.00
27.88
69.60
29.28
90.80
86.40
37.13
98

---

On Path to Multimodal Generalist: General-Level and General- Bench
Table 75:
Results NLP Group, from #L-5 #L-7.
#1.-5
#L-6
#1-7
Model
(Non-Trad QA)
(Advance QA)
(Math Abdliy)
#11
#21
#31
#11
#21
#31
#41
#11
#21
#31
SoTA Specialist
43.46
77.01
53.59
48.87
61.77
83.33
32.01
88.00
75.00
81.00
69.34
26.80
18.10
15.14
17.70
17.43
46.32
33.22
71.20
48.60
29.20
30.89
Bwen2.5-7B-Insteee
28.44
17.60
14.79
18.52
20.58
18.50
18.91
70.60
32.80
11.20
26.82
Gemma-2-9b-
33.64
21.78
15.99
20.32
21.18
57.76
29.47
72.60
46.20
24.20
34,41
ChatGl
29.10
21.31
7.75
26.56
24.08
12.04
21.72
54.80
24.40
0.00
32.15
Vicuna-7b--
3.30
6.10
4.14
2.27
0.73
1.05
2.19
60.00
23.40
0.00
36.43
Internl M-Chat-7b
30.64
22.06
9.71
22.50
26.23
12.30
21.58
71.60
26.20
1.80
33,92
GPT-J-6B
2.03
1.86
0.05
0.41
1.22
56.40
24.40
0.00
25.68
F In-sstructt
26.33
23.70
14,10
18.79
21.22
49.90
25.67
70.00
27.20
35.00
33.89
Baichuan2-7B 2-BaanBB-BBBeeee
1.12
1.11
4.19
0.52
0.69
9.47
0.61
63.00
24.40
22.40
29.18
34.01
42.06
41.87
36.71
45.25
15.73
29.36
75.40
30.80
0.00
8.19
Yi-Ligntning
25.42
20.37
15.79
16.91
19.55
55.99
20.86
85.83
25.80
18.60
22,95
GPT-3.5-urbb
30.46
12.26
5.86
18.50
12.41
27.83
20.27
64.00
24.40
0.00
43.85
GPT-4v
25.48
10.78
7.04
12.22
10.99
34.89
17.12
64.00
24.40
37.05
21.91
GPT-4o
29.30
11.67
7.64
13.55
11.79
22.52
19.45
64.00
24.40
35.05
21.97
GPT4o-mini
27.23
11.65
12.92
12.92
23.57
16.45
64.00
24.40
34.42
20.65
GPT-4o-4096
29.82
13.40
8.30
13.97
13.12
24.63
23.01
64.00
24.40
0.00
20.47
ChaGGPT-4-latest
30.07
13.75
7.05
14.22
12.06
28.23
22.35
64.00
24.40
0.00
18.59
Claude-3.55Some
31.26
40.07
28.78
32.27
35.84
32.42
39.32
64.60
41.40
52.00
47.62
Claude-3,5-Op
25.90
41.43
26.53
29.89
28.32
26.08
32.91
59.20
39.00
39.60
42.33
Emu2-32B
26.88
34.62
21.72
22.77
25.92
25.93
28.18
53.40
37.80
40.60
38.71
DetGPT
19.99
31.33
17.94
20.13
19.03
20.62
27.87
50.00
34.60
32.00
29.10
IntemVL2.558B
31.39
45.60
$1.60
39.12
25.42
17.02
50.29
31.76
51.66
$2.69
48.58
V12.5mL.
42.78
69.20
44.60
38.83
26.18
8.28
50.89
31.73
50.05
31.95
41.91
NEXT-GPT-VI.5
3.30
5,40
3.87
1.67
0.73
9.05
11,60
59.30
20.80
0.00
28.67
V.2.52
26.67
60.20
36.80
33.95
22.26
15.69
50.69
28.50
38.05
46.41
38.50
Monkey-10BB
3.61
57.60
19.60
35.77
31.43
20.13
53.69
24.29
55.51
29.66
51.58
DeepSeck-VL--7.
21.28
70.60
29.60
38.93
38.93
70.60
70.60
70.60
70.60
57.66
21.28
Qwen2-VL-7B
28.90
15.17
18.27
3.21
9.33
59.00
28.00
41.96
38.80
Qwen-VL--Char VL.-Chat
2.77
2.71
2.83
1.71
0.61
0.14
1.70
63.70
22.60
7.78
27.42
Qwen-Audin Chat
25.48
18.21
11.82
16.40
16.65
5.38
16.29
61.40
24.60
0.00
34.70
Qwen2-Audio-Instrucc
25.10
20.82
17.08
18.48
14.83
22.94
72.30
0.00
36.75
MoE-LLAVA- P...--.7B---eeeee
24.98
22.15
15.64
18.48
20.99
31.77
21.40
55,40
25.00
43.29
27.91
mPLUGG-Ow22-LLMMA22-
28.29
24.61
11.98
22.22
31.37
8.96
11.52
53.72
27.00
36.59
30.15
Phi-3.5- Instoni
30.25
35.99
20,52
33.28
34.39
54.86
27.17
59.44
30.40
44.65
42.87
Cambrian-1-88
22.34
15.39
10.86
15.86
16.65
28.93
14.32
48.28
24.40
42.66
34.28
MiniGPT4-LLaMA2
3.19
3.57
2.03
0.72
0.16
1.93
50.13
31.20
7.78
38.93
Intem MntemVL-CChtt-1--
17.79
63.00
37.20
2,20
42.92
14.65
52.49
30.51
9.17
51.42
50.62
Mini Wini-intemVL-Chha--BBB
32.81
63.80
23.20
5.00
44.55
22.60
50.09
30.43
16.63
1.30
44.39
Intern DeemLM-XCooposer/-VVV VL-1888
16.86
56.80
28.60
0.00
24.36
12.91
53.89
27.50
25.00
0.00
44.65
GPT4Rol
1.00
63.60
24.80
0.00
24.10
0.00
46.30
5.37
20.01
0,00
10.79
GLaMM
2.96
64.00
0.00
10.70
46.30
25.00
25.00
12.95
LLaVA-NeXT-13B
25.73
22.48
8.39
23.73
19.67
18.19
18.58
57.20
22.80
2.60
28.53
LLaVA-NeXT-34B
25.01
23.54
10.19
18.67
22.67
17.56
19.65
67.40
25.40
4.20
35.85
Pixtral-12B
27.67
27.40
11.43
23.07
20.02
24.11
22.53
63.80
27.20
4.80
36.67
SEED-LLaMA-13B
14.78
8.88
2.75
10.86
13.24
13.61
13.12
43.40
14.60
1.20
19.23
BLIP2
29.63
33.93
11.57
31.67
22.47
30.97
26.53
67.60
27.80
0.00
35,29
MiniMonkey
24.32
20.98
8.68
17.47
24.01
28.28
19.85
55.20
31.80
3.20
36.68
DeepSeck-VL-78
9.17
64.40
27.00
0.40
39.87
7.19
47.70
24.55
46.70
21.73
17.69
LISA-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Chat
25.04
37.77
24.74
25.41
26.30
23.17
33.51
57.40
41.60
42.80
39.87
ShareGPTTv--7B
23.19
28.22
15.83
18.39
16.80
19.34
23.24
48.40
36.00
35.20
32.22
ShareGPT4V-13B
23.23
29.06
21.96
26.99
24.93
23.99
29.35
51.60
35.60
40.60
38.75
99

---

On Path to Multimodal Generalist: General-Level and General-Bende
#L-5
#L-6
#L-7
Model
(Non-Trad QA)
(Adeance QA)
(Math Alility
#11
#21
#31
#11
#21
#31
#41
#51
#1
#21
#31
BLIP-3 (XGen-MM)
24.30
34.92
24.63
26.70
21.70
23.61
33.51
53,40
35.20
41.60
38.25
AnyGPT
10.04
13.99
0.00
11.03
7.64
8.71
13.25
18.80
14.20
13.00
13.45
MiniCPM3-4B
32.94
37.32
22.96
24.09
29.80
27.26
31.46
55.20
43.40
48.20
41.36
LaVTT-V2 (7B)
21.39
31.47
18.52
24.23
24.22
23.68
32.52
39.40
28.20
29.20
30.68
GLM-V VL-Chat
32.60
38.71
21.72
26.52
24.85
29.63
54.20
41.00
46.80
41.84
Gemini-1.5-Pa
29.58
23.54
14.52
34.49
32.95
23.57
25.68
66.60
29.40
38.20
36.80
Bemini-1.5-Flaa
31.04
19.82
11.75
31.64
28.39
21.68
26.45
61.60
26.80
34,20
35,40
BMMGLLLaV-InnemLLMMMe
7.09
9.38
8.62
6.21
18.07
18.80
17.80
Idefics3-8B-Llance
30.14
22.15
12.56
32.20
26.36
18.62
22.47
60.20
25.20
33,20
31,80
Yi- Vi-Vinion-v2
23.42
15.03
15.12
14.57
11.36
22.98
18.79
64.00
24.40
33.49
20.04
Qwen2-VL-P7 VL-72B
29.19
18.23
11.01
30.03
17.87
32.39
18.81
67.21
27.30
43.00
31.16
Otter
41.37
1.81
2.26
2.34
1.78
0.23
3,41
60.03
23.60
7.84
20.83
Show-o
10.81
30.80
6.87
24.04
11.25
8,43
37.82
31.00
6.57
7.14
NExT-Chat
14.70
3.92
7.27
6.41
6.93
0.00
3.96
64.00
24.40
0.00
8.57
IntemVL2-2265
25.60
22.40
15.20
21.60
29.90
39.10
25.20
70.60
29.40
12.80
43.50
Qwen2-VL VL-72B
31.70
38.10
41.20
41.70
40.40
68.80
25.10
73.40
30.10
47.80
35.10
DeepSeek- DeopSeek-VL-22small
VL-2-small
25.90
21.60
12.00
17.30
18.30
25.90
17.60
72.00
26.60
4,40
29.20
ScepSeck-VL.2>
25.60
18.60
12.30
17.10
18.60
33,80
20.10
73.80
29.40
15.20
28.40
LLaVA-One DONVA-On--Viiii-75
Vision-7B
33.10
27.90
12.60
28.80
35.90
63.50
35.40
71.40
44.20
7.00
45.20
LLaVA-One-Vision-
34.40
22.60
32.60
35.80
43.20
75.80
53.20
77.00
52.20
8.60
44,90
SaBVA-8B
28.60
33.70
45.60
26.80
37.80
52.70
33.80
74.40
34.20
Sa2VA-26B
28.70
23.60
24.10
21.30
24.10
59.20
30.20
73.00
55.20
17.40
35.20
CoLVA-2B
27.60
28.40
9.10
24.30
30.90
20.60
23.70
61.60
23.60
5,40
27.50
Col VA-4B
30.60
22.00
12.20
21.20
26.10
29.40
32.30
72.60
47.80
14.00
34,80
Long- LL
22.50
22.40
8.60
18.00
31.10
31.70
18.30
66.60
24.20
0,60
37.10
LM4LV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-VI VI
3.10
14.30
3,87
13.76
0.73
9.05
16.10
53,90
21.20
0.00
26.87
PandaGPT (13B)
2.30
4,70
3.19
1.94
0.73
0.94
2.07
56.96
5.63
0.00
8.75
AnyGPT
10.04
13.99
7.64
8.71
13.25
13.00
13.45
GAMA
21.54
15.39
7.79
18.91
13.97
0.31
1.16
68.30
0.00
0.00
7.03
Pengi
19.78
13.67
4.58
15.34
13.89
0.23
1.02
58.33
0.00
0.00
3,98
SALMONN-7B
9.98
5.37
5.39
2.83
3.56
0.06
2.41
62.52
20.00
0.00
14.51
CALMONN-13B
10.32
5.68
6.89
3.21
5.76
0.00
3,20
48.15
20.40
0,00
15.78
WavLLM
20.73
14.67
6.45
16.78
14.34
0.32
1.07
29.78
0.00
0.00
11.45
ImageBind-LLLe -LM
10.45
7.56
4.34
6.78
8.45
0.34
1.04
35.22
0.00
0.00
10.29
Unified-io-2-XXI
1.20
11.10
5.67
0.00
0.89
0.00
13.56
55.40
10.40
0.00
6.57
ModaVerse-7b-yy
2.10
4.50
0.30
1.34
54.36
0.00
8.34
AudioGPT-GPT4
24.58
10.70
7.04
12.22
10.23
34.67
15.67
63.59
24.30
37.01
21.89
SpeechGPT-7B-com
18.34
14.67
6.45
15.34
13.45
0.20
1.00
78.60
0.00
0.00
8.35
LLaMA-
26.80
18.10
15.14
16.70
7.45
6.23
32.23
73.40
48.60
29.20
30.89
3D-LLM
18.51
35.77
16.70
44.61
43.83
0.00
30.84
63.80
24.40
0.00
12.09
PointLLM-7B
3.27
6.15
2.20
0.72
0.77
2.13
64.00
34.64
PointLLM-133B
3.30
6.08
4.29
2,25
0.73
0.00
2.12
55.40
23.80
0.00
31.16
3D-VisTA
0.51
0.62
3.47
0.61
0.33
0.26
0.11
64.00
24.40
0.00
0.00
MotionGPT-TS
0.81
0.00
0.09
0.16
0.00
0.07
64.00
24.40
0.00
0.03
MotionGPT-LLLMA
4.75
2.24
3.56
2.31
1.68
1.52
1.66
55.20
24.80
0.00
14.34
AvatarGPT
8.53
2.16
3.77
1.59
1.58
0.11
1.19
64.00
24.40
0.00
0.90
LLaMA- Mesh
3.24
6.05
5.98
2.10
0.72
0.93
1.92
68.60
24.60
0.00
39.22
100

---

On Path to Multimodal Generalist: General-Level and General- -Brrch
Table 77: Results on NLP Group, from #L-8 to #L-11.
#L-8
#1.9
#L.-10
#1-11
Model
(Coee Abilit))
(X-Lm
(Tx: Sum)
(Dilog Gen)
#11
#21
#31
#41
#51
#11
#21
#31
#41
#11
#21
#31
SoTA Specialist
80.50
79.50
75.06
76.94
77.20
71.06
83.61
72.83
91.30
48.
76.23
91.35
28.27
Leaa--lamm--.1--
19.72
47.31
32.72
37.18
8.67
54.06
61.57
21.24
76.06
47.56
35.00
50.92
14,80
Owen2.5-7B-Insttee
11.72
54.49
35.18
53.19
54.39
56.55
66.23
30.63
76.70
47.90
33.47
49.16
15.34
Gemma-2-9b-it
18.11
49.10
33.06
59.71
55.22
53.25
71.21
11.99
79.77
36.49
36.88
50.18
13.34
ChatGL.M-6b
11.13
46.71
22.25
56.28
43.05
23.50
39.20
24.54
43.80
22.69
49.0M
13.93
Vicuma-7b-vil
3.79
53.69
24.58
52.13
42.18
53.36
60.79
30.15
74.92
45.00
28.18
48.56
5.06
Intern M-Chatt7b
7.33
53.69
26.96
54.17
1,50
42.63
51.67
25.74
59.02
43.33
26.05
47.63
13.01
GPT-J-6B
2.21
53.19
17.64
44.19
8.13
8,48
17.59
0.98
13.83
22.76
27.38
Falcon3-7B-Instmen
12.70
53.29
35.64
47.73
61.20
54.26
59.21
30.75
78.92
46.71
32.30
47.20
15.56
Buichuan2-7B-BB
18.02
46.51
33.39
49.72
50.66
37.06
54.76
13.24
58.91
22.42
12.63
20.43
4.92
Ministral-SB-Instmnce
16.74
49.90
12.77
5.34
29.54
56.93
70.39
21.21
78.95
47.77
33.55
49.06
13.73
Yi-Ligntning
20,95
49.70
37.06
56.55
53,59
60.59
74.60
31.47
83.43
43.32
32.40
50.38
15.29
GPT-3.5-trbo
17.51
54.09
33.62
39.57
37.16
63.11
77.81
33.81
86.27
46.69
32.20
50.18
14.40
GPT-4v
27.81
48.70
35.28
35.05
33.51
63.12
79.38
32.70
86.25
43.08
32.87
50.39
13,96
GPT-4o
28.47
51.70
33.80
36.01
33.20
64.63
81.15
34.15
87.52
44.59
32.80
50.67
14.49
GPT4o-mini
29.30
51.30
36.24
37.52
32.79
64.87
79.53
34.15
44.44
33.42
50.97
GPT-1o-4096
29.30
47.90
34.00
36.37
33.50
64.24
80.61
33.27
87.42
43.00
32.65
51.12
15.49
ChatGPT-4o-latstt
28.60
49.30
35.93
35.82
33.28
64.18
80.93
33.16
87.38
44.10
33.39
50.28
15.08
Claude-3.5-
33.84
54.49
51.79
59.84
41.81
55.89
52.89
31.70
65.44
43.18
44.77
73.60
20.70
Claude-3.5-Opus
37.47
51.30
46.75
56.24
45.08
57.25
52.53
29.57
63.71
36.54
46.37
62.96
17.39
Emu2-32B
32.04
44.91
45.72
46.81
39.13
52.12
47.20
25.60
56.98
35.04
37.31
66.70
13.05
DetGPT
21.64
38.92
37.46
42.17
33.02
42.92
40.35
20.22
50.60
33.39
33.55
57.84
11.12
IntemVL2..-8B
55.64
27.19
62.17
39.17
30.91
44.99
13.88
33.23
31.30
19.89
44.79
89.00
9.07
V12.554B
50.96
25.71
57.20
42.76
29.77
46.58
12.58
26.61
30.53
24.94
41.01
89.68
7.82
NEXT-GPT-VI.5
2.60
50.34
19.56
50.25
40.61
$1.90
56.63
29.68
67.75
41.50
25.40
46.30
5.06
I.te-2VL.2.5-BB
32.80
27.27
47.17
40.82
27.67
42.10
12.37
35.38
27.72
22.89
42.25
86.33
9.07
Monkey-10B-on 10B-chat
61,05
26.19
72.42
45.67
32.96
45.92
7.21
36.08
33.61
8.00
62.68
91.14
11.00
DeepSeck-VI.-7B
61.84
30.88
72.05
47.26
31.63
47.87
14.46
43.94
1.83
30.94
51.35
88.00
10.37
Qwen2- VL-7B
7.19
52.00
21.93
24.82
29.55
51.07
15.98
35.21
23.71
47.81
12.62
Qwen-VL-Chat
2.37
53.69
26.76
45.80
9.56
35.24
50.27
15.34
63.81
24.47
16.08
40.63
12.83
Qwen-Audio-Chl Chat
8.70
50.89
19.75
42.35
48.19
53.76
61.65
29.92
76.89
44.99
26.21
48.45
10.47
Qwen2-Auutio-nstrruc
13.57
53.69
20.76
43.56
40.36
57.88
60.68
28.59
78.96
41.77
29.40
48.45
12.08
LLLAVA-Phi2-227BB
Ph22-2.7B--4--388
8.13
53.69
23.32
53.18
53.79
55.75
62.18
11.40
75.50
46.05
33.73
48.80
11.16
mPL.UG-OW2-LLMMA2-77
8.50
46.91
31.44
50.86
43.54
31.63
27.47
12.00
50.07
43.70
31.77
48.15
10.65
Phi-3.5-Vision-Insttue
18.92
53.09
33,90
58.48
54.41
55.85
49.05
13.62
74.88
47.24
31.66
49.57
12.74
Cumbrian-1-8B
2.37
53.49
24.93
63.10
14.25
36.46
60.68
10.69
54.16
34.43
28.95
34.16
8.90
MiniGPT4-LLaMA2
2.51
35.05
55.30
68.62
35.30
11.57
37.60
22.71
48.11
10.42
Intern Wi-cWt--Chut-1155
57.70
30.73
65.14
44.44
44.44
47.94
13.49
43.81
28.73
16.33
39.75
83.44
0.00
Hin--ineemVL-Chh--BBBB
41.39
24.82
61.32
43.74
29.68
46.01
12.77
36.12
29.01
13.13
43.70
84.72
0.00
InternL Xww.poXXoommoo----- -L------------ L.BB
38.12
26.67
58.53
46.80
32.32
46.73
12.45
18.42
18.91
22.65
36.37
89.48
0.00
GPT4Rol
15.18
7.10
17.79
22.13
13.56
20.58
4.23
9.92
17.10
2.52
17.63
23.03
0.00
GLaMM
20.84
0.00
5.40
16.10
21.56
2.14
0.00
21.51
40.08
0.00
LLaVA-NeXT-13B
6,32
36.33
20.27
33.54
19.56
36.23
32.17
8.47
36.07
32.18
25.54
44.36
9.57
LLaVA-NeXT-34B
8.54
34.93
29.87
45.67
27.05
40.98
35.86
14.76
57.21
36.32
34.89
45.60
12.88
Pixtral-12B
11.12
48.70
27.43
40.28
26.79
37.43
36.27
13.24
45.04
36.25
32.56
44.09
12.00
LEED-LLLMA-118
6.32
27.94
15.79
22.01
7.39
26.19
32.94
7.26
20.15
24.31
14.23
32.14
4.80
BLIP2
3.67
22.18
16.73
29.64
6.22
37.23
31.77
13.81
48.51
33.50
34.77
44.32
8.56
MiniMonkey
14.91
53.09
28.92
51.37
49.53
46.17
39.55
31.19
63.30
42.16
29.16
49.10
12.73
DeepSeck-VL.C. VL-7BB
34.22
4.67
23.61
29.68
16.64
43.10
11.22
30.94
15.02
20.13
17.55
67.17
11.07
LISA-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Chai
27.55
44.31
46.48
46.92
40.04
52.95
50.06
26.57
58.20
35.14
41.01
61.89
16.03
ShareGPTaV-77
25.34
40.32
39.39
44.56
33.93
43.31
38.89
20.14
49.65
33.37
32.77
57.34
10.51
SharGGTT4V-13B
27.13
40.78
46.31
34.36
43.03
23.19
55.32
36.16
34.16
61.82
12.86
101

---

On Path to Multimodal Generalist: General-Level and General-Bene
#L-88
#L-9
#L-10
#L-11
Model
(Code Abilit)
NMT)
(Tst
#11
#21
#31
#41
#51
#11
#21
#31
#41
#11
#21
#31
#11
BLIP-3 (XGen-MM)
29.16
44.71
46.65
50.89
39.95
50.52
44.40
23.96
57.25
34.78
34.46
63.27
13.21
AnyGPT
12.37
17.40
17.53
17.56
14.45
18.44
17.13
9.47
19.59
12.22
15.13
21.36
0.00
MiniCPM3-4B
35.26
43.60
48.38
47.14
45.70
46.22
45.74
34.05
54.93
37.62
41.81
65.77
19.79
LaVIT-V2 (7B)
24.75
39.80
36.70
45.34
28.72
39.30
33.25
26.56
41.02
28.57
35.75
54.39
13.20
GLM-VL-Chat
30.68
48.68
45.02
45.57
31.90
34.14
38.47
63.65
20.85
Gemimi-1.5-PP 1-5-Pr
25.60
52.30
35.24
40.91
43.54
58.65
68.28
38.72
85.46
44.08
32.73
49.32
14.27
Demini-1.5-FFaat
23.21
48.70
33.05
36.72
33.20
52.75
64.15
32.93
80.12
38.24
31.34
50.15
12.74
OMG-LLaVA-Imtmm..MM
9.38
14.98
25.23
26.27
25.69
9.41
7.47
23.35
12.62
12.47
26.74
Idefics3-8B- 8B-Llam33
18.49
51.30
32.80
36.87
32.96
47.93
52.87
14.33
68.93
36.85
28.12
51.32
11.08
Yi- Vi-Vision-22
5.47
53.09
31.72
35.39
31.72
60.47
72.07
21.26
54.83
46.97
27.92
48.72
12.69
Qwen2-VL-72B VL-72B
25.07
52.66
32.31
39.28
50.83
64.91
82,45
28.42
84.52
45.12
31.45
49.72
17.27
Otter
5,43
47.33
17.20
18.67
9.76
9.96
30.37
2.25
9.22
28.02
17.26
24.31
6.51
Show-o
3.41
3.65
4.43
13.22
5.89
10.73
0.00
0.00
NExT-Chat
0.00
46.30
4.54
25.11
2.39
6.91
12.16
3.62
10.40
27.04
20.52
29.58
8.18
Inter VL2-26B
18,20
51.10
31.80
50.80
54.10
41.30
54.50
26.90
61,00
44,20
29.70
48.20
13.80
Qwen2-VL-72B VL-72B
24.80
53.40
32.30
52.10
26.10
61.10
75.40
30.60
84.10
45.30
31.70
49.90
DeepSeek-VI.-5 V1--2-mmall
13,80
46.90
29.80
66.20
49.40
56.00
60.70
30.80
73.60
41.30
29.90
46.00
15.20
DeepSeck-VLe. VL-2
20.00
44.70
31.90
62.60
55.50
56.10
64.60
31.30
75.30
43.50
30.10
46.80
16.30
LLaVA-Ons Vision-7B
21,50
53.70
34.10
63. 10
59.50
53.30
62.40
29.50
75.70
44,90
23.20
49.30
17.70
ILAVA-One- Viseoniiionn22
11.80
53.50
35.30
45.30
70.50
56.40
70.60
18.30
75.70
44.10
29.40
50.90
17.70
Sa2VA-8B
14.40
52.30
35.60
59.90
56.60
57.30
62.80
31.10
45.60
33.20
49.60
Sa2VA-26B
13.40
51.30
35.60
58.50
45.50
59.70
60.90
34.20
79.80
25.70
33.30
50.50
16.90
CoLVA-2B
8.80
54.10
29.10
42.80
51.80
42.20
36.20
26.40
58.20
41.90
27.90
46.40
12.80
CoI.VA-4B
11,80
53.70
34.00
52.10
55.10
54.30
60.50
29.70
74.50
44.70
30.90
49.40
17.20
Long-LLaVa
16.90
46.30
25.20
62.30
41.50
44.30
46.00
30.30
75.00
43.30
21.90
49.10
6.30
I.M4IVV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-Vi V1
12.30
50.34
15.62
50.23
41.09
50.37
46.63
26.78
67.75
41.30
24.60
44.60
5,30
PandaGPT (13B)
0.68
45.37
24.58
52.13
54.58
7.56
5.37
3.84
22.39
7.37
3.89
6.46
0.00
AnyGPT
12.37
17.40
17.53
17.56
14.45
18.44
17.13
9.47
12.22
21.36
GAMA
5.49
53.69
18.79
34.56
67.50
0.08
0.59
0.00
0.03
0.38
0.34
0.65
7.01
Pengi
3.54
23.56
14.34
24.78
67.50
0.05
0.34
0.00
0.05
0.23
0.27
0.04
4.58
SALMONN-7B
1.11
53.89
10.89
25.56
56.36
11.43
12.42
6.40
15.42
9.99
9.87
1.15
SALMONN-13B
1.39
55.89
11.24
27.34
48.33
12.54
13.69
6.56
17.65
11.54
2.99
11.34
1.37
WavLLMM
5.60
54.67
19.20
30.68
67.50
0.09
0.56
0.00
0.36
0.58
6.45
ImageBind-LLM
5.46
46.78
15.45
29.58
67.50
0.06
0.34
0.00
0.03
0.00
0.00
0.00
6.32
Unified-io-2-XXI
3.67
6.70
10.46
15.84
10.31
15.78
4.21
5.75
14.67
2.52
13.56
21.50
0.00
ModaVerse-7bvv0
0.68
43,50
23.60
46.60
7.56
3.84
22.39
7.37
6.46
AudioGPT-GPT4
27.56
48.50
35.46
35.45
33.51
63.05
78.50
33.24
84.34
43.08
32.45
51.23
14,05
SpeschGPT-7B-com
5.67
45.67
20.34
36.59
67.50
0.05
0.61
0.00
0.03
0.45
0.41
0.23
6.79
LLaMA-Omni
19.72
47.31
32.72
37.18
8.67
54.06
61.57
21.24
16.45
17.56
15.00
20.94
4.80
3D-LLM
2.13
46.31
0.00
0.00
0.65
27.70
35.62
51.18
4.04
32.50
29.52
22.22
4.87
PointLLM-7B
3.69
31.93
30.18
50.53
20.17
63.64
42.48
25.13
40.92
PoinLLLM-113
3.53
53.69
0.00
0.00
41.05
42.57
$5.99
25.94
70.58
44.87
30.72
41.31
4.71
VisTiSTA
0.65
46.31
0.00
0.00
0.01
0.86
2.93
0.06
0.05
2.41
2.42
1.70
0.21
MotionGPT-TS
0.23
46.31
4.94
0.05
0.40
0.02
0.01
0.08
0.02
0.31
0.46
0.26
MotionGPT-LLAMA
2.58
46.91
20.50
0.34
5,88
58.54
26.14
45.71
7.41
41.80
20.25
43.45
3,50
AvatarGPT
10.73
46.31
3.63
1.06
0.98
2.24
8.00
0.00
1.64
9.50
9.42
8.44
8.63
LLaMA- Mesh
3.86
53.69
30.35
53.56
54.75
54.69
72.44
26.26
76.21
45.89
28.61
47.
4.93
102

---

On Path to Multimodal Generalist: General-Level and General-Bon Bench
Table 79:
Results NLP Group, from #L.-12 #L-16.
#1-11
#1-13
#1.14
#1-15
#1-16
Model
(TXT Gen)
(Time Series)
(Ix)
(Txi Emsill)
(Sem Analy)
#11
#21
#31
#41
#50
#11
#lt
#1f
#11
#21
#31
#41
SoTA Specialist
93.02
75.19
83,46
85.89
97.19
0.31
94.40
91.41
88.00
98.00
85,40
72.80
Meta-Llama-3.. L.laa-3-..11- 3-1-
-88--nnttttt Instructt
39.83
29.90
31.12
42.10
83.74
7.95
76.40
$1.80
44.79
80.00
75.00
63.80
Owen2.557B-Insstee
50.67
33.58
31.85
47.24
90.90
5.74
88.20
79.35
66.40
49.00
63.80
46.80
Gemma-2-9bas
36.37
25.40
22.22
9.33
86.00
5.75
84,46
55.82
75.60
83.80
62.20
48.00
ChatGLM--6
32.84
27.89
48.30
74.85
10.91
41.80
45.81
50.20
47.80
0.00
0.00
Vicuna-7b-v1..
30.01
30.10
23.34
46.16
90.29
11.41
0.00
0.00
0.00
0.00
0.00
0.00
Internl Iiternl.M-Chare. MM-Char-7b
32.59
26.43
22.48
9.72
91.15
10.23
77.80
6.87
58.80
14,20
31.00
3.60
GPT-J-6B
13.98
9.38
31.53
10.69
9.46
11.41
0.00
0.00
0.00
0.00
0.00
Falcon3-7B-Inntruet
44.34
35.09
26.53
44.22
90.59
5.15
88.80
85.89
64,20
26.80
44.40
47.20
Baichuan2-7B 2-auannn--BBBBe
49.50
36.31
14.53
55.40
93.31
7.85
0.00
0.00
0.00
0.00
0.00
0.00
Sonisral--B--Instuut--22
15.54
16.33
32.30
23.68
29.11
11.08
84.80
72.60
53.80
67.00
61.60
44.40
Yi-Ligntning
52.74
34.45
28.52
52.30
95.39
5,37
72,60
56,24
78.20
70.60
66.40
43.80
GPT-3.5-turoo
35.00
31.90
27.20
38.32
83.48
1.55
73.80
63.19
50.20
41.80
55.80
28.00
GPT-4v
40.36
32.15
28.03
39.08
83,20
3.16
86.20
83.23
85.60
80.60
66.20
28:00
GPT-4o
43.98
33.99
29.40
41.49
83.20
2.58
85.40
86.30
71.40
90.20
69.00
39.40
GPT4o-mini
42.97
32.93
42.10
82.31
4.24
82.00
86.50
82.00
84.40
58.60
47.40
GPT-40-4096
43.32
34.24
28.16
43.04
83.06
3.08
87.40
83.84
80.20
88.00
71.20
40.40
ChatGPT-4o-latest
41.21
32.09
22.33
42.67
82.63
2.73
88.60
77.51
80.40
89.60
69.00
41.60
Claude-3.5-Sonee
59.94
48.94
44.02
47.91
90.18
3.88
68.59
52.58
47.87
51.86
59.26
41.41
Claude-3.5-Opus
58.00
47.16
35.40
42.21
88.65
3.67
30.74
30.74
30.74
30.74
30.74
30.74
Emu2-32B
51.27
43.36
42.43
82.09
9.53
57.54
48.78
44.02
46.12
50.11
34.79
DetGPT
44.80
38.08
26.30
37.63
76.72
16.87
52.97
46.25
42.05
43.83
44.60
30.15
LL2.55VL2.5-8B
87.00
72.20
$4.60
54.60
91.40
75.20
55.40
68.40
48.80
52.40
82.80
42.99
V12.54\
81.80
70.40
69.00
69.20
90.20
70.80
46.00
61.80
48.20
54.60
80.60
39.20
NEXT-GPT-VI.5
25.40
30.40
22.98
45.23
86.43
1.06
68.90
43,20
32.43
28.90
33.65
20.15
IntecnV1.2.52B
83.20
69.20
25.40
30.20
85.40
79.60
50.60
$1.40
48.20
51.60
83.60
42.40
Monkey-10B- 10leey--OB-chatt
38.60
8.80
0.00
0.00
65.60
20.40
0.20
3,00
10.60
4.00
16.80
8.60
DeepSeck-VL-7BB
78.40
68.80
83.00
83.00
85.20
83.00
62.20
50.60
54.20
54.20
84,40
56.40
Qwen2-VL.e VL-7B
32.99
34.13
86.23
6.48
64.00
37.00
0.00
2.00
12.00
Qwen-VL-Chat
14.75
22.33
32.49
8.70
8.24
7.46
0.00
0.00
0.00
0.00
0.00
0.00
Qwen-Audio- Chat
44.56
34.02
30.28
45.78
89.65
0.98
74.50
58.43
59.30
37.50
50.10
45.30
Qwen2-Audio-Instruut
43.34
32.16
30.73
40.75
88.39
1.23
77.60
58.69
61.00
38.40
51.40
47.40
MoE-LLLVVA-Phi2. 2e
4e-344
38.88
31.12
2.10
40.69
91.02
6,35
64.20
60.94
58.80
78.20
53.00
43.00
mPL.UG-OWI2LLLMA222
34.43
33.43
28.22
44.03
90.75
6.16
76.20
38.04
36.20
25.80
59.00
32.40
Vis.on.- Visioon-nnn-ttre
47.43
35.54
27.99
42.16
92.78
3.80
58.40
68.92
70.40
77.00
59.20
46.80
Cambrian-1-88
31.08
27.58
2.68
41.38
90.04
3.14
59.80
0.00
13.40
30.40
19.80
3.20
MiLM22
28.55
34.40
31.91
85.35
7.46
0.00
0.00
0.00
0.00
0.00
InterVV--C VL-Chat--Catt V1-5
65.80
73.20
69.80
68.40
89.20
73.80
48.40
65.80
54.40
55.60
79.80
30.00
Mini-Intemm LL-hhm-B-LCaaa--V1
65.80
44.60
53.40
56.80
84.80
68.60
48.60
50.80
50.20
$2.40
64.40
72.40
InternL WWw.MM-XOOOoo-2---V/ VL-
72.80
60.60
52.40
52.40
85.60
5.60
0.00
3.20
0.00
0.00
0.00
21.20
GPT4Rol
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GLaMM
0.00
DO
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-13B
30.70
21.54
23.34
28.77
70.14
10.24
77.24
68.83
49.40
66.80
48.80
45.00
LLaVA-NeXT:34B
27.31
25.17
21.52
30.19
79,86
8.85
74.83
72.25
49.60
72.60
53.40
46.20
Pixtral-12B
34.42
26.08
23.77
33.09
77.35
8.12
80.91
66.31
54.80
71.20
51.80
44.80
SEED-LLaMA-13B
17.58
13.29
13.04
17.92
42.39
11.16
13.20
34.80
30.20
32.20
25.10
28.40
BLIP2
27.41
22.16
25.04
18.99
84.01
6.43
79.00
66.20
48.40
73.80
52.20
46.40
MiniMonkey
37.57
33.30
21.51
52.99
88.93
10.57
84.00
68.92
46.00
74.60
50.80
46.80
PeopSeck-VL-77
1.40
0.00
0.80
2.80
0.35
0.80
28.40
32.00
2.20
0.00
0.00
LISA-7B
0.00
0.00
Do
0.00
0.00
0.00
0.00
0.00
0.00
CLgVLM-Chat
53.67
42.22
31,42
42,56
81.70
20.45
53.55
51.28
45.25
47.31
49.32
29.67
ShareGPT4V-78
48.38
36.65
30.58
38.87
79.03
27.95
49.45
44.62
37.27
44.93
48.38
27.06
ShareGPT4V-13B
52.42
39.82
29.59
38.93
81.80
24.69
54.03
49.62
45.91
43.94
48.08
34.05
103

---

On Path to Multimodal Generalist: General-Level and General-Bend
#L-12
#L-13
#L-14
#L-15
#L-16
Model
Series)
Fanall)
(Sem Andly)
#11
#21
#31
#41
#51
#11
#11
#11
#11
#21
#31
#41
BLIP-3 (XGen-MM)
51.43
38.84
31.86
41.39
84.73
21.54
55.67
48.61
47.80
48.81
53.39
31.05
AnyGPT
17.27
16.66
12.12
15.78
28.03
33.39
19.48
16.48
17.92
18.21
17.06
11.55
MiniCPM3-4B
52.48
51.50
43.03
46.59
91.38
19.62
57.84
52.66
46.27
55.05
54.65
36.88
LaVIT-V2 (7B)
45.79
36.18
31.67
40.84
67.22
24.13
42.71
37.63
33.18
38.75
43.22
26.92
GLM-VL Vh-Chat
54.43
41.76
88.66
21.37
58.65
49.30
52.65
53.46
37.94
Demimi--15--Pre
52.98
41.15
37.65
46.32
92.34
4.78
84.20
83.70
82.40
86.40
64.80
47.20
Demini-1.5-FFR
49.43
33.48
30.06
41.47
88.95
7.92
80.60
77.41
74.20
80.20
58.60
39.80
OMG-LLaVA-IntemLMMR
24.83
21.02
15.89
18.73
52.60
56.52
26.20
19.08
28.60
20.40
21.60
13.20
Idefics3-8B-Llamee
47.65
34,92
29.01
42.15
84.39
4,31
74.20
76.90
71.00
78.20
60.40
38.40
Yi- Vi-Vinio
35.39
29.41
25.08
36.80
76.91
7.32
59.60
49.69
63.20
85.40
60.20
42.60
Owen2-VL-725B
31.34
31.24
37.84
38.85
84,19
4.45
90.00
80.67
85.33
82.00
69.33
44.00
Otter
13.98
17.65
9.29
11.66
14.88
11.51
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
4.53
4.35
0O
0.00
0.00
0.00
0.00
0.00
NExT-Chat
11.43
16.88
1.92
9.43
26.49
11.54
0.00
0.00
0.00
0.00
0.00
0.00
InternVL2-225
41.20
31.20
15.40
38.90
88.40
64.60
80.20
79.90
64.00
64.80
52.00
42.40
Qwen2- VL-72B
42.50
35.30
42.30
94.70
3.90
83.60
83.60
84.60
86.60
68.00
43.20
DeepSeek-VV DeepSeck-VL-22small
39.30
34.60
28.20
44.90
92.60
12.90
60.40
$5.00
59.00
86.20
61.80
46.40
DeepSeck-VL-2
41.00
35.90
26.20
44.50
94.00
8.80
81.80
68.50
64.20
85.60
60.80
49.40
LLaVA-One Vision-7B
29.90
34.30
11.20
42.80
93.90
5.90
77.40
65.40
65.20
60.40
45.80
47.80
LLaVA-One- -laVA-One-Vision-722 Vision-72B
51.80
32.30
11.30
39.70
83.90
3.60
84.80
10.40
82.40
61.20
64.40
29.40
Sa2VA-8B
41.40
35.90
26.00
45.40
93.80
86.80
76.90
65.40
59.00
47.20
Sa2VA-26B
41.50
36.10
27.20
46.10
94.40
5.70
86.60
85.10
75.20
59.00
65.00
44.60
CoLVA-2B
30.90
33.10
21.20
54.90
90.00
11.00
75.40
3.10
30.40
67.30
25.40
35.80
Col VA-AB
47.60
33.70
24.90
44.80
92.30
9.80
84.40
76.90
79.00
66.20
27.60
45.60
VA
41.20
29.80
31.60
50.40
89.20
11.40
68.60
41.70
41.40
66.80
56.20
46.20
L.MIIV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-VI VI
27.80
30.20
23.10
45.68
85.93
2.47
68.70
43.20
33.47
26.80
31.51
23.62
PandaGPT (13B)
4.78
5.47
6.07
36.48
28.67
0.41
0.00
40.80
0.00
0.00
15.60
0.00
AnyGPT
17.27
12.12
15.78
28.03
33.39
19.48
17.92
17.06
11.55
GAMA
8.11
0.29
1.92
6.04
7.74
0.78
0.00
0.00
0.00
0.00
0.00
0..0
Pengi
7.94
0.05
0.89
10.89
6.43
0.45
0.00
0.00
0.00
0.00
0.00
0.00
SALMONN-7B
4.38
9.57
6.03
30.93
81.93
0.78
1.40
0.61
6.60
0.00
22.20
7.20
SALMONN-13B
4.86
10.76
6.45
30.78
84.67
1.56
0.00
0.00
0.00
0.00
15.60
0.00
WavLLM
18.95
0.46
1.64
11.32
5.46
0.69
0.00
0.00
0.00
0.00
5.40
0.00
ImageBind-LLL
13.27
0.21
1.23
5.78
3.26
0.83
0.00
0.00
0.00
0.00
3.40
0.00
Unified-in-2-XXI.
0.00
0.00
0.00
0.00
0.00
23.50
34,60
44.30
5.60
23.50
0..00
VodaVerse-7b-v0
4.78
36.48
28.67
0.41
0.00
40.80
0.00
15.60
0.00
AudioGPT-GPT4
40.23
33.16
28.04
39.05
84.34
3.15
85.90
84,53
84.70
86.50
66.40
29.70
SpecehGPT-7B--cm
8.94
0.37
1.56
15.78
9.29
0.58
0.00
0.00
0.00
0.00
0.00
0.00
LLaMA-Oma
3.89
9.90
31.12
42.10
68.40
5.34
37.50
35,80
43.81
60.30
25.4
34.50
3D-LLM
2.08
24.46
2.85
59.26
65.27
47.00
33.95
0.40
0.00
51.00
53.60
PointLLM-7B
27.05
27.77
1.32
31.27
79.90
00
0.00
0.00
0.00
0.00
0.00
LM-113
28.84
28.58
1.01
35.02
78.19
00
0.00
0.00
0.00
0.00
0.00
0.00
3D-VisTA
0.00
0.30
0.00
1.31
1.45
0.00
0.00
0.00
0.00
0.00
0.00
MotionGPTTTS
0.00
0.08
0.86
5.92
11.11
0.00
0.00
0.00
0.00
0.00
0.00
MotionGPT-LLaME
19.65
7.15
20.58
14.94
9.31
11.06
0.00
0.00
0.00
0.00
0.00
0.00
AvatarGPT
1.65
12.21
0.09
8.15
7.88
11.11
0.00
0.00
0.00
0.00
0.00
0.00
LLaMA A-Meshh
30.73
32.20
31.62
41.21
85.18
11.41
0.00
0.00
0.00
0.00
0.00
0.00
104

---

On Path to Multimodal Generalist: General-Level and General- Bench
Table 81:
Results on NLP Group, #L-17, 17, part A.
#1.-17
Model
(Aflect Computing)
#11
#21
#31
#41
#51
#61
#81
SoTA Specialist
93.40
95.80
97.33
95.80
96.80
97.40
85.93
95.80
84.71
Meta-Llama-3.I
60.40
53.40
79.60
83.60
75.20
51.80
47.60
50.66
80.20
Qwen2.5-7B-Instteee
53.20
92.40
79.80
77.40
57.40
27.31
86,40
22.73
Gemma-2-9b-i
56.80
68.60
89.40
77.80
74.60
55,40
58.92
88.40
41.41
ChatGI.M-Gb
3.00
0.00
83.80
0.00
60.20
0.00
30.70
22.20
19.86
0.00
0.00
0.00
0.00
0.00
0.00
16.80
0.00
0,30
Internl.e M-Chatt-h
49.00
59.40
59.00
19.60
63.00
1.80
30.76
49.00
17.25
GPT-J-6B
0.00
0.00
0.00
0.00
0.00
0.36
Falcon3-7B-Insteee
53.60
65.80
90.60
87.00
72.60
60.00
43.21
85.20
23.67
Baichuan2-7BB 2B-eee
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.18
Ministral-8R. -n-aa-88tttntttt2441111
53.00
60.20
83.20
73.00
60.00
52.40
42.80
84,40
19.50
Yi-Ligntning
46.60
73.60
94.20
85.20
78.20
59.00
48.38
88.40
14,00
GPT-3.5-turb
53.80
65.80
92.60
83.40
69.60
52.20
46.65
89.60
41.10
GPT-4v
62.00
68.40
94.00
92.00
76.60
59.40
57.19
88.40
53,92
GPT-4o
59.60
80.20
92.20
92.00
80.00
60.40
56.09
85.60
58.61)
GPT4o-mini
53.60
60
91.60
81.40
75.00
56.80
87.40
GPT-40-4096
58.60
80.60
85.20
90.60
79.20
60.40
54.04
85.20
40.77
ChatGPT-4o-latest
57.40
80.20
92.20
91.00
78.00
59.40
57.20
87.00
$5.85
Claude-3.5-Sonnel
65.61
51.42
67.78
72.64
60.45
51.51
54.36
71.67
39.57
Claude-3.5-Opus
30.74
30.74
30.74
30.74
30.74
30.74
30.74
30.74
30.74
Emu2-32B
53.44
47.58
57.62
63.95
53.24
39.91
42.49
60.45
25.76
DetGPT
48.97
42.55
48.73
58.75
45.39
37.59
40.39
60.73
25.05
IntemVLL..58B
92.00
45.70
55.60
75.25
78.60
41.99
56.25
86.80
35.21
V.255
92.60
42.99
41.60
82.41
67.20
43.79
53.89
87.59
30.10
NEXT-GPT-VI.5
0.00
15.34
0.00
21.34
18.59
13.29
15.78
0.00
0.30
L.te-mVL.2.52BB
44.80
34.40
43.20
85.27
39.00
43.20
40.14
81.00
24.36
Monkey-10B-one
45.40
25.40
5.40
37.21
49.60
0.60
39.34
0.60
6.73
DeepSeck-VI--7B
87.00
46.00
66.80
59.91
48.60
46.60
51.86
86,40
24.87
Twen2-VL-Th-
56.00
53.00
27.00
59.00
41.00
27.00
6.55
Owcn-VL--Cha
0.00
0.00
0.00
0.00
0.00
0.00
14.82
0.00
8.01
Qwen-Audio- Chat
46.60
45.80
68.30
47.50
74.50
31.90
42.50
83.40
19.01
Qwen2-AuudooInstrruct
48.20
70.00
47.80
73.20
32.20
40.89
84.20
19.89
49,80
56,00
92.20
80,00
92.20
47,40
40.25
92.20
21.41
mPLUUG-OW2-LLaMA22-7
53.80
55.60
84.80
54.80
68.80
43.40
31.38
84,80
7.98
Phi-3.5-Vision-nnttee
57.60
50.60
88.60
86.20
60.20
58.20
59.53
88.60
24.95
Cambrian-1-8B
45.00
42.40
0,00
63.40
3,00
53.60
22.87
0.00
23.69
MiniGPT4-LLMMA2
0.00
0.00
0.00
0.00
18.67
0.00
Intern nttem\L-Chht-V115
82.60
39.80
61.00
77.70
66.20
38.80
52.50
65.40
38.67
BiiilntemmVL-Chu-44B--1
69.00
47.00
54.00
62.78
61.80
39.00
44.62
81.80
27.36
BeenLLM-XComposee/ 1.-1-.
0.00
40.20
20.40
75 66
24.20
1.60
28.34
67.20
24.27
GPT4RoI
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
GLaMM
0.00
0.00
0.00
32.31
0.00
0.00
0.00
LLaVA-NeXT-13B
47.20
51.60
81.60
60,20
62.20
53.40
34.65
82.80
27.98
LLaVA-NoXT-34B
56.80
52.20
82.20
63.40
65.50
56.80
37.59
83.20
31.87
Pixtral-12B
53.40
57.20
88.80
67.40
67.30
57.80
42.07
85.40
LEED-LLLMA-138
27.80
35.40
57.40
16.75
39.60
26.20
22.53
44.60
17.43
BLIP2
51.60
53.60
84.40
64.80
67.20
51.80
29.43
83.00
18.33
MiniMonkey
57.80
54.80
90.60
61.60
64.60
54.40
37.92
83.80
22.22
DeepSeck-VL.. -L-7B
0.00
0.00
13.60
0.00
0.00
0.00
17.03
0.00
3.60
LISA-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Cog VLg-Chhh
59,80
45.53
57.04
68.49
52.96
44.74
47.84
61.99
32.61
ShareGPTAV-77
51.16
39.94
51.54
62.69
45.67
40.59
36.31
55.02
25.49
ShareGPTAV-133B
52.77
50.41
65.68
52.99
37.92
42.55
58.30
30.48
105

---

On Path to Multimodal Generalist: General-Level and General-Bence
#L-117
Model
Computiing
#11
#21
#31
#41
#51
#6T
#71
#81
#91
BLIP-3 (XGen-MM)
54.88
50.33
51.44
68.68
52.94
41.51
44.27
59.55
30.82
AnyGPT
19.32
0.00
20.85
22.88
20.28
16.73
17.69
23.17
10.67
MiniCPM3-4B
56.27
45. 12
53.29
62.79
50.98
43.43
47.19
59.28
31.01
LaVIT-V2 (7B)
48.87
34.25
41.14
50.13
41.12
33.80
34.92
51.52
23.71
GLM-VL-Chi
56.44
45.58
55.63
60.20
53.13
42.98
57.31
27.52
Gemimi- 1.5-Pro
59.00
78.40
92.40
89.20
83.40
57.60
56.09
88.20
51.21
Gemini- 1.5tiii-1.--PBe 1.5-Flash
53.40
74.60
88.20
85 65.60
75.60
52.20
52.11
84,20
38.73
DMG-LLLaA-IItemm..MM.
22.40
20.60
26.40
28.20
49.60
24.40
38.46
22.60
Idefics3- 8B 8B-Llama3
53.60
68.
89.00
83,40
76.60
51.20
53.65
82.00
39.89
Yi- Yi-Vision-v2
50.00
64.40
89.00
82.00
48.40
54.60
47.46
87.40
30.27
Qwen2-VL-722
59.33
78.66
96.00
90.00
79.33
56.67
67.40
82.00
53.80
Otter
0.00
0.00
0,00
0.00
0.00
0.00
18.40
0.00
0.80
Show-o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NExT-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.60
0.00
0.00
IntemVVL-226)
58.60
61.80
90.20
80.60
66.10
54,20
38.90
76.40
44,00
Qwen2-V VL-72B
55.40
77.60
94.40
90.80
77.60
58.40
85.20
DeepSeck-VL-224 VL-2--mmal
51.20
49.00
91.00
49.00
52.80
44.40
35.60
88.60
18.60
DeepSeck-VL-2
54.40
49.80
88.80
77.00
66.60
50.20
41.10
88.20
31.70
LLaVA-Ons Vision-7B
49.60
49.20
86.60
60.66 60
61.40
45.20
47.70
88.20
24.40
LLaVA-One-Vision: 722
33.20
22.40
91.20
49.40
77.80
0.00
57.10
19.80
38.90
Sa2VA-8B
57.80
92.00
79.20
74.00
56.00
52.70
82.00
Sa2VA-26B
52.40
77.40
93.60
86.60
78.80
57.40
40.00
86.20
14.00
CoLVA-2B
50.80
56.00
87.00
72.80
44.80
51.40
48.70
82.40
14.90
CoI.VA-BB
50.60
56.00
90.00
56.00
76.60
51.60
36.40
83.60
25.70
LaVV
56.80
49.40
88.20
53.80
42.20
55.20
37.90
66.20
16.30
LM4LV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-V V1
0.00
24.57
0.00
21.34
23.47
15.94
25.85
0.32
0,30
PandaGPT (13B)
5.70
0.40
0.00
0.00
2.60
0.80
0.00
0.00
0.10
AnyGPT
0.00
20.85
22.88
20.28
16.73
17.69
23.17
10.67
GAMA
3.70
0.40
0.00
0.00
0.00
0.80
15.78
0.00
0.30
Pengi
0.00
0.10
0.00
0.00
1.10
0.20
0.00
0.00
0.10
SALMONN-7B
10.00
3.80
1.20
0.80
3.20
2.20
21.42
3.00
12.13
SALMONN-13B
12.40
0.40
0,00
0.00
2.60
0.80
23.45
3,40
13.04
WavL.MM
3.60
0.00
0.00
0.00
2.30
0.00
16.12
0.00
0.30
ImageBind-Iee -LM
5.60
0.40
0.00
0.00
2.60
0.80
0.00
0.00
0.10
Unified-io-2-Xe 2-XXX.
5.40
64.10
56.30
45.90
0.00
0.00
0.00
0.00
0.00
ModaVerse-7bbb.
5.70
0.40
0.00
0.80
0.00
AudioGPT-GPTA
65.90
69.50
93.60
89.60
75.40
53.51
56.14
86.49
$4.37
SpeechGPT:7B-com
0.00
0.00
0.00
0.00
0.00
0.00
15.78
0.00
0.30
LLaMA-Omui
43,50
35.70
19.70
13.60
15.20
11.80
17.40
20.66
20.20
3D-LLM
0.40
0.20
84.20
17.40
55.60
0.00
2.95
52.80
16.49
PointLLM-7B
0.00
0.00
0.00
0.00
0.00
0.30
PoinILLM-133
0.00
0.00
0.00
0.00
0.00
0.00
16.82
0.00
0.30
3D-VisTA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MotionGPT-TS
0.00
0.00
0.00
0.00
0.00
0.00
NotionGPT-LLAME
0.00
0.00
0.00
0.00
0.00
0.00
0.00
7.66
0.00
AvatarGPT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.27
0.00
LLaMA- Mesh
0.00
0.00
0.00
0.00
0.00
0.00
0.00
15.97
0.00
106

---

On Path to Multimodal Generalist: General-Level and General-Bene Bench
Table 83:
Results on NLP Group, #L-17, part B.
#L.-17
Model
(Aflect Computing)
#101
#111
#121
#131
#141
#15
#161
#171
#181
SoTA Specialist
78.40
62.22
77.51
90.32
91.55
92.98
36.80
95.57
80.20
Meta-Llama-3.
18.33
39.00
35.30
31.53
0.00
29.25
1.10
0.65
2.12
Qwen2.5-7B-Instteee
59.40
46.59
49.22
0.00
44.98
0.72
75.60
Gemma-2-9b-i
69.80
50.29
54.08
0.00
50.73
4.51
4.07
64,40
1.89
ChatGI.M-Gb
7.00
0.27
0.27
0.00
0.14
0.00
68.60
0.00
0.00
0.08
0.00
0.09
0.00
0.02
0.00
0.00
0.00
Internl.e M-Chatt-h
2.20
0.88
0.51
0.00
0.69
0.00
0.00
63.20
0.00
GPT-J-6B
0.00
0.06
0.07
0.02
0.00
Falcon3-7B-Instreen
24.60
31.51
31.91
0.00
30.03
0.00
3.02
68.20
0.46
Baichuan2-7BB -B-Base
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Ministral-8R. B--aa-8--ttnnntt24411111
14.00
12.41
17.72
0.00
27.96
0.12
0.22
67.40
Yi-Ligntning
56.60
6.69
15,54
0.00
43.75
0.25
5,96
1.02
67.20
GPT-3.5-turb
47.60
44.48
40.65
0.00
51.93
0.46
5.08
70.00
1.29
GPT-4v
67.80
42.53
50.08
0.00
68.
0.90
12.11
73.00
2.35
GPT-4o
73.40
51.79
58.21
0.00
63.77
5.70
11.35
77.40
3.56
GPT4o-mini
64.20
52.58
$7.97
0.00
61.64
2.32
72.60
1.40
GPT-40-4096
72.80
52.57
58.51
0.00
69.79
6.23
10.55
74.00
3.51
ChatGPT-4o-latest
71.00
51.79
58.88
0.00
66.71
3.71
10.62
74.90
2.21
Claude-3.5-Sonnel
44.31
30.21
33.02
24.35
44.93
26.17
10.88
50.31
35.37
Claude-3.5-Opus
30.74
30.74
30.74
30.74
30.74
30.74
30.74
30.74
30.74
Emu2-32B
35.85
19.48
25.50
11.72
40.02
14.10
5.42
39.99
23.62
DetGPT
31.89
14.01
18.71
9.20
33.16
9.58
2.75
34.90
18.84
IntemVLL..58B
58.20
34.88
43.77
86.80
50.65
86.80
1.12
0.24
71.00
48.40
38.29
35.97
0.00
33.47
1.64
0.42
72.60
NEXT-GPT-VI.5
0.00
8,08
5.09
16.42
0,00
3.02
14.37
34,70
0.00
Inte-mVL.2.52BB
37.40
15.14
12.96
0.00
10.30
0.20
0.00
0.11
72.40
Monkey-10B-on
0.00
0.24
0.00
0.60
0.00
0.00
0.00
0.00
1.80
DeepSeck-VL--7B
40.80
3.95
5.12
86.40
36.00
86.40
0.15
0.15
66.60
Twen2-VL-Th-
12.00
0.00
0.00
26.00
0.00
Owcn-VL--Cha
0.00
0.19
0.24
0.06
0.00
0.02
0.00
0.00
0.00
Qwen-Audio- Chat
4.70
0.45
0.56
0.00
3.56
0.00
0.34
60.13
0.01
Qwen2-AuudooInstrruct
4.80
0.58
0.68
0.00
0.00
60.60
7.00
0.38
1.15
0.00
39.60
0.00
0.00
92.20
0.00
mPL.UG-OWD-LLLMAA2-77
37.20
1.82
0.25
0.00
76.40
0.00
0.00
84,80
0.00
Phi-3.5-Vision-Instned
57.00
22.39
20.94
0.00
89.20
0.08
0.89
88.60
0.20
Cambrian-1-8B
6.20
13.05
4.13
0.00
0.20
0.14
0.91
0.00
0.18
MiniGPT4-LLMMA2
0.00
4.27
5.38
0.06
0.00
0.02
0.00
Intern nttem/L-ChhttV1155
25.40
27.88
32.06
0.00
36.49
0.00
0.00
0.18
69.00
BaiilntemmVL-Chu-44B--4
41.80
7.72
16.57
0.00
29.42
0.00
0.00
0.00
61.00
InternI Xww.LLMMXXOomosee22-2 -LBB.8B
12.80
0.00
0.00
0.00
0.00
0.00
0.00
0.00
60.20
GPT4RoI
0.00
0.00
0,00
0.00
10.77
0.00
0.00
0.00
0.00
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-13B
40.20
16.07
22.60
0.00
17.83
0.00
0.51
68.40
0.43
LLaVA-NoXT-34B
52.60
22.65
24.35
0.00
14.75
0.12
0.77
72.40
0.67
Pixtral-12B
51.40
32.78
17.4
0.00
26.12
0.06
1.16
70.80
LEED-LLLMA-138
32.20
4.51
2.17
0.00
1.78
0.00
0.00
30.40
0.00
BLIP2
31.60
40.94
24.82
0.00
29.08
0.00
0.88
67.40
0.57
MiniMonkey
37.00
0.25
0.88
0.00
0.21
0.00
0.00
66.40
0.00
DeepSeck-VL.. VL-7B
0.20
0.18
0,00
0.00
1.80
0.00
0.00
0.00
0.20
LISA-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Cog Cog-Chhhhat
36.30
25.60
27.64
12.07
40.89
13.97
7.55
35.65
25.95
ShareGPTAV-77
29.86
15.82
19.33
11.63
33.60
13.18
5.58
32.50
16.51
ShareGPTAV-133B
32.91
23.36
19.63
11.00
40.88
13.60
35.32
20.79
107

---

On Path to Multimodal Generalist: General-Level and General-Bence
#L-117
Model
(Affect Computting
#101
#111
#121
#131
#141
#157
#161
#171
#181
BLIP-3 (XGen-MM)
39.56
20.57
19.83
14.77
38.76
17.31
5.44
37.51
19.70
AnyGPT
14.45
8.84
8.39
4.93
15.69
6.82
3.98
15,07
10.04
MiniCPM3-4B
39.65
26.91
28.14
22.07
37.25
15.44
10.46
31.81
28.96
LaVIT-V2 (7B)
27.53
20.62
20.58
13.19
31.68
13.61
8.48
20.76
28.42
GLM-VL-Chi
39.69
26.90
26.00
20.48
37.62
12.36
28.40
Gemimi- 1.5-Pro
67.20
45.61
47.57
2.02
51.96
4.78
6.17
75.20
0.60
Gemini- 1.5tiii-1.--PBe 1.5-Flash
58,40
35.49
38.14
1.24
43.26
2.39
4.08
69.60
0.40
DMG-LLL.AA-mtemL..MM..
9.60
1.42
0.88
0.00
1.10
0.00
20.20
Idefics3- 8B 8B-Llama3
42.20
28.71
28.65
2.28
33.57
2.54
2.25
63.40
0.00
Yi- Yi-Vision-v2
38.60
32.69
32.36
0.00
37.49
0.24
3.18
71.43
0.47
Owen2-VL-728
69.34
55.78
61.37
0.00
78.03
2.49
10.18
75.81
2.59
Otter
0.00
0.00
0,00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
NExT-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
33.67
0.00
IntemVLL-2261
48.80
34.10
35.40
0.00
38.90
0.00
0.00
63.40
0.00
Qwen2-V VL-72B
73.20
55.30
60.60
0.10
0.30
76.60
DeepSeck-VL-2245 VL-22-mmal
41.60
11.50
13.30
0.00
27.80
0.10
1.10
67.00
0.10
DeepSeck-VL-2
34.20
26.10
32.00
0.00
26.60
0.00
0.00
69.60
0.40
LLaVA-Ons Vision-7B
57.00
14,50
10.70
0.00
16.60
0.30
0.90
65.80
0.20
LLaVA-One-Vision- 72s
0.20
53.90
50.40
0.00
58.60
2.50
11.30
60.20
1.50
Sa2VA-8B
60.60
27.20
33.10
0.00
0.70
69.80
Sa2VA-26B
60.80
37.40
36.00
0.00
36.50
0.20
5.40
73.60
1.30
CoLVA-2B
23.80
0.50
2.90
0.00
21.20
0.00
0.00
62.00
0.00
CoI.VA-BB
22.00
28.90
29.50
0.00
34.50
1.70
1.30
75.60
0.30
LaVV
42.60
1.50
0,40
0.00
1.00
0.00
0,00
54,00
0.00
LM4LV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-V V1
0.00
8.12
7.24
13.57
0.00
3.01
4.79
35.62
0.00
PandaGPT (13B)
0.00
0.04
0.00
0.09
0.00
0.01
0.00
0.00
0.00
AnyGPT
8.84
8.39
4.93
15.69
3.98
10.04
GAMA
0.00
0.08
0.00
0.09
0.00
0.02
0.00
34,70
0.00
Pengi
0.00
0.04
0.00
0.05
0.00
0.01
0.00
23.50
0.00
SALMONN-7B
1.20
0.00
0.00
0.00
1.17
0.00
5.60
3.40
SALMONN-13B
1.40
0.04
0.00
0.09
1.19
0.01
0.00
5,90
0.00
WavL.MM
0.00
0.07
0.00
0.06
0.00
0.02
33.78
0.00
ImageBind-Iee -LM
0.00
0.04
0.00
0.09
0.00
0.01
0.00
0.00
0.00
Unified-io-2-XC
0.00
0.00
0.00
0.00
4.67
0.00
0.00
0.00
0.00
Modaaeerse-bb-v
0.00
0.04
0.09
0.01
0.00
AudioGPT-GPTA
66.40
41.05
51.08
0.00
67.96
0.80
12.01
72.00
2.31
SpeechGPT:7B-com
0.00
0.08
0.00
0.09
0.00
0.02
0.00
34,70
0.00
LLaMA-Omui
16.40
18.50
5.40
1,56
0.00
29.25
1.10
0.33
2.02
3D-LLM
5.80
0.00
0.00
0.00
0.00
0.00
0.00
65.60
0.00
PointLLM-7B
0.00
0.09
0.10
0.03
0.00
PoinILLM-133
0.00
0.09
0.00
0.10
0.00
0.03
0.00
0.00
0.00
3D-VisTA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MotionGPT-TS
0.00
0.00
0.00
0.00
0.00
0.00
NotionGPT-LLAME
0.10
0.00
0.00
0.00
0.00
0.00
0.05
0.00
0.00
AvatarGPT
0.10
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaMA- Mesh
6.12
0.00
1.62
0.89
0.07
2.21
0.05
0.05
0.00
108

---

On Path to Multimodal Generalist: General-Level and General-Bec Berral
Table 85:
Results NLP Group, #L-18.
#1.188
Model
(NER)
#1
#21
#31
#41
#51
#61
#71
#81
#91
#107
#111
#121
#131
SoTA Specialist
83.51
74.45
67.00
76.02
92.00
88.08
94.21
97.65
96.74
81.97
78.21
89.18 18
82.32
1-8B-Instruct
6.81
17.76
15.53
5.86
14.09
18.77
25.43 43
30.05
51.57
72.00
12.70
31,22
16.62
Owen2.5-7B-Instmud
40.07
25.67
27.54
26.97
36.38
56.83
79.95
45.58
17.98
40.24
23.05
18.58
Gemma-2-9ba
43.03
32.32
26.62
38.48
33.81
39.95
73.65
82.90
78.18
18.78
43.01
28.66
34,53
ChatGL.M-bb
0.05
0.13
0.00
0.01
0.00
0.18
0.00
0.00
0.00
0.39
0.57
0.15
Vicuma-7b-v1.55
0.04
0.00
0.02
0.00
0.00
0.56
0.00
0.17
0.00
0.00
0.00
0.03
0.13
Internl Mornn.MMCC M-Chat-7b
1.65
0.12
0.47
1.20
1.37
0.72
0.19
13.63
0.00
0.52
12.40
1.31
0.78
GPT-J-6B
0.00
0.52
0.00
0.00
0.00
0.00
0,00
0.03
Falcon3-7B-Instruc
30.52
15.97
14,21
22.00
7.11
13.81
30.45
78.19
48.84
11,30
41.06
22.58
23.29
Baichuan2-7B: BB--Be
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
6.46
6.20
0.52
4.57
5.11
1.75
2.11
29.85
3.11
3.36
8.65
4.90
Yi-Ligntning
19.60
38.11
24.46
48.11 11
25.09
25.29
25.00
31.84
44,80
37.15
1.81
42.17
4.14
GPT-3.5-turbo
40.48
41.50
9.12
21.80
26.01
28.93
0.91
59.32
2.34
14.68
39.67
24.06
27.84
GPT-4v
47.34
57.67
29.84
50.56
70.18
49.48
61.78
96.06
87.41
23.94
50.49
41.81
37.24
GPT-4o
60.84
48.43
36,26
50.89
74.90
50.71
82.38
94.89
85.62
29.12
49.34
46.08
37.00
GPT4o-mini
53.21
33.52
42.35
40.03
48.17
78.55
95.78
93.14
27.61
42.31
37.98
35.95
GPT-4o-4096
60.07
51.72
34,95
50.73
72.61
47.95
81.52
96.30
77.38
28.60
49.36
45.90
36.41
ChatGPT-4o-latstt
55.99
46.35
35.81
50.06
72.10
47.39
76.72
95.28
56.99
28.39
49.87
45.55
33.16
Claude-3.5-Sonnet
28.75
21.09
20.31
29.93
24.02
25.56
30.27
41.19
36.19
19.92
23.67
34.53
26.76
Claude-3.5-O0
30.74
30.74
30.74
30.74
30.74
30.74
30.74
30.74
30.74
30.74
30.74
30.74
30.74
Emu2-32B
16.88
14.72
14.19
21.02
17.85
16.18
24.75
32.26
28.44
13.39
18.51
23.37
16.32
DetGPT
10.75
15.19
9.71
12.47
15.07
10.30
15.66
29.04
23.02
6.23
16.82
18.50
13.25
I2255VV2..58B
7.37
30.33
17.51
33.97
7.54
15.29
9.72
21.90
33.78
13.19
13.52
73.71
9.71
V12.-mV
8.71
28.29
18.36
26.35
17.75
13.19
6.16
23.17
17.07
14.97
58.62
78.65
55.51
NEXT-GPT-VI.5
9.04
0,00
0.02
0.00
0.00
0.07
5.85
0.17
8.53
12.46
8.45
3.03
10.13
I.2e-2V
0.65
1.36
2.09
9.17
1.21
1.13
2.03
2.91
1.27
0.52
0.47
9.23
1.74
Monkey-10B--n 10B-chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.20
0.00
PrepSeck-VL-78
0.25
0.81
1.86
5.42
1.05
0.65
1.34
2.07
4.07
1.51
2.32
30.05
2.19
Qwen2- VL-7B
0.00
0,42
0.00
1.08
0.00
0.00
0.35
0.00
0.65
Qwen-VL-Chan VL-Chat
1.00
0.27
0.11
0.22
0.47
0.70
1.78
2.19
0.09
0.09
0.83
0.08
0.81
Qwen-Audio-Cae Chat
1.58
4.47
1.03
2.65
0.45
2.56
10.32
7.60
16.78
0.13
12.45
2.89
1.45
Qwen2-Audio-Instruec
1.59
4.41
1.22
2.92
0.62
3.35
11.62
8.10
18.72
0.27
13.19
2.97
MoE-LLAVA. 2.E-LLAVV--hii222....
Phi2-2.7B-4e
1.13
1,42
1.70
1.96
0.80
2.00
12.35
25.27
10.40
0.23
2.87
0.36
1.83
mPLUUG-OW2-LLaM22-7
0.08
0.10
0.04
1.72
0.20
0.56
2.38
11.95
0.87
0.32
2.54
0.17
0.27
Phi-3.5- -is--Viision-Instuet
5.41
3.86
0.41
3.73
0.90
3.31
23.37
61.43
29.48
1.73
21.49
6.07
5.61
Cumbrian-1-SB
0.30
4.95
0.55
2.45
1.30
2.56
0.00
2.44
0.00
0.00
3.53
1.27
0.32
MiniGPT+-LLMM2
8.34
1.54
12.56
3.49
7.42
15.11
7.58
3.27
8.67
Intem MntemVL-Chat-V1--
6.38
29.98
15.53
31.03
8.93
13.62
4.20
22.11
14.35
7.80
12.32
70.80
14.85
Mini-Intem www.-niimm---Cha--BB----
3.11
16.73
5.84
20.87
4.56
10.52
5.20
15.31
5,30
9.81
32.76
61.38
26.63
InternL XXC.ooo-oo -L-I.8B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GPT4Ro1
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-13B
5.01
5.43
2.28
3.34
0.68
2.59
22.10
52.72
31.21
0.22
3.88
1.34
3.08
LLAVA-NeXT-34B
7.45
7.92
5.13
4.98
1.75
3.68
26.94
61.26
32.22
0.74
8.85
2.34
2.42
Pixtral-12B
8.23
4.74
13.72
0.69
4.74
25.01
63.06
34.76
0.82
5.72
4.76
SEED-LI SEED-LLaMA-135 LLaMA-13B
1.74
0.65
0.66
2.85
0.10
0.57
4.82
16.58
4.76
0.03
0.42
0.18
0.25
BLIP2
1.42
3.49
0.45
16.51
0.00
2.17
4.56
35.70
19.64
0.00
7.83
0.18
0.31
MiniMonkey
0.32
0.79
0.21
2.78
0.00
0.38
5.74
27.81
25.35
0.21
3.46
0.22
0.32
DeepSeck-VL. VL-7B
0.00
0.05
0.09
0.12
0.05
0.07
0.00
0.00
0.00
0.27
1.14
0.61
0.44
LISA-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-CHE
17.28
21.47
9.66
19.91
20.38
14.37
25.75
38.26
26.80
13,30
22.25
23.08
20.29
ShareGPT4V-7B
10.53
14.50
5.39
14.02
14.62
12.19
14.38
31.20
21.91
4.94
15.69
18.11
8.88
Shar-GGT4V-133
17.46
17.27
17.92
15.32
14.32
17.51
31.09
23.35
11.16
20.00
20.46
16.81
109

---

On Path to Multimodal Generalist: General-Level and General-Bene Bench
#L-18
Model
(NFR)
#11
#21
#31
#41
#51
#61
#71
#81
#91
#107
#111
#121
#131
BLIP-3 (XGen-MM)
18.38
15.56
13.15
16.92
16.46
13.65
18.39
33.99
29.00
9.63
18.34
22.41
12.03
AnyGPT
7.82
8.57
5.26
0.00
6.02
5.71
10.10
10.93
10.35
6.15
9.34
9.06
7.64
MiniCPM3-4B
17.83
15.66
14.07
19.69
16.49
12.26
26.47
29.23
26.81
10.83
19.29
31.37
17.12
LaVIT-V2 (7B)
11.16
14.05
9.28
11.98
9.97
11.13
11.15
23.96
15.05
9.25
18.54
26.57
17.22
GLM-VL- VL-Chat
16.49
11.54
17.76
13.38
10.06
23.00
28.73
24.81
8.91
20.49
31.68
16.10
Gemini-1.5-PP
48.53
50.66
29.19
43.31
49.74
46.51
67.45
90.12
82.53
19.62
47.10
42.75
36.80
Demini-1.55-Flash
41.72
42.52
27.37
37.98
40.29
42.16
60.03
82,95
79.40
16.23
43.91
38.49
32.13
OMG-LLaVA --LLLLLa--mmmmLLM220B8
4.55
2.20
4.43
2.26
3.80
6.82
11.64
13.71
0.00
3.47
2.05
Srefies3-8B-Llanse
27.46
32.75
21.06
28.65
26.51
25.56
45.38
73.20
66.23
14.18
19.59
25.97
26.54
Yi- Yi-Vision-v2
25.61
19.34
5.84
22.92
21.08
7.48
38.30
76.28
58.70
13.61
35.33
17.81
23.02
Qwen2-VL-72 VL-72B
50.59
56.57
23.52
40.31
48.31
40.65
85.92
93.81
87.47
18.33
52.48
43.63
35.58
Otter
0.00
0.00
0.00
0.00
0.00
0.18
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
0.00
0.00
0.00
0.00
0.00
0.00
NExT-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
IntemVL.2-26B
29.30
19.90
0.60
25.20
20.70
15.00
0.90
73.30
21.70
0.70
34.50
13.10
17.30
Qwen2- VL-72B
47.00
52.20
26.20
43.10
54.20
45.10
84.90
94.20
84,90
19.00
49.20
40.00
33.20
DeepSeek VL-2-small
5,50
12.40
0.10
4,40
3.50
1,20
15.70
55.70
12.90
3.30
16.10
2.60
3.40
DeepSeck-VL. VL-2
20.90
14.60
1.70
15.50
10.30
3.80
13.70
80.60
15.30
3.70
21.80
3.70
1.70
LLaVA-One- Sex--On--Vision777 Vision-7B
3.10
3.80
2,70
11.90
5.40
7.30
0.80
2.70
0.10
1.00
11.60
1.40
2.70
LLAVA-One- S0AA-On--Visio--22
44.40
32.70
11.90
35.00
62.00
11,10
71.30
91.00
81,10
19.20
32.60
37.10
28.80
Sa2VA-8B
29.50
22.70
38.60
16.90
24.50
73.50
0.00
7.70
45.70
Sa2VA-26B
32.20
21.40
11,30
25.20
6.70
22.90
60.00
80.50
4.40
10.80
42.20
15.30
18.30
CoLVA-2B
3.00
1.20
0.40
0.70
2.80
0.80
5.20
17.60
0.00
0.20
2.20
0.50
0.10
VoI-VA-4B
29.50
6.80
2.70
14.40
18.50
12.70
65.60
77.50
58.80
4.40
23.90
11.30
Long-LLaVa
0.10
0.20
0.10
1.20
0.10
0,60
1.10
26.20
0.10
0.00
0,40
0.00
0.20
LM4LV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-V V1
10.03
0.00
0.02
0.00
0.00
0.07
9.08
0.17
10.30
10.35
16.74
5.03
8.19
PandaGPT (13B)
0.04
0.00
0.02
0.00
0.00
0.56
0.00
0.17
0.00
0.00
0.00
0.03
0.13
AnyGPT
8.57
6.02
5.71
10.10
10.93
10.35
6.15
9.54
9,06
10.00
GAMA
0.04
0.00
0.02
0.00
0.00
0.58
5.93
0.17
8.05
0.00
0.00
0.03
0.13
Pengi
0.04
0.00
0.02
0.00
0.00
0.13
0.00
0.15
0.00
0.00
0.00
0.01
0.08
SALMONN-7E
0.14
0.00
0.11
0.00
0.21
1.67
4.74
0.00
0..00
0.86
0.00
0.31
SALMONN-13B
0.04
0.18
0.02
0.00
0.00
0.56
1.89
5.17
0.00
0.00
0,00
0.03
0.13
WavLL.M
0.03
0.00
0.00
0.07
5.69
0.23
8.23
0.00
0.00
0.03
0.12
ImageBind-LLNN -LM
0.04
0.00
0.02
0.00
0.00
0.56
0.00
0.17
0.00
0.00
0.00
0.03
0.13
Unified-in-2-XXI
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ModaVerse-
0.02
0.56
0.00
0.17
0.00
0.00
0.00
0.03
AudioGPT-GPTT
46.90
56. 18
29. 14
49.32
78.34
45.90
61.78
95.67
85.41
20.56
23.45
40.81
35.23
SpeechGPT-7B-com
0.04
0.00
0.02
0.00
0.00
0.07
5.85
0.17
8.53
0.00
0.00
0.03
0.13
LLaMA-
6.23
14.67
15.40
5.86
12.40
14,30
23.50
28.95
13.45
0.00
0.00
5.22
4.34
3D-LLM
0.00
0.00
0.00
0.00
0.00
0.11
0.00
0.00
0.00
0.16
0.00
0.00
0.24
PointLLM-7B
0.02
0.00
0.56
0.00
0.00
0.00
0.00
PointLLM-113B
0.04
0.00
0.02
0.00
0.00
0.56
0.00
0.00
0.00
0.00
0.00
0.03
0.16
3D-ViSTA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MotionGPTTTSS
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MotionGPT-LLAMA
0.00
0.00
0.00
0.00
0.00
0.00
0.16
0.00
0.00
0,00
0.04
0.00
0.00
AvatarGPT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Lash--Mesh
0.00
1.05
0.19
0.06
1.24
0.57
0.94
0.62
2.49
0.17
0.06
0.68
0.44
110

---

On Path to Multimodal Generalist: General-Level and General-Ben Bench
Table 87:
Results NLP Group, #L-19.
#1.-19
Model
(Cog QA)
#11
#21
#31
#41
#51
#6t
#71
#81
#91
SoTA Specialist
91.87
95.27
83.64
75.57
91.61
65.23
89.20
72.40
88.20
Meta-Llama-3.1-4 -nSttrrrttt
43.70
38.80
70.40
0.00
21.74
7.65
42.00
46.80
5.20
Qwen2.5-7B-Instee
59.80
72.20
29.29
83.00
10.80
28.20
48.40
8.60
Gemma-2-9b-it
64.40
76.20
0.00
39.48
68.60
6.95
43.20
48.20
15.60
ChatGLMM66l
6.40
46.00
0.00
3.52
0.20
1.20
16.80
1.60
Vicuna-7b-v1.5
0.00
0.00
0.00
4.26
0.00
0.00
0.00
0.00
0.00
InternI.M AiterllMM-hhate M-Chat--h
8.20
27.00
0.00
4.94
21.60
1.19
23.00
20.80
5.00
GPT-J-6B
0.00
0.00
0.00
0.00
Falcon3-7B-Instrutt
51.00
70.20
0.00
40.25
74.80
6.97
6.80
47.60
10.40
Baichuan2-7B. B-auannn--BBBe
0.00
0.00
0.00
0.08
0.00
0.00
0.00
0.00
0.00
Siiistra-88--nstruee--222
47.60
62.60
0.00
14.00
63.40
40.00
42.80
6.20
Yi-Ligntning
76.80
77.60
0.00
45.33
72.40
6.19
44.80
49.40
13.00
GPT-3.5-turho
52.00
81,80
0.00
29.30
80.00
8.18
43.00
48.40
11.20
GPT-4v
70.00
86,20
0.00
67.28
77.40
9.78
45.80
47.80
4.80
GPT-4o
83.20
85.90
0.00
63.34
76.80
8.69
45.80
48.80
10.20
GPT4o-mini
69.80
83.00
9.44
43.20
48.80
15.60
GPT-4n-4096
79.60
86,20
0.00
60.19
75.80
9.87
46.20
48.20
5.00
ChatGPT-4ollatsst
80.20
86.80
0.00
59.31
73.80
8.97
44.60
48.40
8.00
Claude-3.5-Sonne
41.60
57.67
6.36
24.33
66.38
11.04
34.29
39.44
11.05
Claude-3.5-Opus
30.74
30.74
30.74
30.74
30.74
30.74
30.74
30.74
30.74
Emu2-32B
30.80
44.61
0.00
20.31
56.97
5.94
22.64
34.82
0.00
DetGPT
23.02
41.98
0.00
16.31
50.67
2.95
15.86
32.02
0.00
IntemmVL2.5-8B
49.40
76.20
0.00
27.01
77.20
6.93
38.00
48.00
5.60
InternV1.2.5-4B
40.00
67.00
0.00
19.77
72.00
30.20
46.80
7.60
NEXT-GPT-VI.5
19.05
4.32
8.59
5.43
2.30
9.44
6.32
0.00
0.00
InternVL.2.522B
19.60
36.00
0.00
11.90
71.00
1.31
3.00
43.40
4.40
Monkey- 10B-chat
3.00
0,80
0.00
3.49
11.20
0.00
1.20
7.40
0.00
DeepSeck-VL-7B
39.00
64.60
0.00
5.27
73.20
4.02
23,00
46.80
0.21
Qwen2-VL-7B VL-7B
0.00
1.00
25.
0.00
Owen-VL-Chat
1.00
0.00
0.00
4.60
0.00
0.12
13.08
26.91
0.08
Qwen-Audio-Chan
13.50
49.78
0.00
5.34
0.80
5.34
6.30
27.50
3.20
Qwen2-AudiooIIstmuet
15.20
50.60
0.00
5.52
1.00
30.00
3.80
MoE-LLAVA-Phiie
1.13
53.60
0.00
3.73
64,00
6.09
4.47
16.47
0.36
mPLUGG-Ow12LLaNAZZ-
0.08
20.60
0.00
2.06
31.40
0.16
8.20
21.21
0.17
Sii-3.--Vision, Instrrcct
5.41
75.20
0.00
4.17
68.80
6.13
9.31
22.83
6.07
Cambrian-1-8B
0.30
5.80
0.00
2.28
83.40
3.67
0.00
16.70
1.27
MinGPTT-LLLMA2
8.34
0.00
0.00
3.96
19.31
3.44
MiemVL--Chat-Vi
53.20
66,80
0.00
32,30
69.80
4.64
19.00
42.20
4.00
Mini-Intemi Homm-Intem-L-Chhaa-ee -L-h------Cha
27.00
50.80
0.00
12.96
66.60
5.46
33.60
48.00
5.20
InternLM- WXw.LLMMXOOomo222-21111
5.00
17.60
0.00
0.00
46.80
0.00
85,20
27.60
2.80
GPT4Rol
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GLaMM
0.00
0.00
0.00
0.00
0.00
0.00
LLaVA-NeXT-L3B
17.77
49.94
0.00
17.63
63.20
5.36
18.80
37.40
4.40
LLaVA-NeXT-34B
21.08
63.60
0.00
33.85
67.40
6.47
17.40
47.60
5.00
Pixtral-12B
23.71
61.74
32.45
72.20
20.20
48.80
5.40
SEED-LLaMA-I3B
1.03
14.22
0.00
7.55
34.60
0.76
7.60
26.20
0.80
BLIP2
23.31
41.16
0.00
8.77
$7.60
2.04
19.80
46.20
1.00
MiniMonkey
16.20
36.60
0.00
9.63
62.20
0.18
16.40
48.60
0.60
PoopSekk-VL-71
0.00
0.20
2,00
2.05
14.20
0.00
0.00
0.00
4,00
LISA-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-Chat
26.85
50.96
0.00
19.45
53.17
6.88
20.10
36.22
4.88
ShareGPT4V-77
26.63
43.07
0.00
15.45
49.86
1.22
17.29
31.63
1.56
ShareGPT4V-133
26.69
42.06
0.00
3.64
33.37
2.05

---

On Path to Multimodal Generalist: General-Level and General-Bende
#L-19
Model
QA)
#11
#21
#31
#47
#51
#61
#71
#81
#91
BLIP-3 (XGen-MM)
31.79
49.11
0.00
17.94
54.09
4.65
20.88
34.96
5.53
AnyGPT
10.00
18.03
3.13
7.06
18.88
3.42
8.66
11.78
0.00
MiniCPM3-4B
27.08
46.04
6.44
17.45
59.27
13.48
24.74
36.41
15.05
LaVIT-V2 (7B)
20.80
39.84
3.73
10.97
41.93
5.39
14.56
26.45
9.32
ELM-VL-Che
25.03
45.62
7.49
13.30
23.83
37.89
15.67
Gemini-1.5 1-5-Po
52.34
85.00
0.00
36.45
75.24
7.30
44.20
46.80
6.00
Gemini-1.5-Flas
48.95
79.53
0.00
32.93
70.45
6.96
40.80
39.40
7.00
OMG-LLaVA-ImleLLMZZ00
5.26
10.14
0.00
3.10
28.71
7.60
9.40
0.00
Idefics3-8B-Llamal
38.09
62.18
0.00
22.93
69.20
5.24
31.00
40.20
4,80
Yi-Vision-v2
53.80
50.03
0.00
12.43
72.20
6.61
42.80
27.25
8.40
Qwen2-VL-722 VL-72B
63.33
71.85
0.00
61.37
77.33
18.97
16.00
58.67
11.33
Otter
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Show-o
0.00
0.00
0.00
0.00
0.00
0.00
NExT-Chat
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
InternVLL-200
60.40
78.00
0.00
31.70
70.80
0.70
41.40
45.20
0.60
Qwen2- Qwen2-VL-72B
66.60
83.40
75.40
43.60
48.20
0.80
DeepSeek- DeepSeck-VL-2-small
23.80
31,00
0.00
7.80
67.20
5.40
10.40
45.60
11.40
DeepSeck-VL-2
33.00
72.80
0.00
1.70
72.00
6.40
44,40
48.60
1.40
LLaVA-One- Mime-Viiion-8
48.80
68.00 08.00
0.00
24.10
74.60
4,20
8.60
45.60
7.80
LLaVA-One- -iis-Onn-Vision-22
60.00
76.00
0.00
62.70
0.00
7.60
44.00
0.20
7.20
Sa2VA-8B
55.60
78.80
13.30
42.80
47.40
1.60
Sa2VA-26B
72.40
76.40
0.00
46.80
75.00
6.40
46.00
45.80
9.80
CoLVA-2B
14.00
7.40
0.00
3.30
42.00
1.20
2.40
22.80
3,00
VA-4B
38.20
66.20
0.00
27.10
75.60
8.80
17.00
47.20
10.60
Long-LLaVA
9.40
$1.60
0.00
18.20
80,00
5.70
2.80
22.00
4,00
L.MIIV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-Vi VI
32.16
14.35
8.59
5.43
3,40
9.54
3.63
0.00
0.00
PandaGPT (13B)
0.00
0.00
0.00
2.76
0.00
0.00
0.00
0.00
0.00
AnyGPT
18.03
3.13
7.06
3.42
4.43
GAMA
0.00
0.00
0.00
4.23
0.00
0.00
0.00
0.00
0.00
Pengi
0.00
0.00
0.00
2.16
0.00
0.00
0.00
0.00
0.00
SALMONN-7B
0.40
0.20
0.00
0.25
0.80
3.40
8.20
SALMONN-13B
0.80
0.40
0.00
0.76
1.10
0.00
5,00
3,60
8.50
WavLLM
0.00
0.00
0.00
4.57
0.00
0.00
0.00
0.00
ImageBind-LLe -LM
0.00
0.00
0.00
2.76
0.00
0.00
0.00
0.00
0.00
Unified-io-2-XXI.
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
ModaVerse-7b-v
0.00
0.00
0.00
AudioGPT-GPT4
71.34
83.45
0.00
56.80
73.20
9.45
42.10
45.35
4.30
SpecchGPT-77B-com
0.00
0.00
0.00
4.65
0.00
0.00
0.00
0.00
0.00
LLaMA-Omai
0.00
0.00
0.00
0.00
20,56
7.85
1.80
4.60
4.60
3D-LLM
3.80
0.00
0.00
7.90
0.40
0.00
0.00
0.00
0.00
PoinLLLM-7B
0.00
0.00
0.00
Poi31.LM-133
0.00
0.00
0.00
4.26
0.00
0.00
0.00
0.00
0.00
3D-VisTA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MotionGPT-TS
0.00
0.00
0.00
0.00
0.00
0.00
MotionGPT-LLAN LLAMA
0.18
0.00
0.00
0.00
1.19
0.00
0.00
0.00
0.00
AvatarGPT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LaMA--Mstt
0.27
0.00
0.00
0.00
4.68
0.00
0.10
0.00
0.00
112

---

On Path to Multimodal Generalist: General-Level and General- -ener
Table 89:
Results NLP Group, from #L-20 #L-22.
#1-20
#1-21
#1-22
Model
(Event Exi)
(Sem Par)
(Ling Par)
#11
#21
#11
#27
#31
#11
#21
#31
SoTA Specialist
62.89
54.32
69.00
89.40
74.80
93.30
91.40
92.43
Meta-Llama-3.
1.36
14.80
34.00
38.00
25.20
57.80
36.80
68.46
Swen2.5-7B-Insteee
3.67
27.80
51.20
57.60
29.40
78.40
49.60
44.49
Demma-2-9bban
18.60
20.20
48.60
47,80
30.20
84,40
55,40
72.27
ChatGL-M6h
0.00
5.40
28.20
20.40
22.80
64.60
50.60
20.91
Souua--b-vv..5
0.00
0.00
26.20
20.40
22.80
24.20
21.20
0.79
Internl M-Chat-7hh
0.02
6.60
35.80
24.00
21.40
54.40
31.00
21.86
GPT-J-6B
26.20
20.40
22.80
24.20
21.20
23.80
Falcon3-7B-Insten
3.77
18.60
42.60
45.20
31.60
79.80
48.00
48.46
Baichuan2-7B5 Baatuann--BBBase
0.00
0.00
6.15
0.00
0.00
0.00
0.00
0.00
0.33
18.40
28.00
24.40
24.20
53.80
34.40
33.12
Yi-Ligntning
21.67
29.00
38.40
20.80
28.60
83.60
67.00
30.88
GPT-3.5-turho
15.30
19.20
0.00
0.00
0.00
0.00
0.00
70.06
GPT-4v
35.72
32.00
32.00
42.99
4.40
0.20
3.80
68.72
GPT-4o
42.03
37.00
37.00
45.60
13.60
1.40
5.40
78.69
GPT4o-mini
25.02
25.00
25.00
53.60
8.80
0.60
6.80
GPT-49-4096
39.20
38.80
60.80
87.20
36.40
86.60
1.00
56.27
ChatGPT-4o-ltest
38.29
32.40
59.60
79.80
25.60
61.40
0.00
41.54
Claude-3.5-Sonnct
14.30
25.37
36.00
43.00
28.60
33.40
30,20
39.97
Claude-3.5-Opus
30.74
30.74
30.74
30.74
30.74
30.74
30.74
30.74
Emu2-32B
10.58
16.97
25.60
28.60
25.20
30.40
27.40
37.35
DetGPT
2.19
10.62
24.40
29.40
18.20
23.20
21.20
28.58
IntemVL2.558B
1.20
18.40
40.59
30.40
25.40
66.20
41.99
$2.82
IntemmVL2..54BB
2.87
16.20
37.80
24.80
25.60
39.40
38.80
34.23
NEXT-GPT-VI.5
4.63
9.80
28.95
21.45
22.10
32.70
19.50
4.37
V.2.52VL2..-BB
0.00
7.80
29.60
32.80
24.60
53.20
26.60
25.63
Monkey- Bockkey---B-chh 10B-chat
0.00
1.40
5.20
20.00
22.40
19.20
10.00
0.00
DeepSeck-VL-7B
0.21
16.00
34.80
36.60
24.00
59.60
39.00
24.30
Qwen2 VL-7B
20.17
23.38
21.00
29.00
15.38
Owen-VL-Chat
0.01
0.64
20.80
20.31
19.63
18.59
21.20
9.12
Qwen-Audio-Che Chat
0.00
4.60
30.40
23.10
22.50
49,60
38.70
20.30
Qwen2-AudiioIIsstruct
0.00
5.00
31.80
24.00
23.60
50.80
39.60
21.11
0.00
9.78
16.40
19,80
21.00
20.30
40.80
23.77
mPL.GG-OW1-LLLMA22-77
0.06
6.52
16.79
18.50
21.01
19.43
25.40
23.83
Phi-3.5-Vision-Innse
0.03
21.83
15.82
20.06
25.37
21.30
33.00
19.64
Cambrian-1-8B
0.00
8.56
12.86
21.60
17.88
20.74
2.40
22.24
MiniGPT4LLLMA2
0.00
0.00
13.40
18.94
20.32
17.55
24.00
22.15
IntemVL-Chhat-i--5
0.00
11.80
32.80
27.60
26.00
61.60
40.00
43.23
Mini-InemmVL-Chat--P
0.00
15.80
33.40
27.60
23.40
55.20
30.00
36.06
InternL MCMM--XXoomoo222-2 VL-1188
0.00
0.00
31.60
34.00
31.60
61.60
38.20
18.72
GPT4Rol
0.00
0,00
1.40
9.00
22.40
6.00
10.20
0.00
GLaMM
0.00
0.00
0.00
0.00
1.40
0.00
0.00
LLaVA-NeXT-13B
0.00
16.20
28.80
25.60
22.20
21.20
29,20
23.45
LLaVA-NeXT-34B
0.00
17.20
30.60
28.60
20.40
32.20
33.40
22.17
Pixtral-12B
24.80
33.80
32.80
26.20
24.60
31.20
0.00
4,20
11.60
17,80
6.80
15.80
14,20
7.23
BLIP2
0.33
5.80
33.60
25.80
21.40
49.80
33.60
29.06
MiniMonkey
0.00
3.60
30.20
26.00
21.80
51.60
31.80
20.80
DeepSeck- VL-7B
0.00
0,00
22.40
20.80
22,60
24.20
26.40
24.79
LISA-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
CogVLM-CHe
13.80
19.88
26.40
34.00
20.00
27.00
24.20
33.50
ShareGPT4V-7B
5.70
13.20
20.40
28.00
20.80
20.60
20.40
27.92
ShareGPT4V-13B
25.20
32.80
19.40
25.20
21.80
32.65
113

---

On Path to Multimodal Generalist: General-Level and General- -eerral-Bench
#L-20
#L-21
fL-22
Model
(Fvent
(Sem
(1ing
#11
#21
#11
#21
#31
#11
#21
#31
BLIP-3 (XGen-MM)
6.27
14.19
26.80
31.60
22.20
25.00
23.40
33.68
AnyGPT
4.43
5.69
9.60
11.20
10.20
11.00
9.80
11.72
MiniCPM3-4B
13.48
21.15
29.80
35.80
28.60
31.80
35.60
41.51
LaVIT-V2 (7B)
7.11
14.27
24.80
24.20
24.60
27.40
26.40
28.87
GLM-VL-Chat
10.77
22.53
29.00
32.60
24.00
29.60
33.20
40.37
Gemini-1.5-Pro
22.16
28.62
48.60
56.40
29.40
83.80
73.20
71.42
Gemini-1.5-Flas
13.45
22.31
43.20
49.20
27.20
82.60
66.20
62.03
C0G-LLLaVA-IntemLLMMM1
2.40
13.60
10.40
6.80
9.00
5.60
11.67
Idefics3-8B-Llamal
13.92
18.00
35.80
43,80
28.20
69.40
68 68.60
58.39
Yi-Vision-v2
7.89
15.60
12.40
2.40
25.87
6.20
0.60
41.90
Qwen2-VL-72B VL-72B
27.94
23.33
55,33
62.00
30.32
87.33
0.00
70.02
Otter
0.00
0,00
26.67
12.67
17.62
18,00
19.33
17.69
Show-o
0.00
0.00
0.00
0.00
0.00
5,30
1.20
7,84
NExT-Chat
0.00
0.00
2.00
4.66
7.91
1.33
24.66
3.43
IntemnL2-226B
0.00
12.40
36.80
38,20
25.20
73.80
37.40
41.90
Qwen2- VL-72B
23.60
54.60
73.40
38.10
89.80
71.60
73.20
DeepSeek-VL-222 VL-2-small
0.30
15.00
30.60
30.60
24.40
53.20
46.20
28.00
DeepScck-VL-22
0.20
10.60
35.80
28.00
24.80
62.40
41.00
41.60
LLaVA-One-Visie Vision-7B
0.10
14.60
42.00
31.00
27.20
71.40
46.00
34.30
LLaVA-OneVViiion225
8.30
30.20
46.00
74,20
36.40
91.40
67.40
57.00
Su2VA-8B
19.20
45.40
20.60
27.80
75.00
38.40
Sa2VA-26B
2.80
18.60
43.00
20.40
28.40
81,80
49.20
43.60
CoLVA-2B
0.00
1.20
25.60
20.20
22.40
44.60
37.40
12.60
VA--AA
2.60
13.80
38.60
30.60
26.40
76.80
37.00
31.90
Long-LLaVA
0.00
14.80
28.00
35.20
25.20
58.00
43.20
25.00
LM4LV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Vitron-V VI
3.62
9.60
24.36
19.37
22.30
31.60
20.70
5.23
PandaGPT (13B)
0.00
0.00
13.60
25.80
12.50
16.30
25.70
1.45
AnyGPT
5.69
9.60
10.20
11.00
9.80
11.72
2.57
GAMA
0.00
0.00
15.67
19.86
24.10
18.45
21.32
4.35
Pengi
0.00
0.00
13.60
25.80
12.50
10.75
16.98
0.65
SALMONN-7B
0.00
0.00
15.20
4.80
20.00
16.80
7.60
11.98
SALMONN-13B
0.00
0,00
15.60
5.80
12.50
17.30
7.90
12.66
WavL.LM
0.00
0.00
15.69
20.81
24.32
19.54
20.23
3.45
ImageBind-Lan -LM
0.00
0.00
13.60
25.80
12.50
16.30
25.70
0.66
Unified-io-2-XXI
0.00
5.60
1.50
2.50
15.40
3.10
8.40
0.00
ModaVerse-76e 7b-v0
25.80
12.50
16.30
25.70
0.66
AudioGPT-GPTS
32.37
23.00
31.80
40.22
4.40
0.20
3.40
73.34
SpecchGPT-7B-com
0.00
0.00
27.40
20.80
23.10
23.80
20.10
0.65
LLaMA-Omai
1.23
15.10
13.00
17.80
15.10
26.40
13.40
17.34
3D-LLM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.35
PoinLLM-777
0.00
0.00
0.00
0.00
PoinLLLM-13B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
3i-VisTA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MotionGPT-TS
0.00
0.00
0.00
0.00
0.00
0.00
0.00
MotionGPT-LLAME
0.00
0.00
12.28
19,80
22,80
22.80
22.20
0.00
AvatarGPT
0.00
0.00
0.00
0.40
0.00
0.00
0.80
0.00
LLaMA- Mesh
0.02
0.00
26.20
20.40
22.80
24.20
21.10
19.42
114

---

On Path to Multimodal Generalist: General-Level and General-Bench
D Discussions and Future Investigation
We propose a leveled evaluation of MLLMs, with a hierarchical framework called General-Ieved and a large-scale benchmark dataset General-Bench. Yet we believe several aspects of this work can be further improved. In addition, as the next step to achieve more capable multimodal generalists toward AGI, we believe some points are worth further investigation.
Further refinement of the General-Ievel framework. Although this framework is concrete starting point for building true multimodal generalists, there are still areas for improvement, especially in terms of the algorithms. For example, the coordination average used to compute Level-3 in the General-Level framework assumes a balance between the number of comprehension and generation tasks, which is an unrealistic assumption. Additionally, we have relaxed the measurement of synergy by assuming that a model's synergy capability is reflected in its ability to surpass SoTA specialists in task performance, avoiding a direct measurement of the synergy effect. Future work will consider optimizing this aspect to provide a more robust definition.
Expanding the General-Bench dataset include more comprehensive tasks and modalities. To ensure the evaluation of multimodal generalists is complete and unbiased, the General-Bench dataset should be further expanded. Currently, the dataset is somewhat imbalanced across different modalities and tasks; for example, there is more data for image-related tasks than for audio and 3D modalities, and there are more comprehension tasks than generation tasks. Moreover, multimodality should be more broadly defined to include not just visible information such as language, vision, and sound, but also other signals and types of information. For instance, LLMs' reasoning abilities have been shown to be significantly enhanced through learning code. As multimodal models, it is necessary to support some coding capabilities, which, in turn, could theoretically improve the model's 's understanding and reasoning in other modalities, such as vision. Finally, our current benchmark primarily considers tasks in which individual modality is operated in isolation. However, in reality, good multimodal generalist should be capable of modality-switchie and modality-interleave reasoning (in both comprehension and generation) under multi-turn user-machine interactions. In the future, we plan to incorporate tasks and datasets that assess interleaved modality capabilities.
Rethinking Evaluation Paradigm for Model Capabilities. Many current task evaluation methodologies still follow conventional paradigms. In most cases, automatic and scalable evaluation strategies are preferred (we provide a detailed overview in Appendix $B.4). While such approaches may suffice for relatively straightforward masks-such as multiple- choice or tlassification--teyyy often fall short when applied to format-free tasks, particularly those involving multimodal generation. For instance, in video or 3D generation, traditional metrics like FID or FVD are increasingly considered inadequate, as they fail to reliably capture the quality and fidelity of the generated content. Consequently, there is growing reliance on human evaluations. To improve scalability, many recent works have begun employing LLMs to simulate human-level judgment (Zheng et al., 2023). However, this "LLM-as-a-judgee" approach introduces challenges in terms of evaluation stability and reproducibility, which remain open research problems. In addition, our current General-Levv evaluation framework adopts a single primary metric per task, which may inherently introduce bias. We argue that future evaluations should incorporate multiple complementary metrics to provide more comprehensive assessment. Lastly, as multimodal generalist models continue evolve with stronger reasoning capabilities, corresponding benchmarks should also be upgraded to evaluate the interpretability and traceability of their intermediate reasoning processes.
Optimizing model architecture to support more functionalities and modalities with stronger performance. Our above experiments reveal that very few MILMS currently achieve unified capabilities. Most models support only 1-2 modalities or abilities, which severely limits their qualifications as generalists. Some models, while supporting multiple tasks and modalities, still exhibit limited performance in individual tasks. Future research could focus on optimizing model architectures to support as many functions and modalities as possible while also delivering stronger performance. We note that simply integrating multiple models or modules using an agent-based approach can increase the number of supported functions and modalities but does not necessarily improve performance (i.e., it still cannot surpass individual specialists). A more promising approach may be to leverage the Mixture of Experts (MoE) strategy to construct more unified MLLMs Recently, more advanced understanding and generation capabilities have also been achieved through some of the latest architectures, such as those that combine autoregressive and diffusion frameworks.
Strengthening synergy capabilities is the key focus. As emphasized in this work, achieving synergy is the fundamental requirement, and is crucial for ensuring the model has more powerful capabilities (compared to specialists). To accomplish this, several aspects need to be considered. First, at the architectural level, is necessary to design essential modules or mechanisms that allow the model to flexibly and effectively transfer features learned from different tasks and modalities (Fei
115

---

On Path to Multimodal Generalist: General-Level and General-Bench
al., 2024a; Pan al., 2025). This is key to enabling the "learning by analogy" capability. Second, the learning level, achieve stronger capabilities across multiple tasks and modalities, the model must be trained in way that prevents it from forgetting previously learned knowledge when learning new tasks. Also, recently, by incorporating training techniques such as Reinforcement Learning from Human Feedback (RLHF), models have achieved more powerful reasoning capabilitiess and improved generalization.
E Author Contribution
All authors contributed to this project in various capacities, including idea conceptualization, data annotation, model implementation, paper writing, and project supervision. To provide transparent overview, Table 91 summarizes the contributions and responsibilities of all co-authors. Given the extensive scope and workload of this project, we enlisted the help of large group of contributors. Among them, some individuals made contributions but did not qualify for co-authorship due to partial involvement or insufficient or voluntary contributions. Nevertheless, we acknowledge their efforts and list them in Table to express our gratitude for their support.
Table 91: Detailed author list and contribution statement.
# Name
Group All
Role
Responsible Datasets Responsible Models All specialists in audio- generation tasks
1
Hao Fei
1) Project general leader: design and imple- All audio-generation the idea, including General-Level evalua- datasets tion and General-Benne planning.
Audio MLLMs: WavLLM, ImageBind-LL.M, Unified-
2) Worker for the audio group.
3) Designed data collection methods and se-
io-2-XXL. ModaVerse-7b-
lected models for verification.
v0.
AudioGPT-GPT4,
4) Responsible for all paper writing, illustra-
SpeechGPT-7B.com.
tions, and polishing.
I.LMA-Ommi
5) Managed online deployment of data and au- tomated evaluation systems.
6) Maintained the project website. 7) Provided
computing resources.
2 Yuan Zhou
1) Project co-leader: give the formal and All Image group formula definitions for the 5 levels of the datasets
Working
image-related Image-oriented MI.LMs:
GPT4-o.
GPT4-o-mini.
General-Lev evaluation framework, along with the corresponding formula derivation.
GPT4-V
2) Led the image group, managing tasks and execution.
3) Constructed and polished over 150 datasets; implemented around 30 SoTA specialists and
MLLMs.
4) Verified task and data management, and de- played systems.
5) Developed evaluation scripts and automated
testing systems.
Juncheng
Working
Project co-leader for image group: Led All
image-related Specialists and MLLMs sup- porting image-related skills
Imuge group
image group for supervised dataset collection datasets
and image-based MILMS evaluation.
Xiangtai Li
Working for Video group
Project co-leader for video group: Led the All video-related datasets video group for supervised datasets collection
Specialists and MLLMs sup- porting video-related skills
and video-based MILMs evaluation.
5 Qingshan Xu
Working for 3D group
Project co-leader for 3D group: Led the 3D All 3D-related datasets group for supervised datascts collection and 3D-based MLLMs evaluation.
Specialists and MLLMs sup- porting 3D-related skills
116

---

On Path to Multimodal Generalist: General-Level and General-Benu
Name
Group
Role
Responsible Datasets
Responsible Models
Bobo Li
Working
Project co-leader Language group: Led Language-related
Specialists supporting L-2.
Lunguage
Language group for supervised datasets collec- datasets: L-2, 14. 15, 16,
14, 15. 16, 17. 18. 19, 20. 21, 22 skills, and MILMS in- cluding Qwen2.5-7B-Instruee, Baichuan2-77B Brrchuan2-B-Base- BBBBsse, Vicuna-
group
tion and Language-based MLLMM evaluation. 17, 18, 19, 20, 21, 22
7b-V1.5, Falcon3-77 Bee--B--Inssett
Ministral-S8 Instruct-2410
7 Shengqiong Wu
Working Project co-leader for the audio group: Led Audio-related datasets: Specialists supporting A-C-I, audio comprehension group for the collection A-C-1, 2, 3,4, 6, 2, 3, 4, 5, 6, 7, 8, skills, of supervised datasets and evaluation audio-bused MLLMs.
Audio group
MLLMs including
Qwen-Audio-Chan Qwen2-
Audio-Instruct, Vitron-V1,
GAMA, Pengi, WavI.LM,
SALMONN-7B SALMONN-
13B
SpeechGPT-7B-cog
AudioGPT-GPT4, AnyGPT,
PandaGPT-11B, ImageBind-
LLM. ModaVerse-7bbvo,
Unified-io-2-XXI, NExT-
GPT-V1.5
Yaoting Wang
Working for Image group tion
Implement image-related MI.LMs for evalua-
MILMSs including Qwen2-
VL-7B, Qwen-VL--Cat,
MoE-LLLVVA-Ph22 -ww12-
4e-384,
mPLUG-OWe
LLaMA2-7b, Phi-3.5-Vision
Instruct. Cambrian-1-8h.
MiniGPT4-LLMM2-7BB
Junbao Zhou
Working
Collect 3D-related datasets Implement 3D-related datasets: D-C- Specialists supporting D-C-1,
3D group
related Specialists and MLLMs for evaluation 1, 3, 4, 6, 7, 8. 10, 2. 3. 4. 5, 6, 7, 8. 9, 10. 11.
11, 12,
12, skills, MILMS
including 3D-LLM-2.1B
PointLLM-7B, PointLLM 13B, DD-ViSTA
10 Jiahao Meng
Working for
Collect video-related datasets and Implement Video-related datasets: V. Specialists video-related Specialists and MIL.Ms for eval- C-2, 3, 4,420
supporting
Video group
V-C-2, 3, 4, 20 skills,
uation
and MLLMs including
Long-LLaVA-999 DeepSeek-
VL-2-small, DeepSeck-VI.c VLaVA--One-vision-7e
11 Qingyu Shi
Working for Video group
Collect video-related datasets and Implement Video-related datasets: V- Specialists supporting V-G-1, video-related Specialists and MLLMs for eval- G-1, 2,
2. 3. tasks, and MLLMs in- cluding VidAgent
uation
12 Zhiyuan Zhou
Working Collect image-related datasets Implement Image-related datasets: I- Specialists supporting I-C-5, Image group
image-related Specialists and MILMs for eval- C-5, 14, 15, 26, 28, 30, 7, 14, 15, 26, 28, 30, 34 uation
skills, MLLMS includ- ing InterVLL2.5225 In- te2n InternVL2.5.
8B. Monkey-10B-chat. DeepSeck-VL-7B-Chu. VL-7B-Chat
13 Liangtao
Working Collect image-related datasets Implement Image-related datasets: I- Specialists supporting
Image group image-related Specialists and MLLMs for eval- C-5, 13, 19, 26, 27, 28 uation
C-5. 13, 19. 26, 27. 28
skills, MLLMs includ- ing Mini-Imeen-L-Chat-BB-WW
IntemVL-Chat-V-ss,
M-CompoMXCoompsee...... VL-
1.8B, GPT4Rol-7B, GLaMM
117

---

On Path to Multimodal Generalist: General-Level and General-Benee
Name
Group
Role
Responsible Datasets
Responsible Models
Minghe Gso
Working Collect image-related datasets Implement Image-related datasets: 1- Specialists supporting
Image group
image-related Specialists and MLLMs for eval- C-8, 9. 17, 21, 25, 26, 28, uation
8. 9. 17, 21, 25, 26. 28. 31, 35 skills, MILMs including BL,IP, miniMon- key, DeepSeek-VEL VL-7B-Base,
31, 35
LISA
15 Dacan Zhang
Working for Collect 3D relaeeae datasets and Implement 3D- 3D 3D-related datasets; D-G- Specialists
D-G-I, 2, supporting 8, skills, and MLLMs
3D group
related Specialists MLLMs evaluation 1,
including MotionGPT.
T5. MotionGPT-LLaMA.
AvatarGPT, LLaMA-mesh
16 Zhiqi Ge
Working for Collect image-related datasets and Implement Image-related datasets: I- Specialists supporting I-C-4. Image group image-related Specialists and MLLMs for eval- C-4, 26, 34 uation
26, 34 skills, and MILMS including Claude-3.5-Sonnet, Blaude3355-Opus Emu2-37B,
DetGPT
17 Weiming Wu
Working for Implement image-related MI.LMs for evalua- Image group
MI.MMs including Otter,
tion
Show-o, NExT-Chat, Yi-
vision-v2, Qwen2-VI--2BB
18 Siliang Tang
Working for Collect image-related datasets and Implement Image-related datasets: I- Specialists supporting I-C- Image group image-related Specialists and MILMs for eval- C-26, 34 uation
26, 34 skills, and MLLMs including Claude-3.5-Somme Claude-3.5-Opus. Emu2-37B,
DetGPT
19 Kaihang Pan
Working for Collect image-related datasets and Implement Image-related datasets: Specialists supporting I-C- Image group
image-related Specialists and MLLMs for eval- C-17, 22, 23, 39 uation
17. 22, 23, 38, 39 skills,
and MLLMs including
Pixtral-12B, SEED-LLaMA- 13B, LLaVA-NeXT-133,
LLaVA-NeXT-344
20 Yaubu Ye
Working for Collect image-related datasets and Implement Image-related dutasets: I- Specialists supporting I-C-3, Image group image-related Specialists and MILMs for eval- C-3, 7, 28, I-G-1, (,-,,3, 3, 4, 7, 28, I-G-1, 3, 4, 5, 6, 7, uation
6, 8, 12, 14,
12, 14, 15 skills, and
MILLMs including BLIP-3
(XGen-MM), CogVLM Chat, ShareGPT4V-7B.
ShareGPT4V-13B
21 Haobo Yuan
Working for Collect video-related datasets and Implement Video-related datasets: V. Specialists supporting V-C-5, Video group
video-related Specialists and MLLMs for eval- C-5, 7. 10. 11, 12, 7. 9. 10, 11. 13 uation
13
skills, and MLLMs including IntemmV InternVi- 2.5-8B, IntemVV--2-26B, ImtrnVLL-2.5-268
22 Tao Zhang
Working for Video group
Collect vidco-related datasets and Implement Video-related datasets: V- Specialists supporting V-C- video-related Specialists and MLLMs for eval- C-14, 15, 16, 17, 18, 19 uation
14, 15, 16, 17, 18, 19
skills, MLLMs includ- ing CoLVA-2B, CoLVA-4B, Sa2VA-8B, Su2VA-26B Specialists supporting I-1,
23 Tianjic Ju
Working for Collect language-related datasets and Imple- Language-related
Language ment language-related Specialists and MLLMs group
datasets: L-1, 2, 3. 5,
2, 3, 4. 5. 6, 7, 8, 9, 10, 11, 12. 13 skills, and MLLMs including Meta-Llama-3.1 8B-Instruct, Gemma-2-9b-it.
for evaluation
6, 9, 10, 11, 12, 13
GPT-J ChaGGLM-6B,
I Yi-
lightning
118

---

On Path to Multimodal Generalist: General-Level and General-Benel
Name
Group
Role
Responsible Datasets
Responsible Models
Zixiang Meng
Working Collect image, video-related datasets and Im- Image and plement image, video-related Specialists and Video group MLLMs for evaluation
Image-related datasets: Specialists supporting I-C-6.
C-6, 28, 38, und Video- 28, 38 and V-C-20 skills, and
related datasets: V-C-20 MILMS including Gemini-
1.5-Pro, Gemini-1.5-Flash,
LLaWA-IntemLLM20..
25 Shilin Xu
Working for Collect Video group sideo-related Specialists and MLLMs for eval- C-1, uation
datasets and Implement Video-related datasets: V. Specialists supporting V-C-
2, 4 skills, MLLMs including Intern
VL-2522L Qwen2-
VL-7B, Qwen2-VL-72B
26 Liyu Jia
Work for im- Collect image-related datasets and Implement image-related dataset col- MLLMs including GPT4-o- image-related Specialists and MLLMs for eval- lection: I-C-6, 17, and 4096.
age group
uation
27 Wentao Hu
Work Collect image-related datasets Implement image-related dataset col- MLLMs including gpt-3.5. turbo, chatept
age group
image-related Specialists and MLLMs for eval- lection: I-C-17, 28
uation
28 Meng Luo
Working for
Collect video-related datasets and Implement Video-related datasets: V. Specialists supporting V-C- video-related Specialists and MLLMs for eval- C-2, 5,6
Video group
2. V-G-1, 5, 6 skills, and
uation
MLLMs including LM4LV
29 Jiebo Luo
Discussion
Discussed the high-level directions and goals of the project. Provided important and insight- ful feedback for the overall system design.
& Advisory
30 Tat-Seng Chua
Discussion
Discussed the high-level directions and goals of project. Provided important and insight- ful feedback for the overall system design.
& Advisory
Hanwang Zhang
Project
1) Project co-supervisor, conceptualized
Supervision
idea of General-Leve and the entire process.
2) Provided computing resources.
32 Shuicheng Yan
Project
1) Project co-supervisor, co-coneeptualized
Supervision
idea of General-Lever and supervised the en-
tire process.
2) Provided computing resources.
119

---

On Path to Multimodal Generalist: General-Leve and Ceneral-Bencl
Table 93: List of some contributors without authorship.
Name
Contribution
Zhengzhe Liu
Contributed image group: assisted in image-oriented dataset preparation and model testing.
2
Zhongze Luo
Evaluating Yi-vision-v2 on 77 dataset; evaluating Show-o on 65 datasets.
3
Chunhan Li
Evaluating Show-o 136 image comprehension datasets and partial 67 datasets.
4
Qirui Huang
Evaluating Yi-lighming on 117 NI.P datasets and evaluating Show-o 44 image generation.
5
Jiaxin Zhu
Assist in evaluating certain MLLMs certain datasets.
6
Ming Lei
Evaluating Otter on certain datasets.
7
Zhangyu Wang
Evaluating Otter on certain datasets.
8
Lin Liu
Contributed to the preparation image-related task data during phases 1 and 2.
9
Chengjie Zhou
Contributed the preparation NLP task data during phase
10
Yucheng Han
Contributed the preparation image-related task data during phase 1.
Peng Zhou
Contributed the preparation image-related task data during phase 1.
12
Luanyuan Dai
Contributed the preparation image-related task data during phase 2.
13
Yuxuan Liu
Contributed the preparation image-related task data during phase 2.
14
Xun Jiang
Contributed the preparation image-related task data during phase 2.
15
Peisuo Li
Contributed the preparation of image-related task data during phase 2.
16
Xu Zhang
Contributed the preparation of image-related task data during phase 2.
17
Wenjie Zhuo
Contributed the preparation of image-related task data during phase 2.
18
Lianyuan Fan
Contributed to 3D generation-relate related data collection during phuse 2 (incomplete).
120


Reviews:
[{"review_id": "c93Yd20QFn", "reviewer": "Reviewer_xWYG", "summary": "The authors introduce General-Level, a comprehensive evaluation framework for multimodal generalist models that emphasizes synergy across tasks and modalities. It further presents General-Bench, an extensive benchmark covering over 700 tasks spanning various modalities to assess both comprehension and generation capabilities. The framework categorizes model performance into five levels, reflecting the progression from task-specific skills to cross-modal generalization critical for achieving AGI. Extensive experiments reveal that while current models show progress, significant gaps remain in true synergy and broad task support.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewers' recognition of our work and the valuable feedback provided. Below, we provide detailed responses.\n\n\n\n-----\n**Q1.  But I think the normalization and metric mapping methods are mentioned without thorough empirical validation, leaving their effectiveness in accurately comparing heterogeneous tasks not 100% substantiated.**\n\n**A:** We would like to clarify that the proposed normalization and metric mapping techniques are primarily designed to enable fair and consistent comparisons across heterogeneous tasks. This is necessary because certain evaluation metrics, such as FID, are not naturally bounded within the [0, 1] range, and lower values indicate better performance. While it is reasonable to directly compare models using FID within a single task, averaging performance across multiple tasks becomes problematic when combining metrics with different scales and monotonicities—such as FID (lower is better) and ACC (higher is better).\n\nTo address this, we carefully designed a normalization and metric mapping strategy that brings different evaluation metrics into a unified range. We also took special care in choosing appropriate scaling factors to ensure that the transformed scores remain faithful representations of the original quality measurements. For example, an FID of 25 and an FVD of 100 are mapped in a way that preserves their relative performance levels. This normalization process allows us to compute an overall average performance across tasks without introducing unintended biases.\n\n---\n**Q2.  The synergy effect design would benefit from more ablation studies to isolate the contribution of individual components.** \n\n**A:** Thank you for the suggestion. We would like to point out that a detailed analysis and discussion of the synergy effect in our multimodal generalist framework—across skills, comprehension and generation capabilities, and different modalities—is provided in Appendix B.7. We kindly refer the reviewer to that section for a comprehensive breakdown.\n\n---\n**Q3. While the benchmark provides a broad and impressive evaluation across diverse tasks, it currently lacks in-depth analyses of failure cases, which could limit insights into the specific challenges faced by MLLMs.**\n\n**A:** Thank you for raising this important point. The core motivation behind General-Bench is to provide a comprehensive and systematic evaluation of MLLMs that goes far beyond conventional VQA-style assessments. To this end, we deliberately designed the benchmark to cover a wide range of task formats, modalities, and skills. This enables us to quantitatively and qualitatively measure MLLM performance from multiple perspectives, offering valuable insights into their generalization capabilities and limitations.\n\nThat said, we fully agree with the reviewers that in-depth failure case analysis is crucial for understanding model weaknesses and guiding future improvements. While our current focus has been on establishing broad coverage and performance patterns across dimensions, we acknowledge the value of qualitative case studies. In future work, we plan to include more detailed failure analyses and case-based evaluations to better illuminate the specific challenges MLLMs face and foster more targeted research efforts in this direction.\n\n---\n**Q4. The paper assumes effective cross-modal synergy without adequately addressing integration and robustness issues; I think future work should focus on these aspects for improved model interoperability.**\n\n**A:** Thank for the reviewer’s constructive suggestion. In Appendix B.7, we provide a preliminary analysis of cross-modal synergy, particularly in terms of whether different models have learned synergy effects across modalities. We agree and will conduct a deeper exploration of integration mechanisms and robustness to modality-specific noise or failure for improving model interoperability in the revision.\n\n---\n**Q5. While the paper posits that non-language modalities can enhance language intelligence, the experimental evidence for this reverse synergy might be largely absent.**\n\n**A:** We appreciate the reviewer’s comment and would like to clarify a possible misunderstanding regarding our claim. Our primary argument is that language serves as a strong prior to enhance other modalities, rather than the reverse. While we do not deny that non-language modalities can, to some extent, aid language understanding, our empirical findings suggest that current multimodal signals are not yet capable of boosting language performance beyond that of SoTA NLP models.\nIn other words, we distinguish between helping language understanding and surpassing NLP SoTA performance through multimodal input—two very different thresholds. Our intention was to highlight this gap and motivate future work toward developing truly synergistic models where multimodal information can meaningfully and reliably elevate language intelligence beyond what is achievable through text alone.", "claims_and_evidence": "This paper is quite extensive in both length and content, and thus presents a lot of scientific claims. Many claims are backed by extensive experimental results and comprehensive benchmark data, particularly regarding the framework’s ability to differentiate models based on their task and modality support. Other claims, such as those concerning the detailed characteristics of the datasets, are supported by sufficient details. Also, the findings and conclusions in the experimental section are validated and supported by corresponding experimental data. In particular, regarding the general-level properties, the authors provide ample mathematical proofs in the appendix. But I think the normalization and metric mapping methods are mentioned without thorough empirical validation, leaving their effectiveness in accurately comparing heterogeneous tasks not 100% substantiated.", "methods_and_evaluation_criteria": "The authors propose a completely new evaluation approach, called the General-Level framework. The proposed methods and evaluation criteria, in my opinion, are innovative and largely appropriate for assessing multimodal generalist capabilities. The multi-level General-Level framework and the expansive General-Bench dataset provide a comprehensive way to capture both comprehension and generation across various modalities. Overall, I have not found any obvious issues with the validation methods or processes.", "theoretical_claims": "The paper’s theoretical claims, especially those regarding the “synergy effect” that underpins the proposed 5-layer evaluation framework, are more conceptual than rigorously proven. Their core argument is that current benchmarks for multimodal generalists or MLLMs only compare performance across different tasks, failing to fully assess these models’ true capabilities. The authors provide some mathematical proofs in the appendix concerning certain properties at the General level. I examined the rationale for defining synergy levels and the assumptions underlying the ability of multimodal generalists to outperform specialized models.", "experimental_designs_or_analyses": "The experimental design is very extensive, evaluating over 700 tasks with diverse modalities, which provides a broad view of multimodal capabilities. The analysis comparing generalists against SoTA specialists is methodically structured. Also the paper includes very appealing visualization-based analyses. But there might be several points to be further improved. 1) The synergy effect design would benefit from more ablation studies to isolate the contribution of individual components. 2) While the large-scale benchmark is impressive, some analyses appear to emphasize breadth over in-depth evaluation of failure cases, which limits understanding of the underlying challenges.", "relation_to_broader_scientific_literature": "The paper’s contributions are deeply rooted in the evolving landscape of MLLMs or multimodal generalist models. It provides a novel perspective on understanding and evaluating the capabilities of multimodal generalists. The notion of “synergy”, a central theme in the paper, probably builds on earlier ideas of cross-modal joint learning or transfer learning, where knowledge transfer between modalities should be explored in various studies. But to my knowledge, there isn’t any prior related work that evaluates MLLMs in this way. Furthermore, the comprehensive benchmark (General-Bench) and the tiered evaluation framework resonate with existing efforts in creating standardized tests for model performance, such as LVLM-eHub, MME, and others. By synthesizing these ideas, I think the paper provides a structured method to compare and improve multimodal generalists, advancing the broader conversation on achieving AGI.", "essential_references_not_discussed": "There are no essential related works missing from the citations.", "other_strengths_and_weaknesses": "In the above fields, I have thoroughly emphasized the value and strengths of this work. Overall, I believe this work will bring a significant revolutionary impact to the MLLMs community, potentially leading and even changing the current development direction of large multimodal foundation models.\n\nHowever, I do have a few minor concerns (which might be potential weaknesses) that I hope the authors can address or clarify during the rebuttal phase:\n\n- The experimental study may lack sufficient discussion of failure cases with detailed analyses to fully summarize the common errors made by existing MLLMs. Such insights could help guide future research directions.  \n\n- The paper assumes effective cross-modal synergy without adequately addressing integration and robustness issues; I think future work should focus on these aspects for improved model interoperability.\n\n- While the paper posits that non-language modalities can enhance language intelligence, the experimental evidence for this reverse synergy might be largely absent.", "other_comments_or_suggestions": "See Weakness.", "questions_for_authors": "See Weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tJmGy3xCGT", "reviewer": "Reviewer_KVoS", "summary": "This paper introduces a five-tier General-Level framework that assesses multimodal generalists based on their synergy across comprehension, generation, and cross-modal interactions. Also, inspired by autonomous driving grading, it proposes a new benchmark, General-Bench, covering over 700 tasks and 325K instances across diverse modalities. Evaluation of 100+ state-of-the-art models reveals that most multimodal large language models (MLLMs) struggle to achieve true cross-task and cross-modal synergy. Authors show many interesting insights via their framework and benchmark, highlighting significant challenges on the journey toward genuine AGI overall progress.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer’s recognition of our work and the thoughtful, constructive feedback. Below, we provide point-by-point responses to each comment. Hope you can reevaluate our work if you feel the response is effective and useful.\n\n---\n\n**Q1. The “Limitations and Future Investigation” section is helpful, but future research directions could be elaborated further.**\n\n**A:** Thank you for this helpful suggestion. In addition to the *Limitations and Future Investigation* at section 7, we have also included a more actionable guide in **Appendix C.5: “Path to Advancing Higher in General-Level”**. This section outlines concrete guidelines for how future MLLMs can progress through the levels in our framework, offering clear directions for advancing synergy, modality coverage, and generalization—key factors in moving toward Level-5 multimodal generalists.\n\n---\n\n**Q2. Do the “skills” in Tables 3–7 refer to meta-tasks? Are they directly mapped to the specific tasks in Appendix Table 103?**\n\n**A:** Not really, the “skills” listed in Tables 3–7 refer to **meta-tasks**, but they do not directly correspond one-to-one with the specific tasks in Appendix Table 103. Instead, each skill (i.e., meta-task) includes **multiple specific tasks**. For example, the skill *Crack Detection* includes the specific tasks *Tire Crack Detection* and *Road Crack Detection*. This hierarchical organization allows us to abstract task capabilities at a higher level, making the benchmark both manageable and scalable.\n\n---\n\n**Q3. Why are pure language LLMs included in Table 9, given that the focus is on MLLMs?**\n\n**A:** We intentionally included language-only LLMs in the comparison to provide a **reference point**. This helps readers assess the performance gap between unimodal LLMs and MLLMs on NLP tasks. It also highlights a key finding of our benchmark: **multimodality has not yet enabled MLLMs to outperform SoTA LLMs on core NLP tasks**, which is critical for understanding the current limitations of multimodal synergy and what it would take to reach Level-5 performance.\n\n---\n\n**Q4. Why do GPT-series models not achieve top rankings across all levels? And why do only three models reach Level 4? These results feel somewhat counterintuitive.**\n\n**A:** This is an excellent question. Although GPT models demonstrate strong performance in several areas, our analysis shows that they often excel in **specific task types**, such as vision comprehension, where they behave like specialists. However, they **lack broad modality and task support**, and in many cases, do not even support certain modalities or task formats.\n\nOur *General-Level* scoring framework is based on two core principles:  \n1) The model should be a **true generalist**, i.e., capable across a **wide range of modalities and task types**.  \n2) **Synergy** is the key metric—performance gains must come from meaningful cross-modal, cross-task integration.\n\nGPT models, while powerful, do not consistently meet these criteria across all levels, which explains their limited presence at Level 4 and absence from the top in other levels. Thus, the result is not only reasonable but aligns with our framework’s goals: to reward generality and synergy, not just isolated task excellence.", "claims_and_evidence": "- The authors claim they have developed a new evaluation framework and have indeed proposed a completely new theoretical basis for it.  \n- They assert that they have introduced the most comprehensive and largest-scale benchmark dataset to date; subsequent comparisons with other datasets clearly demonstrate that its scale and scope exceed those of existing benchmarks.  \n- The authors contend that current MLLMs still face numerous issues that existing benchmarks fail to evaluate, and they have verified these problems in their experiments.", "methods_and_evaluation_criteria": "This is a benchmark paper. The authors propose a new evaluation approach for multimodal generalists (MLLMs/agents), focusing on the models' synergy effects across comprehension, generation, and cross-modal interactions. They have also introduced an entirely new benchmark dataset to evaluate over 100 LLMs from different perspectives and methodologies. Further, the authors provide extensive and detailed information in the appendix—nearly 300 pages—to substantiate the reliability of both the evaluation framework and the dataset.", "theoretical_claims": "The authors propose a five-tier General-Level evaluation framework that incorporates innovative theoretical contributions. Their core claim is that current benchmarks for multimodal generalists or MLLMs merely compare performance across individual tasks, which fails to fully assess the true capabilities of these models. Consequently, they introduce a new evaluation approach based on the synergy effects of MLLMs in comprehension, generation, and cross-modal interactions. Further, to validate the soundness of their evaluation framework, the authors provide extensive mathematical proofs in the appendix, which I have reviewed and found to be both mathematically correct and robust.", "experimental_designs_or_analyses": "The authors conducted extensive evaluations on over 700 multimodal tasks for more than 100 MLLMs. The assessments include individual task results, meta-task outcomes, the number of supported tasks, comparisons where models surpass state-of-the-art specialists, and the final ranking of models across different levels. Also extensive visualization analyses reveal the performance preferences of various models, all of which are both interesting and insightful. I found the experimental scope to be so vast, with validation approaches and perspectives that are both reasonable and comprehensive.", "relation_to_broader_scientific_literature": "I believe there are two core contributions:  \n1) The authors introduce an entirely new perspective for evaluating the rapidly increasing number of MLLMs. Instead of simply comparing performance across various tasks, they propose a capability grading system akin to that in the autonomous driving industry, based on the core idea of synergy. This approach is poised to revolutionize the field of MLLM evaluation and guide the development of the MLLM community. \n2) I think the authors have developed an ultra large-scale evaluation benchmark for MLLMs that encompasses 145 skills across more than 700 tasks with over 325K samples, involving five common modalities and covering 29 domains. This unprecedented comprehensiveness and high quality ensure that the evaluation results should be extremely reliable, which, to my knowledge, might largely position this benchmark as the future standard for performance assessment in the field.", "essential_references_not_discussed": "The references provided in the paper are adequate.", "other_strengths_and_weaknesses": "I appreciate this work because it makes a clear contribution to the community. I believe the significance and value of this paper will be revolutionary in the field. Currently, research on multimodal generalists—whether MLLMs or agents—is gaining increasing traction and is progressively oriented toward developing more powerful models, as the authors claim. An important question for the community is how these multimodal generalists should evolve: should they focus on achieving higher performance or on supporting a broader range of capabilities? Simply assuming that higher scores on various multimodal tasks equate to a more capable generalist is too simplistic. The authors reject this notion and propose an entirely new evaluation approach. They apply a five-level classification system, borrowed from the autonomous driving industry, to rate multimodal generalists, where each level represents a specific range of capabilities, with further differentiation possible within each level. This idea is truly eye-opening, as it is not only theoretically rigorous and correct—thanks to a carefully designed scoring algorithm ensuring key attributes—but also highly feasible. The authors have introduced a novel evaluation perspective and methodology that is set to revolutionize the field.\n\nFor the second point, I was struck by the enormous effort reflected in this work. For instance, the authors have contributed an new dataset—possibly the largest benchmark I have seen—which includes 145 skills across more than 700 tasks with over 325K samples, involving five common modalities and covering 29 domains. Its high quality ensures extremely reliable evaluation outcomes, and this benchmark is likely to become the future standard for performance assessment in the field. Also the team evaluated 100+ current state-of-the-art MLLMs. Lastly, the paper, including an appendix spanning nearly 300 pages, provides exceptionally detailed information, which I find impressive.\n\nOther strengths include the meticulous craftsmanship of the paper; both the writing and the visual presentation (such as the organization and visualizations) are of high quality. The paper is exceedingly detailed, providing necessary detail, and the experimental findings and conclusions are both fascinating and highly instructive for the community.\n\nAs for potential weaknesses, the only concern I can identify is that the authors might need to further strengthen their guidance for the community by providing clearer directions for future research. Although they include a \"Limitations and Future Investigation\" part in Section 7, I feel it could be even more detailed, as noted in the following comments.", "other_comments_or_suggestions": "I believe the authors could offer further guidance from multiple perspectives to the future research community on how to steer the development of MLLMs to achieve higher performance within the General-Level evaluation framework.", "questions_for_authors": "I have a few minor questions:  \n- Do the skills in Tables 3–7 refer to meta-tasks, corresponding to the specific tasks shown in Appendix Table 103? Given the enormous scale and hierarchical structure of the dataset, I couldn’t fully grasp this aspect from the paper.  \n- In Table 9, I noticed that some pure language LLMs are included. Since the evaluation targets MLLMs, why evaluate the performance of LLMs?  \n- Although explanations may have been provided in the experiments, I remain curious: why did the GPT series models fail to achieve leading rankings across all levels, and why do only three models have scores at Level 4? Does this result seem reasonable, as it appears somewhat counterintuitive to me?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EgCqEGrHlK", "reviewer": "Reviewer_7pDZ", "summary": "The paper introduces General-Level, a framework inspired by the autonomous driving industry's capability grading system, to classify Multimodal Language Models (MLLMs) across five levels based on their synergy in comprehension, generation, and multimodal interactions. To support this classification, the authors propose General-Bench, an extensive multimodal benchmark covering 700+ tasks and 325,800 instances across diverse modalities, including text, images, video, audio, and 3D. The evaluation of over 100 MLLMs reveals that most models fail to exhibit true synergy across modalities and tasks, challenging the idea that current MLLMs are progressing toward Artificial General Intelligence (AGI). The authors argue that synergy—the ability to transfer knowledge between modalities and tasks—should be the key metric for evaluating multimodal generalists. However, their assumptions, such as the claim that multimodal synergy can enable generalist models to outperform task-specific SoTA specialists, remain debatable. The benchmark itself is massive and computationally expensive, making it impractical for fast model iteration. The results suggest that multimodality does not necessarily enhance language abilities, contradicting some empirical findings. The paper aims to redefine multimodal evaluation standards but introduces several questionable assumptions and practical challenges.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate you carefully reviewing our paper and raising meaningful questions and constructive suggestions. Below we address your concerns one by one and are open to further discussion. Hope you can raise your evaluation.\n\n---\n\n**Q1. Discussion of related evaluation benchmarks (HELM, VHELM, etc.)**\n\n**A:** Thank you for highlighting this. We acknowledge the contributions of holistic benchmarks like HELM, VHELM, and MMBench, which provide open, standardized infrastructures for evaluating vision foundation models. Our *Generalist Benchmark* aligns with this direction and is designed to serve as an open platform enabling broad community participation and transparent comparison of MLLMs. We will include a more detailed comparison with these existing efforts in the revision.\n\n---\n\n**Q2. The benchmark’s large scale makes evaluation costly and impractical for rapid iteration...**\n\n**A:** We believe the benchmark's scale is a strength, reflecting diversity across tasks, modalities, and formats. For any single task, we limit evaluation to ~500 instances—computationally reasonable for most models.\n\nTo enhance usability, we will further propose a **multi-scope evaluation structure** with three leaderboard types:\n- **Scope-1**: Full-spectrum leaderboard (current General-Bench), for general-purpose MLLMs.\n- **Scope-2**: Modality-specific leaderboards, for modality-specialized models.\n- **Scope-3**: Fine-grained task-cluster leaderboards under each modality.\n\nAll rankings are computed via our *General-Level* framework, allowing users to select appropriate evaluation scope based on their model’s capacity and resource constraints. This makes the benchmark scalable and adaptable: lightweight scopes allow fast iteration; broader scopes offer more visibility at higher computational cost—it's up to the user.\n\nWe do not plan to release dev sets for each task, as this benchmark is strictly designed for **zero-shot evaluation**. Model development and training are left to the developers.\n\n---\n\n**Q3. The assumption that synergy enables generalists to outperform SoTA specialists is unrealistic.**\n\n**A:** We understand the concern regarding synergy and its comparison to SoTA specialists.\n\nAs explained in Appendix C.3 (“Rationality of Scoring Relaxation”), our design follows two steps:  \n(1) Define **synergy** as the core metric for capability levels;  \n(2) Propose a **practical scoring method**, given real-world model training constraints.\n\nIdeally, **synergy** could be defined as a model performing better on a joint distribution $P_\\theta(y|A,B)$ than on $P_\\theta(y|A)$ or $P_\\theta(y|B)$ separately. However, isolating such distributions is infeasible, as large generalist models are already jointly trained on many tasks. Retraining to cleanly separate task spaces is impractical.\n\nThus, we relax the evaluation: we treat cases where generalists **match or exceed SoTA** specialist performance in a task (without in-domain fine-tuning) as indirect but valid signals of synergy—i.e., effective cross-task/modality generalization.\n\nThis is supported by multiple studies showing generalists can outperform fine-tuned specialists:  \n- [1] shows GPT-4, via prompt engineering alone, achieves SoTA in medical QA benchmarks without fine-tuning.\n- [2] shows OpenMedLM, via prompting techniques, surpasses prior fine-tuned open-source models on multiple medical benchmarks.\n- Flamingo [3] outperforms fine-tuned specialist models on six vision-language tasks.\n\nThese findings support our core assumption: **the stronger a model’s synergy capability, the more likely it is to surpass SoTA specialists when synergy is effectively activated.** This avoids costly pairwise modeling and enables practical, scalable synergy evaluation.\n\n[1] *Can Generalist Foundation Models Outcompete Special-Purpose Tuning?*  \n[2] *OpenMedLM: Prompt Engineering Can Outperform Fine-Tuning in Medical QA*  \n[3] *Flamingo: A Visual Language Model for Few-Shot Learning*  \n[4] *Perceiver IO: A General Architecture for Structured Inputs & Outputs*  \n[5] *Segment Anything*\n\n---\n\n**Q4. Multimodal synergy *does* improve language—models like Vicuna, Qwen2, and LLaMA benefit from image-text data.**\n\n**A:** We appreciate this feedback and apologize for the unclear statement. We do **not** deny that multimodal data can improve language understanding. Our point is more specific: **such improvement has not yet enabled models to outperform SoTA NLP specialists on core language tasks.**\n\nThere is a clear distinction between *enhancing language performance* and *exceeding SoTA NLP models*. While models like Vicuna, Qwen2, and LLaMA show better results with image-text pretraining, our large-scale evaluation shows they still fall short of outperforming fine-tuned language specialists. Therefore, our statement does not contradict existing evidence. But for sure we will refine the statement for clarity in the revision.\n\n---\n\n**Q5. Typos: Line 60 “Yhe” -> “The”**\n\n**A**: Thanks！Will correct this.", "claims_and_evidence": "see below.", "methods_and_evaluation_criteria": "see below.", "theoretical_claims": "No", "experimental_designs_or_analyses": "see below.", "relation_to_broader_scientific_literature": "Authors need to discuss more their work and prior holistic evaluation benchmarks like HELM, VHELM, etc.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths**:\n\n1. General-Bench covers a vast range of modalities, making it one of the most extensive multimodal evaluation suites to date.\n2. The proposed General-Level system offers a structured way to assess MLLM capabilities beyond simple task performance.\n3. The paper correctly highlights the lack of cross-task and cross-modal synergy in existing MLLMs.\n4. The authors claim they will maintain an open benchmark and leaderboard, potentially aiding long-term MLLM progress.\n\n**Weaknesses**:\n\n1. Benchmark Construction Relies on Existing Datasets and Is Computationally Expensive:\nThe main contribution of the paper, General-Bench, is largely a repurposing of existing benchmarks rather than a fundamentally new dataset. The biggest issue is its sheer size, making model evaluation prohibitively compute- and time-intensive. While the authors claim they will maintain the benchmark and leaderboard, this does not address the core problem: practical usability during model development. Large-scale evaluations are not feasible for fast iteration, which is why even in the current era, development sets remain essential.\n\n2. Overly Strong and Questionable Assumptions:\nThe paper assumes that \"a model’s synergy capability enables it to outperform SoTA specialists in specific tasks by leveraging knowledge across tasks or modalities.\" This is highly unrealistic. There is no current evidence that a general-purpose LLM can outperform a domain-specific model in its specialized task. For instance, no LLM has surpassed a fine-tuned BERT model in NER. Furthermore, pursuing an all-encompassing LLM for every task is inefficient, both computationally and economically. The cost of using a large LLM for a task that a smaller, specialized model can perform better is unjustified.\n\n3. Misleading Claim About Multimodal Synergy Not Enhancing Language Performance:\nThe paper concludes that multimodality does not enhance language abilities, but this is not always correct. Empirical results from training Vicuna, Qwen2, and LLaMA show that models trained with both image-text and text-only data consistently outperform those trained with text-only data on language benchmarks like MMLU. This trend is reproducible across multiple models, contradicting the paper’s claim. The authors fail to acknowledge variations in training setups that could impact this conclusion.", "other_comments_or_suggestions": "Line 60: Yhe -> The", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ywZu63zOAQ", "reviewer": "Reviewer_Ewcj", "summary": "This paper pioneers the idea of a General-Level framework to evaluate MLLMs, allowing for an accurate assessment of MLLM capabilities.\nThe authors provide significant observations and principles on benchmark setting and design a sophisticated evaluation metric at different levels to maintain the rationality of the benchmark. They build a General-Bench based on the proposed framework, which is a massive benchmark to evaluate the comprehension and generation capabilities of MLLMs across various modalities such as image, video, audio, 3D, language. A comprehensive evaluation result with a massive analysis and discussion is provided in this paper.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are more than excited to receive your very strong recognition of our work, which means a lot! Also thanks for your detailed review and valuable suggestions. We have done our best to address all concerns and are happy to engage in further discussion to improve the clarity and quality of our work.\n\n---\n\n**Q1.  General-Bench focuses on capability while overlooking critical risks (e.g., hallucination, bias amplification). Could high-scoring models in your framework inadvertently reward unsafe behaviors? Should 'safety synergy' be introduced as a separate dimension in the newly proposed framework?**\n\n**A:** Thank you for the reviewer’s thoughtful and forward-looking comment. When designing the scope definition and task list curation for General-Bench, we paid special attention to mitigating potential bias by ensuring diversity in data sources, annotation methods, and task formats. Moreover, we prioritized using well-established datasets with high-quality annotations, and in some cases, relied on manual annotations to further minimize the risk of hallucinated content.\n\nRegarding the safety perspective, we explicitly focused on evaluating positive and safe skill sets when defining the skill scope of our benchmark. During data collection and task construction, we carefully reviewed whether the expected model outputs could involve safety-sensitive content. If any safety risks were detected, we excluded those instances from the final benchmark.\n\nWe sincerely appreciate the reviewer’s suggestion to consider `\"safety synergy\"` as a separate evaluation dimension. We agree that safety is a critical component of trustworthy multimodal AGI systems and should be an integral part of generalist benchmarking. In future versions of General-Bench, we plan to incorporate safety-focused tasks and metrics to better evaluate models not just on capability, but also on responsible and safe behavior.\n\n---\n**Q2. Since the benchmark contains more than 700 tasks, which is tremendous, evaluating models on General-Bench likely requires massive compute. Does this contradict the push for sustainable AI? Have you quantified the carbon footprint, and if not, should this be a mandatory disclosure for future benchmarks?**\n\n**A:** Thank you for raising this important concern. We agree that sustainability is a crucial consideration in designing and deploying large-scale AI benchmarks.\n\nAlthough General-Bench includes over 700 tasks, it is intentionally designed to be modular and flexible. Model developers are not required to run evaluations on the entire task suite. In practice, given prior knowledge of a model’s capability boundaries, developers can select a relevant subset of tasks for evaluation to obtain a fair and rigorous comparison across other models. \n\nWe also acknowledge the importance of quantifying carbon footprint in building more responsible and sustainable AI systems. While we have not yet included CO₂ impact reporting in the current version, we plan to integrate optional carbon estimation tools in future releases, following emerging best practices in large-scale evaluation.\n\n\n---\n**Q3. In the current setting, is each modality equally important on the path to AGI? The authors mention that they would like to incorporate more modalities into this benchmark. The current modalities seem to be the primary modalities for intelligence, while the newly added modalities do not seem as important as the current ones. If each modality is equally important, the current configuration may not fairly reflect the model's capabilities.**\n\n**A:** Thank you for this thoughtful question. We would like to clarify that General-Bench does not assume all modalities are equally important on the path toward AGI, nor do we assign equal weight to each modality in our evaluation. On the contrary, we recognize that different modalities play distinct and complementary roles in intelligent systems, and their importance can vary depending on the task, environment, or application context.\n\nThe currently included modalities—language, image, video, audio, and 3D—have been extensively studied and are more readily available at scale, which naturally draws more research attention. However, other modalities such as heatmaps, sensor data, charts, or even tactile signals also encode rich, structured information about the world. These may be particularly crucial in specialized domains such as medical diagnosis, physical reasoning, or embodied intelligence.\n\nOur motivation for incorporating more modalities is not to suggest that each one is equally critical for AGI, but rather to build a broader and more flexible evaluation framework that can assess how well models generalize across diverse input types. \nWe appreciate the reviewer’s suggestion and will revise the paper to make our position on modality importance and evaluation scope more explicit and transparent.\n\n---\n**Q4. Typos: Line 2328: all task -> all tasks ...**\n\n**A:** Thanks！Will correct this.", "claims_and_evidence": "The claims are clear. This paper provides enough evidence. The authors claim that existing benchmarks rely too heavily on single-task performance metrics; they present evidence by evaluating 100+ MLLMs and showing that most fail higher-level synergy requirements. They further claim that current benchmarks lack coverage of diverse formats and advanced capabilities.", "methods_and_evaluation_criteria": "This paper proposes General-Bench to evaluate multimodal synergy across comprehension, generation, and cross-modal synergy. It introduces a five-level classification system that measures how well models preserve synergy at increasingly complex levels. Experimental results show that even the best-performing MLLMs struggle with higher-level synergy tasks, suggesting current models are far from achieving full multimodal generalization. This paper can offer a systematic and accurate measure of progress toward AGI.", "theoretical_claims": "This paper provides detailed definition of General-Level. I think the proof is rational.", "experimental_designs_or_analyses": "1. The task selection in the benchmark is comprehensive, containing more than 700 tasks and covering all the main tasks across various modalities, fully reflecting the capability of each MLLM.\n2. The selected models also mainly cover the primary open-source and closed-source models, reflecting a high coverage of MLLMs.\n3. The observations and analyses for the experiments provide in-depth insights, such as current MLLMs focusing more on content comprehension than supporting generation, and that multimodality does NOT enhance language.", "relation_to_broader_scientific_literature": "The paper provides a novel MLLM benchmark, inheriting from previous MLLM benchmarks, such as MME, MMMU, and MMT-Bench. However, it also introduces innovations, such as expanding the simple ranking into a five-level ranking, providing a more comprehensive and reasonable evaluation for MLLMs.", "essential_references_not_discussed": "This paper includes enough references, and the authors have adequately discussed the relevant works.", "other_strengths_and_weaknesses": "Strengths：\n1. This paper derives a stunning and impressive idea: introducing the five-level capability grading mechanism from the autonomous driving industry into MLLM evaluation. This category-based ranking can comprehensively assess the MLLM’s synergy capabilities across comprehension, generation, and multimodal interactions.\n2. The definition of General-Level is reasonable and conforms to the requirements of the two assumptions in Section 3.2. In addition, Appendix C.1 provides further explanation, and the general-level is convincing as an evaluation criterion for MLLM capability ranking.\n3. This paper proposes a panoramic evaluation for various multimodal tasks. Regardless of modality coverage or task number, the output performs better than previous MLLM benchmarks. The task selection shows variety and covers various domains, making it very suitable to be a standard MLLM benchmark.\n4. The paper conducts a thorough evaluation on the benchmark, selecting more than 100 MLLMs. This is a substantial workload and clearly demonstrates the current MLLMs’ performance on the new evaluation criteria. Also, the observations gained from the results, such as \"multimodality does NOT enhance language\", provide valuable insights for MLLM research.", "other_comments_or_suggestions": "There might be some minor issues\n- Line 2328: all task -> all tasks\n- Caption in Figure 1: hinge -> hinges\n- Line 60: Yhe -> The", "questions_for_authors": "1. The idea of utilizing General-Level to expand upon classical MLLM benchmarks is novel. However, General-Bench focuses on capability while overlooking critical risks (e.g., hallucination, bias amplification). Could high-scoring models in your framework inadvertently reward unsafe behaviors? Should 'safety synergy' be introduced as a separate dimension in the newly proposed framework?\n2. Since the benchmark contains more than 700 tasks, which is tremendous, evaluating models on General-Bench likely requires massive compute. Does this contradict the push for sustainable AI? Have you quantified the carbon footprint, and if not, should this be a mandatory disclosure for future benchmarks?\n3. In the current setting, is each modality equally important on the path to AGI? The authors mention that they would like to incorporate more modalities into this benchmark. The current modalities seem to be the primary modalities for intelligence, while the newly added modalities do not seem as important as the current ones. If each modality is equally important, the current configuration may not fairly reflect the model's capabilities.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "797oAOP9d1", "reviewer": "Reviewer_Ux81", "summary": "This paper presents a comprehensive benchmark that includes over 700 existing tasks, providing a foundation for evaluating multimodal large language models (MLLMs). It introduces a five-level classification framework designed to systematically categorize MLLMs based on their capabilities and functionalities. Furthermore, the paper conducts an extensive evaluation of hundreds of models using the proposed benchmarks, offering valuable insights into their performance across various dimensions. By establishing a standardized assessment methodology, this work aims to advance research in the field and facilitate the development of more effective and versatile MLLMs.\n\n## update after rebuttal\nI would like to thank the authors for the rebuttal. As the authors didn't update Table 1 of the draft as they mentioned in the rebuttal, I am not able to judge whether they will do so properly. I keep my original rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your time, meaningful questions, and constructive suggestions. Also, your recognition of our paper means a lot to us, which is the source of power to push us forward and enhance this work/project for a greater meaning to the community. We address each concern in detail below and hope that our clarifications help improve your evaluation of our work.\n\n\n\n---\n**Q1. The five-level classification method lacks clarity, as it does not provide well-defined criteria for each level. In particular, the concept of \"synergy\" between different modalities is not clearly articulated, making the classification ambiguous. For instance, Unified-IO-2 and Next-GPT, which are any-to-any modality generative models capable of producing visual and audio outputs from multimodal inputs, are categorized as Level-2, suggesting they lack synergy in comprehension and generation. Meanwhile, DeepSeek-VL and LLaVA-One-Vision, which do not even possess visual generation capabilities, are classified as Level-3, implying they exhibit synergy across tasks. This inconsistency raises concerns about the validity of the classification framework, as it does not appear to align with the actual capabilities of these models. A more precise and well-justified definition of \"synergy\" is necessary to ensure the classification accurately reflects the models’ multimodal abilities.**\n\n**A:** Thank you for the detailed and thoughtful feedback. Due to space constraints, we provided the full definitions and criteria for each of the five levels in Appendix C. We kindly refer the reviewer to this section for a comprehensive explanation.\n\n`First`, it is important to note that the levels in our framework are not mutually exclusive, but rather hierarchical. That is, a model classified at Level-3 will also have valid scores at Level-2, and so on. In Table 1, Unified-IO-2 and Next-GPT are shown as example models under Level-2 to illustrate that they satisfy the baseline criteria for this level—not to imply they are limited to Level-2. In fact, as shown in Table 19 (Appendix), both Unified-IO-2 and Next-GPT also receive valid Level-3 scores, indicating that they exhibit synergy across comprehension or generation tasks. However, they do not appear at Level-4, which requires a synergy in comprehension and generation.\n\n`Secondly`, as for Level-3, the definition explicitly includes models with synergy in comprehension `and/or` generation. Although DeepSeek-VL and LLaVA-One-Vision do not support generative outputs, their performance on comprehension tasks exceeds that of single-modality specialists, thereby qualifying them for Level-3 based on comprehension synergy alone.\n\nWe appreciate the reviewer pointing out this potential source of confusion, and we will revise the paper to clarify the hierarchical nature of the levels, emphasize that the examples in Table 1 are illustrative rather than exclusive, and provide clearer articulation of \"synergy\" as it applies to comprehension and generation tasks.\n\n---\n**Q2. However, the analysis in Section 5.2 feels somewhat superficial, especially considering the large-scale experiments conducted in this study. The observations are fairly straightforward and lack deeper insights, making them less compelling. It would be much more engaging to see more unique insights drawn from the vast amount of experiments conducted.**\n\n**A:** Thank you for the valuable feedback. Due to space limitations in the main paper, we have provided more in-depth analyses and insights in Appendices B.5, B.6, and B.7, including detailed discussions on synergy across skills, modalities, and comprehension/generation dimensions.\nIn addition, we include fine-grained performance breakdowns for each model across individual skills, which allow for more precise diagnosis of model weaknesses and emerging capability trends. These detailed results offer a solid foundation for identifying underexplored areas and informing future research directions.\n\n In the revision, we will work to highlight additional non-obvious patterns and findings in the main text to better reflect the depth of our analysis.", "claims_and_evidence": "Strength:\n- This paper serves as a benchmark study, featuring an impressive number of tasks, diverse data domains, and a substantial number of evaluated models. I greatly appreciate the authors' extensive efforts in constructing this comprehensive evaluation benchmark.\n\n\nWeakness:\n- The five-level classification method lacks clarity, as it does not provide well-defined criteria for each level. In particular, the concept of \"synergy\" between different modalities is not clearly articulated, making the classification ambiguous. For instance, Unified-IO-2 and Next-GPT, which are any-to-any modality generative models capable of producing visual and audio outputs from multimodal inputs, are categorized as Level-2, suggesting they lack synergy in comprehension and generation. Meanwhile, DeepSeek-VL and LLaVA-One-Vision, which do not even possess visual generation capabilities, are classified as Level-3, implying they exhibit synergy across tasks. This inconsistency raises concerns about the validity of the classification framework, as it does not appear to align with the actual capabilities of these models. A more precise and well-justified definition of \"synergy\" is necessary to ensure the classification accurately reflects the models’ multimodal abilities.", "methods_and_evaluation_criteria": "- The evaluation criteria are well-founded; however, as previously mentioned, I disagree with the classification method applied to different MLLMs.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "The experimental design is thorough, encompassing a wide range of domains, tasks, data, and models.\n\nHowever, the analysis in Section 5.2 feels somewhat superficial, especially considering the large-scale experiments conducted in this study. The observations are fairly straightforward and lack deeper insights, making them less compelling.", "relation_to_broader_scientific_literature": "It is related to many works in multi-modal large language models.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No, the primary concern lies in the reasonableness of the classification method and the depth of the analysis.", "other_comments_or_suggestions": "It would be much more engaging to see more unique insights drawn from the vast amount of experiments conducted.", "questions_for_authors": "Please address my questions above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a new ​​General-Level framework​​ and the ​​General-Bench benchmark​​, a hierarchical evaluation system for multimodal large language models, evaluating across comprehension, generation, and cross-modal interactions. The benchmark, code, and evaluation framework will be open-sourced to support community advancements. Reviewers praised the work’s ​​ambitious scale​​, ​​innovative evaluation framework​​, and ​​transformative potential​​ for guiding MLLM development. Reviewers also suggested improvements in synergy definition clarity, computational practicality, and failure analysis depth. The authors addressed concerns by clarifying hierarchical scoring logic, proposing modular evaluation scopes, and committing to long-term maintenance. Despite minor issues, reviewers unanimously recognized the paper’s ​​technical contribution​​ and ​​community impact​​. Given the resolution of concerns and strong support for its contributions, the paper is ​​recommended for acceptance.
Decision: Accept (oral)