M3GIA: A Cognition Inspired Multilingual and Multimodal General Intelligence Ability Benchmark

Authors: Wei Song, Yadong Li, Xu Jhua, Guowei Wu, Lingfeng Ming, Kexin Yi, Weihua Luo, Yi Du, Fangda Guo, Kaicheng Yu
Primary area: datasets and benchmarks
Keywords: Benchmark, Multimodal, Multilingual, Cognitive

Abstract:
As recent multi-modal large language models (MLLMs) have shown formidable proficiency on various complex tasks, there has been increasing attention on debating whether these models could eventually mirror human intelligence. However, existing benchmarks mainly focus on evaluating solely on task performance, such as the accuracy of identifying the attribute of an object. Combining well-developed cognitive science to understand the intelligence of MLLMs beyond superficial achievements remains largely unexplored. To this end, we introduce the first cognitive-driven multi-lingual and multi-modal benchmark to evaluate the general intelligence ability of MLLMs, dubbed M3GIA. Specifically, we identify five key cognitive factors based on the well-recognized Cattell-Horn-Carroll (CHC) model of intelligence and propose a novel evaluation metric. In addition, since most MLLMs are trained to perform in different languages, we go beyond English to encompass other languages, including Chinese, French, Spanish, Portuguese and Korean, to construct our M3GIA. We make sure all the data relevant to the cultural backgrounds are collected from their native context to avoid English-centric bias. We collected a significant corpus of data from human participants, revealing that the most advanced MLLM barely reaches the lower boundary of human performance in English, and there remains a pronounced disparity in the other five languages. Importantly, we found that designing IQ tests for MLLMs is crucial, as the evaluation of M3GIA achieves a significantly stronger alignment with human preferences compared to traditional task-oriented benchmarks. Moreover, grounded in CHC theory, we discovered that the number of samples seen by the vision encoder has a greater influence on the model's visual capabilities than its parameter size.

Decision: reject
Reviewer scores: 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1729354278736

thread_type:
official_review

<thread_object>
creator: reviewer_1zrf

<content>
summary:
This paper introduces the concept of a cognitive-driven, multilingual, and multimodal benchmark to evaluate the general intelligence of MLLMs, referred to as M3GIA. The benchmark is grounded in the well-established Cattell-Horn-Carroll (CHC) model of intelligence and proposes a novel evaluation metric. It is open-sourced and aims to enhance the cognitive capabilities of MLLMs.

strengths:
1. The use of a taxonomy of cognitive abilities to evaluate the general intelligence of MLLMs is good, as it enables a more systematic evaluation.
2. The benchmark is constructed using unpublished offline data, which is a good practice to prevent data leakage.
3. The benchmark includes multiple language variants, allowing for the evaluation of MLLMs’ general intelligence across different languages.

weaknesses:
1. While the paper mentions that several specific factors from the CHC model of intelligence were selected (lines 237-250), it is unclear why these particular factors were chosen and how they relate to the general intelligence of MLLMs.
2. Although incorporating cognitive science into the evaluation of MLLMs is a positive step, the underlying tasks remain traditional, such as Math, Logo Problem, and Comic Problem. This may detract from the benchmark’s novelty. Given that recent works like MMMLU also include multilingual variants [1], it is not clear how M3GIA is fundamentally different from MMMLU.
3. The paper introduces numerous cognitive concepts and abbreviations, which may make it difficult for readers unfamiliar with cognitive science to follow. For instance, the meaning of “Fluid Reasoning (Gf)” (line 97) in the context of MLLMs is not clearly explained. In my personal aspect, I feel odd about the term "Fluid Reasoning (Gf)" what does it mean?

[1] https://huggingface.co/datasets/openai/MMMLU (Multi-Language Variant of MMMLU)

questions:
1. Is it possible to reorganize existing multi-task, multimodal benchmarks (e.g., MMMLU) to follow the taxonomy of cognitive abilities to evaluate the general intelligence of MLLMs? If not, could you explain why? Does the MMMLU benchmark lack specific tasks that would prevent it from capturing certain cognitive abilities?

[1] https://huggingface.co/datasets/openai/MMMLU (Multi-Language Variant of MMMLU)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644624595

thread_type:
official_review

<thread_object>
creator: reviewer_o6ax

<content>
summary:
This paper introduced a cognitive-driven multi-lingual and multi-modal benchmark, dubbed M3GIA, to evaluate the general intelligence ability of multi-modality large language models (MLLMs). Based on the Cattell-Horn-Carroll (CHC) model from cognitive science, the authors build a benchmark including 1.8K QAs annotated by native speakers in five languages. Experiments and analysis on 24 MLLMs show the significant disparities between MLLMs and human performance.

strengths:
- Based on the CHC theory, this paper brings a new perspective to the MM community for constructing multi-modal benchmarks aimed at evaluating modern MLLMs in terms of human-level intelligence. The background and taxonomy of the CHC theory are clear and meaningful.
  - As a benchmark, the multi-modal QAs annotated by humans are of high quality and useful.
  - The evaluation of both open-source and closed-source MLLMs is extensive and thorough.

weaknesses:
- Though starting from a new perspective of the CHC theory, this paper still evaluates the widely adopted capabilities of MLLMs that have been investigated in previous benchmarks, such as Visual-Spatial Processing, Knowledge, Math Facts, and Text Reading. For example, the MM-vet benchmark builds QAs related to the capabilities of OCR, Math, Knowledge, and Language Generation, using LLMs as examiners to evaluate open-ended generations. The performance of MLLMs in Table 1 also demonstrates a consistent trend between M3GIA and other general multimodal benchmarks, rather than revealing distinct findings.
  - This paper spend extensive content to introducing the CHC model within the main content. However, one point still remains unclear to me: how does the CHC model affect the capabilities of MLLMs? In other words, what specific attributes or behaviors would a powerful MLLM, grounded in CHC theory, exhibit? Are there any case studies or pilot experiments that illustrate the significance of this influence?
  - The paper is missing detailed statistical information about the proposed benchmark, such as the number of images per category and the average number of words in the generated questions.
  - The paper’s experimental section appears to be incomplete due to the absence of results for the few-shot setting.

questions:
- For the Human Performance Baseline, I believe that these results are important for reflecting the difficulty of the created benchmark. What are the educational levels of the participants, and how is the quality of the created questions ensured?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675298115

thread_type:
official_review

<thread_object>
creator: reviewer_5fwl

<content>
summary:
This paper presents a benchmark M3GIA which claims to act as the first “IQ test” for multimodal large language models (MLLM). It is built based on five cognitive factors from the Cattell-Horn-Carroll Model of Intelligence. It includes VQA/text-format questions from tasks like oral vocabulary, concept formation, visualization, math, reading and etc. Besides English, it also includes other languages such as Chinese, French, Spanish, Portuguese and Korean. The authors evaluate their benchmark on a number of API-based and open-source models across different scales as well as human participants. They observe that the best MLLM (GPT4-o) can reach the lower boundary of human performance in English.

strengths:
The paper presents an interesting perspective for constructing benchmarks and suggests we can design benchmarks based on previous cognitive science studies. The contribution of the new resources can be helpful and raise more questions and considerations about benchmark design. They also provide an initial performance analysis of some of the existing models, which can be used as a reference for future research.

weaknesses:
While the authors claim that M3GIA can serve as an IQ test for MLLMs and have built this benchmark based on existing cognition theory, I find it hard to conclude generally that “most advanced MLLM reaches the lower boundary of human intelligence in English”. There are many different categories of questions collected in this benchmark and they can fall under different cognitive factors. However, it is unclear what control factors are in place during the data collection and evaluation process: why this specific type of question is chosen? How are the variances of questions controlled across languages? How broad/narrow is the topic tested in each domain? What are the sample demographics of the annotators? Given there are only 300 questions tested per language, it’s hard to prove that the human responses collected represent the lower bound of human intelligence.

questions:
- Can you provide more details about how you decide the question category under each factor and how is each question selected for each category? Is there any data filtering or quality inspection process from experts to determine whether each question is easy/hard enough to be included?
- How does this dataset differ from other reasoning benchmarks besides it is “cognition-inspired”? If the importance lies in its originality, why do you also include datapoints from other datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732010354448

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Reviewers and AC,

Thank you all for your time and effort in reviewing our paper. We appreciate that reviewers:
1. Found our benchmark **presents an interesting perspective on benchmark construction** (5FWL) and brings a novel perspective for the MM community to design benchmarks aimed at evaluating modern MLLMs in terms of human-level intelligence (o6ax). 
2. **Recognized the high quality and usefulness of our human-annotated multimodal QA data** (o6ax), and appreciated our practice of **using unpublished offline data to construct the benchmark to prevent data leakage** (1Zrf).
3. Found the background and taxonomy of the CHC theory clear and meaningful (o6ax), and **the use of the taxonomy is meaningful**, as it enables a more systematic evaluation (1Zrf).
4. Appreciated our practice of **including multiple language variants.** (1Zrf)
5. Recognized the evaluation of both open-source and closed-source MLLMs as extensive and thorough (o6ax).

We also thank 5FWL for recognizing the contribution of the new resources can be helpful and raise more considerations about benchmark design.

We are addressing each of your questions in the individual responses. Here we would like to emphasize our uniqueness compared to other benchmarks:

> **How does M3GIA essentially differ from other reasoning benchmarks?**

As discussed in 081-092, other reasoning benchmarks like mmbench, mme, TextVQA, HallusionBench, mathvista, and SeedBench are task-oriented, focusing on one or several specific applied tasks. *Their objective is to evaluate model performance on practical application tasks themselves* (e.g., object attribute recognition, OCR, chart-solving, etc.). 

Their approach faces a significant issue that cannot be ignored: **they struggle to provide a justified answer to "why these specific ability dimensions were chosen for evaluation" as their selection of ability dimensions is subjective and lacks a solid cognitive science basis.**

**In contrast, M3GIA does not primarily care about the model's performance in any specific application capability itself. Instead, our questions directly points to the definitions of the CHC factors**, reflecting the model’s performance across these dimensions of human cognition. The metrics it assesses directly correspond to relevant cognitive abilities, unlike other benchmarks where a given task might require multiple, interwoven cognitive abilities that are difficult to decouple.

We would also like to emphasis that, our cognitive-based question types, such as Concept Formation, Visualization, Picture Recognition, Syllogism Problem have not been systematically featured or examined in previous benchmarks. In other words, **no previous benchmark has systematically included and assessed these cognitive-based question types within a single set of test questions.**

Thanks again for all the effort and time.

Best,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a broad benchmark to test models in different cognitively inspired tasks across modalities and skills.

Strengths:  
Provides reasoning for validity.  
Raises discussions

Weaknesses:
Small number of examples (perhaps a reliability test may account for it)
Possibly, overclaiming given the evidence  
Missing data curation explanations  
The relation to other benchmarks raised several questions (perhaps benchmark agreement testing and showing how the results and underlying traits captured differ and not only the motivation might help, [this](https://github.com/IBM/benchbench) tool and data might help)

The paper seems to provide an interesting contribution which I encourage the authors to revise and resubmit to a parallel venue.
Still, it got a lot of feedback that should merit an improved version.
Generally, ICLR calls for major changes during the rebuttal period considering those peer reviewed (despite it clearly not being the case). Here, they were not and as those are many. It is worth another round of peer review and revisions and evaluation to check the new state of the paper and experiments.