MCTBench: Multimodal Cognition towards Text-Rich Visual Scenes Benchmark

## Abstract

Abstract The comprehension of text-rich visual scenes has become a focal point for evaluating Multi-modal Large Language Models (MLLMs) due to their widespread applications. Current benchmarks tailored to the scenario emphasize perceptual capabilities, while overlooking the assessment of cognitive abilities. To address this limitation, we introduce a M ultimodal benchmark towards T ext-rich visual scenes, to evaluate the C ognitive capabilities of MLLMs through visual reasoning and content-creation tasks ( MCTBench ). To mitigate potential evaluation bias from the varying distributions of datasets, MCTBench incorporates several perception tasks (e.g., scene text recognition) to ensure a consistent comparison of both the cognitive and perceptual capabilities of MLLMs. To improve the efficiency and fairness of content-creation evaluation, we conduct an automatic evaluation pipeline. Evaluations of various MLLMs on MCTBench reveal that, despite their impressive perceptual capabilities, their cognition abilities require enhancement. We hope MCTBench will offer the community an efficient resource to explore and enhance cognitive capabilities towards text-rich visual scenes.

## 1 Introduction

Multimodal Large Language Models (MLLMs) [, , , ] have exhibited promising performance across various cross-modal tasks, and revealed potential for widespread real-world applications. In practical applications, many images contain crucial textual elements that are essential for addressing specific challenges, such as key information extraction from receipts. Consequently, the ability to comprehend text-rich visual scenes can significantly enhance the practicality of MLLMs and drive innovative applications across multiple domains.

Recent benchmarks [, , ] have increasingly focused on evaluating MLLMs towards text-rich visual scenes. Nonetheless, the benchmarks are centered around evaluating perceptual capabilities yet overlook the assessment of cognitive abilities, which are a significant strength of MLLMs (as illustrated in Figure 1).

In this paper, we propose a Multimodal benchmark to evaluate the Cognitive capabilities of MLLMs in Text-rich visual scenes (MCTBench). To assess the cognitive abilities of MLLMs thoroughly, we design two types of tasks in the MCTBench: reasoning tasks for comprehension of the in
put scenes, and open-ended content-creation tasks for generating output responses.
Besides, MCTBench integrates various perception tasks to study the differences with cognition tasks, while avoiding evaluation biases from varying dataset distributions.
Fundamentally, MCTBench curates approximately 5.2k text-rich images from a wide range of public datasets, along with 8.5k rigorously annotated question-answer pairs categorized into three tasks: perception, reasoning and content-creation. The perception and reasoning tasks are formatted as multiple-choice questions for convenient evaluation, following common practices in [, , ].
Due to the subjectivity and high cost of human evaluation in open-ended content creation, we establish an automated evaluation pipeline by leveraging sophisticated MLLMs (e.g., GPT-4V) as the evaluator, to compare the predictions of models against the provided references. Our experimental results demonstrate that MLLMs exhibit notably lower performance of cognition capabilities compared to perception in text-rich visual scenes, particularly for text-enhanced models. Furthermore, performances in cognition tasks (reasoning and content-creation) are improved with larger parameter scales.

Figure: Figure 1: The Comparison between previous Benchmarks [, , ], and our proposed MCTBench. Q and GT stand for question and ground truth.
Refer to caption: x1.png

Our main contributions are summarized as follows:

- 1.
We propose a brand-new and large-scale benchmark for evaluating the cognitive capability of MLLMs towards text-rich visual scenes.
- 2.
The evaluation on MCTBench highlights that MLLMs necessitate enhancements in their cognitive capabilities in text-rich visual scenes.
- 3.
We develop an automated evaluation pipeline for the content-creation task, offering researchers an efficient tool for further investigation of cognitive capabilities.

## 2 Related Work

### 2.1 Multimodal Large Language Models

The significant advancements in Large Language Models (LLMs) [, , ] have paved the way for recent research [, , , , ] into developing Multimodal Large Language Models (MLLMs) that integrate visual capabilities. Early works in this field [, , , ] have introduced various vision-language projectors such as Q-former[], Multi-Layer Perceptron (MLP) [], and Perceiver[], which act as intermediaries between LLMs and visual encoders. Furthermore, these efforts have also established robust training paradigms for MLLMs. Building upon these foundational paradigms, recent initiatives [, , , ] have focused on scaling the quality of training data, to enhance general visual capabilities effectively.

A primary challenge in the recent development of MLLMs is attaining fine-grained comprehension, exemplified by tasks such as Visual Question Answering (VQA) on text-rich images.
To address this issue, increasing the resolution and integrating fine-grained visual features have been proven effective across various studies [, , , , , , , ]. Additionally, works such as [, , , , ] have incorporated high-quality, text-rich visual tuning data to refine these models further.

### 2.2 MLLM Benchmarks

As multimodal large language models (MLLMs) continue to exhibit cross-task generality, single-task evaluations (e.g., [, , , ]) are inadequate for a comprehensive performance assessment. Recent works [, , , ] present general MLLM benchmarks comprising multiple tasks. Furthermore, to explore the performance of MLLMs on more complex tasks, MathVista [] evaluates their mathematical abilities, and MMMU [] integrates multiple-discipline questions to benchmark MLLMs in expert domains.

Conversely, text-rich visual scenes are attracting growing attention due to their potential applications. Early works [, , ] focused on single tasks, while OCRBench [] integrates multiple single-task datasets into five representative OCR(Optical Character Recognition)-based tasks. In contrast, our work evaluates MLLMs on complex tasks beyond OCR-based ones in text-rich visual scenes. A similar work is presented in [], which demonstrates the model’s performance on reasoning tasks but only on a limited set of test datasets. Our study provides a broader evaluation of cognition in text-rich visual scenes, pushing the boundaries of what MLLMs can achieve in more diverse scenarios such as content-creation. Table 1 demonstrates the detailed comparison between ours and previous benchmarks.

**Table 1: The comparison between MCTBench and previous benchmarks. Open and MC respectively present open-ended and multiple choice format for answer type. QAs stands for question-answer pairs. Text-Rich Oriented indicates whether the benchmark focuses on text-rich visual scenes.**
| Benchmark | Text-Rich<br>Oriented | #Image | #QAs | Percetion | Reasoning | Content<br>Creation | Answer<br>Type |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MME [] | ✘ | 1137 | 2.2K | ✔ | ✔ |  | Yes/No |
| MMBench [] | ✘ | - | 3K | ✔ | ✔ |  | MC |
| OCRBench [] | ✔ | 450 | 1K | ✔ |  |  | Open |
| SEED-bench-2-plus [] | ✔ | - | 2.3K | ✔ |  |  | MC |
| Contextual [] | ✔ | 506 | 506 | ✔ | ✔ |  | Open |
| MMMU [] | ✔ | - | 11.5K | ✔ | ✔ |  | MC/Open |
| MCTBench | ✔ | 5.2K | 8.5K | ✔ | ✔ | ✔ | MC/Open |

## 3 MCTBench

In this section, we outline the process of constructing the MCTBench. Section 3.1 provides an overview of MCTBench and compares it with previous benchmarks. Section 3.2 describes the procedure of collecting text-rich image sources from publicly accessible datasets. Finally, Section 3.2 explains the annotation process applied to the collected images.

Figure: Figure 2: The pipeline of constructing MCTBench.
Refer to caption: x2.png

### 3.1 Overview

The MCTBench is designed to evaluate the cognitive capabilities of Multimodal Large Language Models (MLLMs) towards text-rich visual scenes. To construct the comprehensive and diverse benchmark, we collected 5,194 images from a variety of public datasets, encompassing a wide array of text-rich scenes such as natural environments, books, scientific contexts, advertisements, e-commerce, and video shots.
We meticulously annotate these images with a total of 8.5k question-answer pairs categorized into three tasks: perception, reasoning, and content creation. Specifically, MCTBench consists of 2,734 perception multiple-choice samples, 2,602 reasoning multiple-choice samples, and 3,130 content-creation samples. Figure 2 illustrates the overall construction pipeline of MCTBench.

### 3.2 Text-rich Images Collection

#### Image Source

The images of MCTBench are collected from 10 different publicly available datasets, aiming to incorporate comprehensive visual scenes to evaluate the cognition of Multimodal Large Language Models (MLLMs). We begin with sampling common general natural scenes (e.g., street views, competitions, road signs) from the COCO [], Flickr30k [], GQA [], SeedBench [] and Visual Genome [] datasets.

Furthermore, we select conventional text-rich multimodal datasets: OCR-VQA [], and VizWiz [] taken by blind photographers.
To further diversify MCTBench, we incorporate three domain-specific scenes with broad potential applications: advertising (AutoUnderAds []), e-commerce (FoodLogoDet-1500 []), and science (ScienceQA []). We randomly extract one frame from each video in the AutoUnderAds dataset. All data sources are specifically selected from the testsets. We adhere to the original licenses stated by all datasets.
OCR-based image filtering. We select the text-rich images from the sourced images, which are guided by the following guidelines.
To maintain the clarity and substantive textual content, we only retain images with valid OCR-recognized characters (with recognition probabilities higher than 0.2) of at least 10 characters.

To ensure text contributes to overall visual semantics, we select images where text regions occupy more than 10% of the image area, after validating the impact of valid text lines on semantic expression.
These meticulous selection criteria resulted in a curated collection of high-quality and crystal-clear text-rich images, designed to challenge and inspire advancements in perceptual and cognitive understanding within textual domains.

#### Annotation

Considering the bias and efficiency of the manual annotation, we employ a GPT-aided approach to generate at least 10 pseudo-questions for each image, and ask annotators to remove low-quality ones.
All answers are human-annotated with two rounds:
(1) Each image with at least 10 GPT-aided pseudo-questions, is randomly assigned to three annotators. Each annotator independently annotates the questions and provides answers.
(2) Quality checkers will review the annotation in the first round. If any question or image quality does not meet our annotation guidelines, the set is re-annotated by the corresponding annotators, who also revise their answers.
Annotators in the second round are required to have at least 2 years of experience in text-rich multimodal scene annotation.
To reach an agreement, we use majority voting to determine the final answer. If majority voting does not reach an agreement (i.e., all three answers are inconsistent), we check if the discrepancy originated in the second round. If so, the question is re-annotated; if inconsistencies persist, it is discarded. The question is discarded if the discrepancy does not arise in the second round.
In addition to content-creation tasks, due to the inherent diversity of responses, we do not provide unified answers. Instead, we offer standard references generated by powerful MLLMs (e.g., GPT-4V) and meticulously reviewed by humans.

#### Quality control

During the image quality assessment, annotators remove low-quality images (e.g., blurry, unclear text, inappropriate, solely tables/documents, and only watermark). For the QA quality assessment, annotators eliminate low-quality questions (e.g., ambiguous, overly generalized, too simplistic) and check the correctness of annotated answers (e.g., logical errors). On the other hand, We filter out multiple-choice options with more than 30% word count disparity and remove the questions that GPT-4 refuses to answer due to ethical concerns. To detail the specific issues addressed by each type, we assigned a label to each question and visualized the terms in Figure 3.

### 3.3 Data Construction

Figure: Figure 3: Visualization of the question for three different tasks using word clouds. In the word cloud, the size of a word indicates how frequently it appears. Best viewed in color.
Refer to caption: x3.png

## 4 Experiments

In this section, we conduct comprehensive experiments to evaluate current MLLMs. Firstly, we outline experimental settings for the evaluated models and metrics used on MCTBench in Section 4.1 and 4.2. The results obtained from these experiments allow us to perform analysis of the selected models across three categories of tasks in Section 4.3. Furthermore, we conduct a case study to investigate performance variations among diverse models in Section 4.5.

### 4.1 Models

To evaluate the performance of various MLLMs on MCTBench, we select a diverse range of models categorized into two primary types: general MLLMs and text-enhanced MLLMs (specifically optimized for text recognition in images). We establish two naive baselines (random choice and frequency choice) as reference points to ensure the robustness and validity of the dataset in Table 2. Random choice involves selecting an option at random as the prediction result, while frequency choice involves selecting the prediction result based on the option with the highest proportion in the ground truth.

#### General MLLMs

The experiment initially evaluates popular closed-source models, specifically Gemini-Pro [] and GPT-4V(ision) []. For open-source models, we select several notable general-purpose MLLMs, including Sharegpt4V [], Honeybee [], LLaVA [, ], Otter [], Yi-VL [], Qwen-VL-chat [] and Deepseek-VL [] as competitive baselines. Additionally, we incorporate CogVLM [] and SPHINX-v2 [], which are enhanced for fine-grained understanding. To assess differentiated performance, we also integrate a larger open-source model, Mini-Gemini [].

#### Text-enhanced MLLMs

Recent researchers propose remarkable works to tackle the understanding of text-rich images via enhancing the textual capabilities. Consequently, we select models mPLUG-DocOwl [], Monkey [, ], InternLM-XComposer2-VL [], CogAgent [] and LLaVA-NeXT [] which have demonstrated strong OCR capabilities in previous evaluations.

### 4.2 Metrics

MCTBench is constructed with three categories of tasks: perception, reasoning, and content-creation. Due to the standard answer provided in perception and reasoning tasks, each QA pair is designed in a multiple-choice format. In contrast, the content-creation task is considered as an open-ended generation problem due to the diversity of responses.

#### Perception and reasoning

Perception and reasoning tasks entail the acquisition of information from input data, comprehension of images and text, and derivation of conclusions. Consequently, employing multiple-choice question-answering can effectively validate the corresponding capabilities of MLLMs, following [, ].
In practice, we prioritize original prompts employed by MLLMs and, if not specified, use those prompts which yield optimal results. For lengthy responses, we use regular expressions and supplementary rules to extract the option answers.
We use mean accuracy to evaluate MLLMs’ perception and reasoning capability.

#### Content-creation

To ensure consistency and efficiency in evaluation, we implement automatic evaluation for content-creation tasks. However, due to the diversity of answers in content creation, standard responses are not feasible, leading us to establish competitive references.
These references are generated through manually crafted responses from text-only GPT-4, based on inputs from OCR recognition and detailed descriptions.
The evaluation is grounded on four principal aspects: relevance, faithfulness, creativity, and instruction following. Subsequently, we employ machines (e.g., GPT-4V) to compare other MLLMs against our references with the mentioned principal, and categorize their performance as Good, Same, or Bad (i.e., the GSB metric). We measure each model’s performance by calculating the percentage of ’Good’ and ’Same’ ratings relative to all questions, indicating how many outperform or match the constructed reference (i.e., $(G+S)/(G+S+B)$).
To assess correlations and validate the reliability of machine evaluation, we also conduct a manual evaluation on subsets of the data using the mentioned metrics (GSB) and compare them with machine evaluation. We integrate three powerful MLLMs (GPT-4V [], Gemini-Pro [] and LLaVA-NeXT []) as evaluators. We measure the evaluation correlation between humans and machines using accuracy and Pearson correlation [] on GSB scores. Table 3 illustrates the results between three machine evaluators and human evaluations, indicating that GPT-4V achieves a top correlation score.

### 4.3 Results

**Table 2: Evaluation results for MLLMs on MCTBench. MC means the average score of the two tasks (perception and reasoning) in multiple-choice format. Cog means the average scores of the two cognitive tasks (reasoning and content-creation). All means the overall average scores of all tasks. ^∗The content-creation task is scored using the percentage of ’Good’ and ’Same’ ratings by the GSB metric except for accuracy used in Perception and reasoning tasks. Numbers in Bold and underline represent the top-2 results in each task.**
|  |  |  | Cognition | Average Scores |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Params | Perception | Reasoning | Content-<br>Creation^∗ | MC | Cog | All |
| Naive Baseline |  |  |  |  |  |  |  |
| Random choice | - | 25.00 | 25.00 | - | - | - | - |
| Frequency choice | - | 25.16 | 25.52 | - | - | - | - |
| General MLLMs |  |  |  |  |  |  |  |
| GPT-4V [] | - | 83.58 | 74.21 | 87.35 | 78.90 | 83.12 | 81.71 |
| Gemini-Pro [] | - | 78.79 | 70.18 | 56.78 | 74.49 | 65.63 | 68.58 |
| Yi-VL [] | 6B | 77.25 | 72.33 | 41.45 | 74.79 | 58.12 | 63.68 |
| Deepseek-VL [] | 7B | 76.74 | 68.79 | 57.25 | 72.77 | 65.01 | 67.59 |
| Honeybee [] | 7B | 72.60 | 67.22 | 73.64 | 69.91 | 71.78 | 71.15 |
| Otter [] | 7B | 58.12 | 54.42 | 31.70 | 56.27 | 43.99 | 48.08 |
| Qwen-VL-chat [] | 7B | 77.98 | 70.68 | 67.53 | 74.33 | 70.93 | 72.06 |
| Sharegpt4V [] | 13B | 74.54 | 69.49 | 66.19 | 72.02 | 69.10 | 70.07 |
| LLaVA-1.5 [] | 13B | 78.09 | 72.56 | 66.47 | 75.33 | 70.90 | 72.37 |
| SPHINX-v2 [] | 13B | 78.02 | 71.94 | 62.30 | 74.98 | 68.64 | 70.75 |
| CogVLM [] | 17B | 71.40 | 69.52 | 65.61 | 70.46 | 68.04 | 68.84 |
| Mini-Gemini [] | 34B | 83.83 | 73.33 | 86.76 | 78.58 | 82.67 | 81.31 |
| Text-enhanced MLLMs |  |  |  |  |  |  |  |
| IXC 2 [] | 7B | 78.05 | 72.10 | 74.45 | 75.08 | 74.76 | 74.87 |
| Monkey [] | 7B | 79.22 | 72.64 | 59.56 | 75.93 | 67.75 | 70.47 |
| TextMonkey [] | 7B | 71.80 | 69.45 | 22.81 | 70.63 | 46.72 | 54.69 |
| mPLUG-DocOwl [] | 10B | 75.05 | 70.06 | 60.87 | 72.56 | 66.71 | 68.66 |
| CogAgent [] | 34B | 58.56 | 56.46 | 56.86 | 57.51 | 57.19 | 57.29 |
| LLaVA-NeXT [] | 34B | 83.87 | 71.64 | 85.30 | 77.76 | 81.53 | 80.27 |

#### Perception

Firstly, experiments are conducted to verify the perceptual performances of each model as baselines. As shown in Table 2, most models achieved satisfying scores on the perception task. Closed-source models (e.g., GPT-4V []) demonstrated excellent accuracy, while some open-source models (e.g., Mini-Gemini []) demonstrated superior perception capabilities, surpassing the their performance. Models with higher resolutions and more parameters (e.g., LLaVA-NeXT [] and Mini-Gemini []), typically performed better. Among similarly-sized models, text-enhanced MLLMs generally outperformed others by effectively extracting text from images and generating precise responses.

#### Reasoning

The reasoning task is more challenging than perception.
It requires not only effective extraction and fusion of visual and textual features, but also involves comprehensive inference to generate accurate responses.
As shown in Table 2, most models exhibited a significant drop in scores due to the increased difficulty of reasoning tasks compared to perception tasks.

Notably, GPT-4V demonstrates exceptional performance among MLLMs, surpassing most models by a significant margin.
Besides, there still exists a positive correlation between a model’s performance and the number of its parameters.
This phenomenon arises from larger models’ enhanced ability, to comprehend text and integrate image information for reasoning more effectively.

Nevertheless, text-enhanced MLLMs have not substantially outperformed general models on reasoning tasks.
Given that text-enhanced models have achieved superior results in perception tasks, we posit their effectiveness in recognizing text within images.
However, achieving higher scores in reasoning tasks necessitates the ability to analyse and summarise effectively.
TextMonkey [] shows the least performance gap and achieves results comparable to the perception task. However, most text-enhanced models are not explicitly trained in this aspect, and consequently do not outperform general MLLMs.

**Table 3: Correlation analysis between automatic machine and human evaluation on content-creation using the accuracy and Pearson correlation coefficient.**
|  | GPT-4V | Gemini-Pro | Best open-source<br>(LLaVA-NeXT) |
| --- | --- | --- | --- |
| Accuracy | 79.38 | 70.71 | 65.22 |
| Pearson Correlation | 0.558 | 0.380 | 0.304 |

#### Content-creation

The open-ended creation task differs from the aforementioned tasks, leading to the observation of a broader array of perspectives.
As mentioned in Section 4.2, the evaluation is based on four principal aspects: relevance, faithfulness, creativity, and instruction following.
Therefore, as the content-creation task emphasizes the generation of suitable text for images, we also notice significantly enhancement on performances by employing larger language models.
However, the performance of text-enhanced MLLMs exhibits considerable variation due to differences in training objectives.
Some models are specifically trained to extract structured text information or comprehend lengthy text inputs.
Consequently, they may underperform relative to general models when tasked with creative endeavours.
We also found that the closed-source model Gemini-Pro [] is unable to achieve good results.
A notable discrepancy in its scores of template following demonstrates that Gemini struggles to create corresponding formats for task specifications.
For instance, in the task of generating a slogan, Gemini may tend to produce lengthy paragraphs instead of concise phrases.
Some text-enhanced MLLMs, such as TextMonkey, also have similar phenomena.

#### Summary

MLLMs have shown certain multi-modal capabilities in perception tasks, with mainstream models achieving commendable performances. However, in reasoning tasks, existing models still have room for improvement. Performance on content-creation tasks indicates that text-enhanced MLLMs trained for different types of tasks may lose some creative capabilities. We suggest that reasoning and creation tasks serve as distinct dimensions for evaluation, offering insights into the model’s comprehension of input and its proficiency in generating output responses. While existing models excel in basic perception tasks, achieving comparable competence in reasoning and creation tasks remains challenging.

**Table 4: The effect of visual and textual information.**
| Input | Perception | Reasoning | Creation |
| --- | --- | --- | --- |
| Image + OCR texts | 75.60 | 71.68 | 65.45 |
| Image (text regions removed) | 62.22 | 62.22 | 46.89 |
| Image (text regions only) | 74.69 | 68.01 | 58.78 |
| Image (baseline) | 78.09 | 72.56 | 66.47 |

### 4.4 Ablation Study

#### The effect of visual and textual information.

In Table 4, we conduct several experiments to reveal the effect of visual and textual information in images of our benchmark.
We use the LLaVA-1.5 as the baseline.
Firstly, we conduct an experiment that adds OCR texts as input.
As shown in the table, even if the OCR texts are contained as input, the model cannot gain explicit improvement. This demonstrates that the MCTBench does not rely solely on texts to get answers.
Furthermore, we conduct an experiment that removes all texts from images to check the importance of texts in our benchmark.
Specifically, we detected and blurred all the texts in images. We noticed that the performance dropped significantly. This also demonstrates that it is difficult for a model to correctly answer the question without text in the image.
Additionally, if we only keep the text region, and delete other background parts in the image. The performance is between the above two experiments, indicating that MCTBench relies both on textual and visual information, to get the final result. To conclude, explicitly adding OCR texts, or removing text/image parts does not help, or even lead to worse performance on MCTBench. MLLMs are required to jointly recognize related textual and visual patterns, to answer the questions correctly.

Figure: Figure 4: The cases of predication from different MLLMs divided into four groups: GPT-4V [], Mini-Gemini [](MGM) and LLaVA-NeXT [] for larger model size, Monkey [] and mPLUG-DocOwl [] for text-enhanced MLLMs, LLaVA-1.5 [] and ShareGPT4V [] for the general MLLMs
Refer to caption: x4.png

Figure: Figure 5: The cases of predication on content-creation tasks from three representative MLLMs: GPT-4V [], Mini-Gemini [] and Monkey []. We mark high-quality sentences in red, words hit the text in the image with underlining, and rate the quality of the generation with stars.
Refer to caption: x5.png

### 4.5 Case Study

For the perception and reasoning tasks, we select representative cases shown in Figure 4. Specifically, we select GPT-4V as a strong reference along with several open-source representative MLLMs, split into three groups: large model size and resolution MLLMs (Mini-Gemini [] and LLaVA-NeXT []), text-enhanced MLLMs (Monkey [] and mPLUG-DocOwl []), and general MLLMs (LLaVA-1.5 [] and ShareGPT4V []).
For the selected perception question, both high-resolution MLLMs and text-enhanced MLLMs perform well, while general MLLMs fail in fine-grained understanding. On the contrary, text-enhanced MLLMs excelled in perception but performed poorly in reasoning tasks. Larger models like Mini-Gemini [], LLaVA-NeXT [], and GPT-4V [] handle reasoning better by effectively integrating textual and visual elements.

For the content-creation task, using GPT-4V [] as a robust reference, we select general and text-enhanced models that performed well on MCTBench, and conduct case studies in three scenarios. As shown in Figure 5, GPT-4V [] significantly surpassed other models in content creation quality. Mini-Gemini [] also showed consistent performance across cognitive tasks, while text-enhanced models like Monkey [] were limited to text recognition and basic descriptions, resulting in less attractive content.

## 5 Limitations

Our dataset primarily focuses on English, which may limit the generalization of our findings to multilingual scenes.
Although we believe that the cognitive capacities of MLLMs should theoretically extend to other languages, we have not empirically substantiated this assertion in the present study.
Additionally, we have only selected a subset of representative models for evaluation due to space constraints. This selection may not cover the full spectrum of currently available MLLMs. Our future work aims to provide evaluation results for a more extensive range of models.

## 6 Conclusion

In this work, we introduce MCTBench, a comprehensive benchmark designed to evaluate the cognitive capabilities of MLLMs in text-rich visual scenes. The MCTBench comprises 5.2k images and 8.5k question-answer pairs, covering a range of tasks including reasoning, content creation for cognitive assessment, and conventional perception. Evaluations of MLLMs on MCTBench reveal that current MLLMs still need further advancements in cognitive capabilities, despite their superior perception performance. We hope that MCTBench will motivate researchers to further improve the cognitive capabilities of MLLMs in text-rich visual scenes, thereby enhancing the practical utility of AI in real-world applications.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1) Complete the missing section 3.3 in the paper. What is the specific data construction process in Section 3.3? Can you provide more details about data stratification, annotation, and preprocessing so that other researchers can replicate the MCTBench data preparation process?
2) Can cognitive abilities be further subdivided beyond reasoning and content generation? For example, is it possible to incorporate more refined categories such as logical reasoning, contextual understanding, and cross-modal reasoning to more accurately assess the cognitive abilities of different models?
3) How consistent are automated rating systems with human ratings? Can comparative experiments be conducted to analyze the reliability of automatic scoring in different types of generation tasks and clarify its accuracy in different evaluation dimensions.
soundness: 2
strengths: 1）MCTBench provides a thorough assessment of both reasoning and content-creation capabilities in MLLMs, offering a well-rounded evaluation framework.
2）By using advanced MLLMs for automated evaluation, the benchmark reduces the need for costly and subjective human assessments in content creation tasks.
3）By distinguishing between perception and cognitive tasks, MCTBench helps identify specific areas where MLLMs need improvement.The finding that larger models perform better in cognitive tasks provides valuable guidance for future model development and scaling strategies.
summary: The paper introduces MCTBench, a benchmark for evaluating the cognitive abilities of multimodal large language models (MLLMs) in text-rich visual scenes. MCTBench includes two main task types: reasoning tasks for understanding scenes and open-ended content-creation tasks for generating responses. It also incorporates perception tasks to differentiate them from cognitive tasks, minimizing bias from dataset variations.
The benchmark compiles about 5.2k images and 8.5k annotated question-answer pairs across three categories: perception, reasoning, and content creation. Perception and reasoning tasks use multiple-choice formats for easy assessment, while an automated evaluation system, using advanced MLLMs like GPT-4V, is set up for content creation due to the challenges of subjective human evaluation.
weaknesses: 1）Incomplete paper with no content in section 3.3
2）Segmenting cognitive abilities into reasoning and content generation may not be enough, and a sufficiently fine-grained benchmark would require a more precise segmentation of the data
3）Automated evaluations have improved efficiency, but their accuracy and consistency with manual evaluations need further validation

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: In Table 4, the “Image (text regions removed)” row shows a perception score of 62.22. Given that this benchmark is designed for text-rich scenes, one would expect perception tasks to be highly challenging, if not impossible, without text information. Could you clarify:
1. How was this score achieved despite the absence of text?
2. If accurate, does this suggest that certain benchmark questions may not fully align with the intended text-rich focus?
soundness: 2
strengths: 1. This paper collect a large-scale benchmark for evaluating the cognitive capability for MLLM, where reasoning and content-creation ability is highlighted
2. For the content-creation task, an automated evaluation pipeline is introduced to enhance efficiency.
summary: This paper introduced MCTBench, a comprehensive benchmark designed to evaluate the cognitive capabilities of MLLMs in text-rich visual scenes. The MCTBench comprises 5.2k images and 8.5k question-answer pairs, covering a range of tasks including reasoning, content creation for cognitive assessment, and conventional perception.
weaknesses: 1. The paper introduces Content Creation as a new evaluation component, but it could benefit from a clearer explanation of the necessity and value of this addition for assessing cognitive abilities. Furthermore, the rationale behind dividing cognitive tasks into “reasoning” and “content creation” would be strengthened with additional justification for this categorization.
2. The paper suggests that MLLMs require improvements in cognitive capabilities within text-rich visual scenes. However, the results presented do not entirely support this conclusion, as cognitive scores do not show a substantial decrease compared to perceptual scores. Since cognition often builds on perception, the separation of these tasks across different data samples may seem too rigid. Evaluating perception and cognition on the same images could better capture their relationship and provide clearer insights into how MLLMs leverage perceptual understanding for reasoning.
3. The automatic evaluation approach could be better supported by a further improvement and a comparison to prior evaluation methods. Specifically, with a Pearson correlation of only 0.558 against human judgment, this score may be insufficient to fully validate the reliability of the automated approach. A higher correlation score would likely provide stronger validation.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 1
questions: 1. The paper is poorly written, with many citation format errors. Section 3.3 is incomplete, and on page ten, there is a figure inserted in the middle of the references section. Additionally, there is no appendix provided.
2. Many details are not clearly explained. For example, the content-creation task lacks sufficient explanation, and the prompts used are not detailed.
3. While the paper claims to "provide a broader evaluation of cognition in text-rich visual scenes," this is only reflected in the word cloud, lacking other relevant support. For instance, under the reasoning task, it is unclear how OCR capabilities are subdivided into fine-grained categories, nor is there a comparison with other benchmarks at a fine-grained level. It also remains unclear which aspects are covered by existing benchmarks and which are not, and if there are any additional examples for OCR abilities not covered by current benchmarks.
soundness: 2
strengths: 1. This paper broadens the scope of OCR ability of MLLMs, rather than  conventional OCR tasks and current MLLM benchmarks. 
2. The benchmark is large-scale and human-annotated, make the benchmark valid and reliable.
summary: This paper present MCTBench, a new multimodal benchmark designed to evaluate the cognitive abilities of MLLMs through visual reasoning and content creation tasks. MCTBench includes perception tasks and employs an automated evaluation process for content creation, revealing that while MLLMs exhibit strong perceptual skills, their cognitive abilities need improvement. This benchmark aims to provide a valuable tool for the community to advance cognitive capabilities in processing text-rich visual scenes.
weaknesses: 1. Paper is poorly formatted.
2. Paper lacks details.
See questions below.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 1
questions: 1. In Table 2, the performance difference between strong models, like GPT-4V and weaker models, like LLaVA1.5-13B is minimal on reasoning tasks. What could be the cause of this result?
2. Is the 79.38 accuracy for GPT-4V to evaluate on content creation task higher enough to replace humans? Could you provide the accuracy of human evaluation?
3. Authors are encouraged to provide results on some of the latest models, such as the InternVL2 series (2024/07/04), and the Qwen2-VL series (2024/08/30). While the results of these models are not mandatory under the guidelines, considering the super-fast advancements in VLMs this year. Could you please include results from some of the aforementioned models to highlight the performance of the latest generation of VLMs?
4. Refer to Weaknesses 3.
5. There is no content in Section 3.3 Data Construction.
6. Figure 5 is in Reference. Authors are encouraged to reformat it.
soundness: 3
strengths: 1. MCTBench fills an important gap by introducing content creation task to evaluate cognitive capabilities of VLMs in text-rich scenes.
2. Authors provide a large, diverse and high-quality human annotated dataset containing perception, reasoning, and content creation tasks.
3. The testing results on 18 VLMs inform readers that there is still room for improvement in reasoning tasks for current VLMs and in content creation tasks, text-enhanced VLMs trained for different types of tasks may lose some creative capabilities.
summary: The paper introduces MCTBench, a novel benchmark, with 8.5k QA pairs, aiming at evaluating the cognitive abilities of VLMs in text-rich visual scenes through visual reasoning and content creation tasks. Using GPT-4V to assist annotators in improving data quality and evaluate content creation efficiently. Several experiments on 18 VLMs are provided and pointing out that text-enhanced VLMs trained for different types of tasks may lose some creative capabilities of content creation.
weaknesses: 1. More newer VLMs, such as Gemini 1.5 Pro (Feb. 2024), InternVL1.5-Chat (Apr. 2024), GPT-4o (May 2024) and Claude 3.5 Sonnet (Jun. 2024) should be considered. 
2. The reliability of automated evaluation using GPT-4V is questioned.
3. The paper lacks further insightful analyses, such as the impact of the resolution of source images on the results, the impact of different language decoders on the results of content creation task.

Meta-review: This paper presents MCTBench, a new multimodal benchmark designed to evaluate the cognitive abilities of MLLM through visual reasoning and content generation tasks.

The strengths of this paper include the introduction of an automatic evaluation pipeline to improve the efficiency of the content generation task, and the distinction between perceptual and cognitive tasks to identify specific areas where MLLM needs improvement.

However, the paper is not well formatted, with section 3.3 incomplete and missing text, and figures inserted in the middle of the references section. In addition, the concerns of all reviewers were not addressed due to the lack of a rebuttal by the authors.

Thus, all reviewers gave negative reviews. There is no reason to overturn the decisions of the reviewers.

Decision: Reject