MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding

## Abstract

Abstract Large Vision-Language Models (LVLMs) have achieved remarkable performance in many vision-language tasks, yet their capabilities in fine-grained visual understanding remain insufficiently evaluated.
Existing benchmarks either contain limited fine-grained evaluation samples that are mixed with other data, or are confined to object-level assessments in natural images.
To holistically assess LVLMs’ fine-grained visual understanding capabilities, we propose using document images with multi-granularity and multi-modal information to supplement natural images.
In this light, we construct MMDocBench, a benchmark with various OCR-free document understanding tasks for the evaluation of fine-grained visual perception and reasoning abilities.
MMDocBench defines 15 15 15 15 main tasks with 4,338 QA pairs and 11,353 supporting regions, covering various document images such as research papers, receipts, financial reports, Wikipedia tables, charts, and infographics.
Based on MMDocBench, we conduct extensive experiments using 13 open-source and 3 proprietary advanced LVLMs, assessing their strengths and weaknesses across different tasks and document image types.
The benchmark, task instructions, evaluation code, and leaderboard can be found on the official website: https://MMDocBench.github.io/ .

## 1 Introduction

Large Vision-Language Models (LVLMs) have attained remarkable performance across various vision-language tasks (Yin et al., ).
However, existing LVLMs, such as GPT-4V (OpenAI, ), LLaVA (Liu et al., ) and MiniGPT-4 (Zhu et al., ), still struggle with understanding fine-grained visual details in images.
For instance, (Tong et al., ) have demonstrated that many LVLMs perform poorly in visual grounding to image details for visual question answering.
This fine-grained visual understanding capability is indispensable for LVLMs in many downstream tasks (Peng et al., ; Xuan et al., ), such as object recognition (Lin et al., ), image segmentation (Minaee et al., ) and forgery detection (Qu et al., ).
As such, it is essential to develop LVLMs’ capability in fine-grained visual understanding.

To achieve this goal, a key prerequisite is establishing benchmarks that can comprehensively evaluate the strengths and weaknesses of LVLMs in fine-grained visual understanding.
However, representative multimodal benchmarks such as MMVet (Yu et al., ), MME (Fu et al., ), and MMT-Bench (Ying et al., ) contain relatively few data samples to examine LVLMs’ understanding of fine-grained details rather than the entire image. Besides, these samples are not isolated from the overall dataset. This makes it difficult to evaluate LVLMs’ ability in fine-grained visual understanding comprehensively.
Moreover, some benchmarks, such as Visual7W (Zhu et al., ), RefCOCO (Yu et al., ), GVT-bench (Wang et al., ), and MMBench (Liu et al., ), have designed specific tasks like object grounding and object counting to evaluate fine-grained visual understanding. However, these tasks are confined to *object-level* details in natural images and do not assess finer-grained details.

To address the limitations, we consider using document images to supplement natural images for the evaluation of fine-grained visual understanding.
As illustrated in Figure 1, document images encapsulate various document content types, including text, figures, tables, charts, and diagrams, all presented within a visual format.
These document images like receipts and research papers are widely used across various domains such as finance, legal, education, and academia (Kim et al., ).
Compared to natural images, document images offer certain advantages as testing data to evaluate LVLMs’ fine-grained visual understanding capabilities.
In particular, 1) document images contain different granularities of information to evaluate the fine-grained visual perception abilities, such as the localization and recognition of text and tables.
The diverse elements (*e.g., *text, table, and chart) tend to occupy only a small part of the entire image, yet convey critical information in various granularities; for example, a receipt image may comprise item description (sentence level), quantity (token level), and barcode (object level).
Moreover, 2) document images require LVLMs to integrate multi-granularity and multi-type information to perform complex reasoning, thereby evaluating their fine-grained visual reasoning abilities. For example, in the bar chart located in the bottom left of Figure 1, LVLMs need to combine the legend, axis labels, and numerical values on the bar chart for comprehensive reasoning.

In light of this, we construct a benchmark, MMDocBench, using document images to assess LVLMs’ capabilities in fine-grained visual document understanding.
We design various OCR-free document understanding tasks from the perspectives of *fine-grained visual perception* and *fine-grained visual reasoning*, where understanding partial and fine-grained details in images is crucial, rather than treating the image as a whole.
For *fine-grained visual perception*, MMDocBench encompasses nine tasks to evaluate the LVLMs’ capabilities in fine-grained information recognition, localization, detection, and extraction, including Text Recognition, Table Recognition, Text Localization, Table Cell Localization, Key Information Extraction, Document Forgery Detection, Document Question Answering (QA), Chart QA, and Infographic QA.
For *fine-grained visual reasoning*, we design six tasks to assess the reasoning ability by integrating fine-grained information: Arithmetic Reasoning, Logical Reasoning, Spatial Reasoning, Counting, Comparison, and Sorting.
We present one example for each task in Figure 1 and refer to Section A.4 in Appendix for more comprehensive examples.

Based on these tasks, we select document images from $21$ document understanding datasets to construct QA pairs for evaluation.
The document images span a wide variety of types, including research papers, book covers, financial reports, scene-text images, receipts, Wikipedia tables, charts, infographics, and other industry documents.
Additionally, in MMDocBench, we also provide annotations of supporting regions (*i.e., *bounding boxes) within the image for each QA pair, as shown by red dotted rectangles in Figure 1. The supporting regions enable the evaluation of whether LVLMs have correctly grounded their predictions on the associated regions in the image, leading to a more comprehensive evaluation.
Furthermore, the output of supporting regions offers significant practical value, making the LVLMs’ responses more informative and interpretable while allowing for rapid cross-checking between the answer and the image.
Finally, MMDocBench contains $2,400$ document images, involving $4,338$ QA pairs with $11,353$ supporting regions.

With MMDocBench, we evaluate $13$ open-source and $3$ proprietary LVLMs that have demonstrated impressive performance in many vision-language tasks.
Experimental results reveal that our MMDocBench poses significant challenges to current LVLMs, especially in region prediction, where almost all LVLMs struggle to identify the supporting regions.
Although a notable gap persists in answer prediction between open-source and closed-source LVLMs, their performance in region prediction is comparable.
The in-depth analyses on LVLMs’ performance across different tasks and document types provide more insights:
1) Localization and detection tasks present significantly greater challenges compared to other tasks;
2) Region prediction on document images containing diverse elements (e.g., table, chart, and figure) is typically more challenging than on general documents, whereas answer prediction shows no significant variation across different document types.

In summary, our contributions are summarized as follows.

- •
We highlight the gap in existing benchmarks for evaluating LVLMs’ fine-grained visual understanding capabilities. Besides, we propose leveraging document images with multi-granularity and multi-type information to complement natural images in assessing the fine-grained visual perception and reasoning abilities of LVLMs.
- •
We construct MMDocBench, a comprehensive benchmark for tracking LVLMs’ progress in fine-grained visual document understanding. MMDocBench defines $15$ main tasks and $48$ sub-tasks over a wide range of document types, involving $2,400$ document images, $4,338$ QA pairs, and $11,353$ supporting regions for a holistic evaluation.
- •
We conduct extensive experiments with $16$ representative LVLMs on MMDocBench. We report the major strengths and weaknesses of LVLMs in different tasks and document types and provide valuable insights, facilitating the advancements of LVLMs in future.

## 2 Related work

### 2.1 Large vision-language models

Large Vision Language Models (LVLMs) are built upon Large Language Models (LLMs) like GPTs (Radford et al., ; Brown et al., ) and LLaMA (Touvron et al., ).
A common approach is to utilize a visual encoder to encode the image first and then apply a visual adapter to align visual and textual representations within the LLMs.
LVLMs are typically trained in two stages, i.e. self-supervised pre-training over large-scale image-text pairs and supervised instruction tuning with annotated data.
Notable examples include Flamingo (Alayrac et al., ), BLIP-2 (Li et al., ), LLaVA (Liu et al., ), MiniGPT-4 (Zhu et al., ), mPLUG-Owl (Ye et al., ), CogVLM (Wang et al., ), MiniCPM-V (Yao et al., ), Monkey (Li et al., ), and InternVL (Chen et al., ).
For instance, BLIP-2 (Li et al., ) utilizes frozen CLIP (Radford et al., ) as its visual encoder and proposes a Q-Former as the visual adapter.
LLaVA (Li et al., ) and MiniGPT-4 (Zhu et al., ) replace the visual adapter with simple linear layers, demonstrating impressive effectiveness.

Recently, increasing efforts have been devoted to enhancing LVLMs’ performance in document image understanding (Ye et al., ; Liu et al., ; Ye et al., ; Liu et al., ; Bai et al., ; Zhang et al., ), generally by accommodating high-resolution input images or/and improving quality of training corpus.
For instance, LLaVA-NeXT (Liu et al., ) extends LLaVa (Liu et al., ) to enhance its OCR capability by increasing image resolution and mixing more document images in visual instruction tuning data;
TextMonkey (Liu et al., ) and MiniCPM-V (Yao et al., ) divide high-resolution images into window patches with a sliding window method and reduce the length of visual tokens via token compression techniques.
Compared with natural images, the understanding of document images requires LVLMs’ interpretation of fine-grained image details. In this work we propose a comprehensive benchmark to evaluate such fine-grained visual understanding capability of LVLMs to facilitate future research.

### 2.2 Evaluation benchmarks for LVLMs

To date, lots of multimodal benchmarks have been constructed to holistically assess LVLMs’ integrated capabilities, such as recognition, knowledge, math, reasoning and safety (Huang & Zhang, ).
Some representative examples include LVLM-eHub (Xu et al., ), MME (Fu et al., ), MMStar (Chen et al., ), SEED-Bench (Li et al., ), MMMU (Yue et al., ), MMMU-Pro (Yue et al., ), MMVet (Yu et al., ) and MMT-Bench (Ying et al., ), etc.
In addition, there are also some benchmarks designed to evaluate one specific aspect of LVLMs.
For instance, POPE (Li et al., ) and HallusionBench (Guan et al., ) focus on evaluating hallucination; MathVista (Lu et al., ) is centered on assessing the visual mathematical reasoning capability; OCRBench (Liu et al., ) assesses the OCR capability on document images with five text-related visual tasks.
These existing benchmarks include some samples that can be used to examine LVLMs’ fine-grained visual understanding capability, but they are not only limited in number, but also difficult to be separated from other samples.
Moreover, their evaluations rely solely on natural language output without supporting region prediction involved, which is a crucial measure for achieving fine-grained visual understanding in LVLMs (Peng et al., ).
Current available benchmarks specially for evaluating fine-graded visual understanding of LVLMs mainly include Visual7W (Zhu et al., ), RefCOCO (Yu et al., ), GVT-bench (Wang et al., ), and MMBench (Liu et al., ), which are however centered on *object-level* recognition and interpretation within natural images, offering limited information granularity.
Our MMDocBench is the first comprehensive benchmark aiming to evaluate LVLMs’ fine-grained visual understanding capability with various OCR-free document understanding tasks.

### 2.3 Visual grounding in LVLMs

Visual Grounding (VG) is a technique that localizes the
relevant object or region to a given natural language description in the visual input (Deng et al., ).
It has been applied in LVLMs to facilitate generating more informative and comprehensive responses, benefiting various downstream applications like Object Recognition (Lin et al., ), Image Segmentation (Minaee et al., ) and Referential Comprehension (Yu et al., ).
Existing grounded LVLMs can be classified into two categories: 1) region-level (i.e., bounding box) grounding based LVLMs, such as OFA (Wang et al., ), Kosmos-2 (Peng et al., ) and Pink (Xuan et al., ); and 2) pixel-level grounding based LVLMs, such as PixelLM (Ren et al., ), GlaMM (Rasheed et al., ), Lisa (Lai et al., ) and Ferret (You et al., ).
However, the training and evaluation of these LVLMs sorely rely on *object-level* tasks involving natural images, overlooking the various granularities in document images.
In this work, we build MMDocBench to foster the advancement of the fine-grained visual understanding capability in LVLMs with a variety of OCR-free document understanding tasks.
In MMDocBench, it requires LVLMs to perform region-level grounding following typical settings in most document understanding tasks, such as Optical Character Recognition (OCR) (Singh et al., ), Key Information Extraction (Huang et al., ) and Table Recognition (Zheng et al., ).

Figure: Figure 2: An illustration of benchmark construction pipeline for MMDocBench.
Refer to caption: extracted/5919359/figs/bench_collect.jpeg

## 3 Proposed MMDocBench

### 3.1 Problem definition

On MMDocBench, the LVLMs are expected to output the precise answer to a natural language query given a document image while highlighting the supporting regions within the image contributing to inferring the answer.
Formally, given a document image $d$ possibly containing text, table, chart, and/or figure, for a question $q$, a Large Vision-Language Model $\mathcal{M}$ is required to produce a response including the answer $a$ and the corresponding regions $R$ as supporting evidence, formulated as

$$ $\displaystyle\mathcal{M}(d,q)$ $\displaystyle=(a,R).$ (1) $$

Each region in ${\mathbb{R}}$ is a bounding box that is represented by the coordinates of its top-left and bottom-right corners in the format of [$x_{1},y_{1},x_{2},y_{2}$].

### 3.2 Construction pipeline

The pipeline for constructing our MMDocBench is illustrated in Figure 2.

Step 1: Taxonomy Design. Targeting at a comprehensive evaluation of LVLMs’ fine-grained visual understanding capabilities, we design the taxonomy of MMDocBench following two principles.

- •
Fine-grained Discrimination: The MMDocBench must provide tasks that can adeptly evaluate LVLMs’ visual comprehension capabilities with sufficient discriminability of fine-grained details in the image, rather than treating the image as a whole.
- •
Diversity: The MMDocBench
must encompass a broad range of tasks in terms of required capabilities (e.g., perception, reasoning), content granularity (e.g., characters, words, tables), and document types (e.g., scientific papers, financial reports, receipts).

To solve the problem in MMDocBench, two capabilities are at the core for LVLMs, i.e. fine-grained visual perception and fine-grained visual reasoning (Liu et al., ).
To investigate both capabilities of LVLMs, we design a total of $15$ tasks in MMDocBench.
Specifically, we encompass nine tasks for fine-grained visual perception, including Text Recognition, Table Recognition, Text Localization, Table Cell Localization, Key Information Extraction, Document Forgery Detection, Document Question Answering, Chart Question Answering, and Infographic Question Answering; and for fine-grained visual reasoning, we include six tasks: Arithmetic Reasoning, Logical Reasoning, Spatial Reasoning, Comparison, Sorting and Counting.
Further, we include one or multiple sub-tasks for each task to cover more diverse document image types, e.g. research papers, book covers, financial reports, scene-text images, receipts, Wikipedia tables, charts, infographics, and other industry documents, leading to a total of 48 sub-tasks in MMDocBench.
See a summary in Table 1.

**Table 1: Taxonomy and statistics of MMDocBench.**
| Main Task | Sub Task | Document Image Type | # Images | # QA Pairs | # Regions |
| --- | --- | --- | --- | --- | --- |
| Fine-Grained Visual Perception |  |  |  |  |  |
| Text<br>Recognition | TextOCR (Singh et al., ) | Scene-Text Images | 100 | 100 | 100 |
| BookOCR (Mishra et al., ) | Book Covers | 100 | 100 | 438 |  |
| Table<br>Recognition | FinTabNet (Zheng et al., ) | Financial Reports | 100 | 100 | 1,864 |
| PubTables-1M (Smock et al., ) | Scientific Papers | 100 | 100 | 3,520 |  |
| Text<br>Localization | Text2Bbox (Šimsa et al. () etc.) | Industry Documents | 100 | 100 | 100 |
| Bbox2Text (Šimsa et al. () etc.) | Industry Documents | 100 | 100 | 100 |  |
| Table Cell<br>Localization | FinTabNet (Zheng et al., ) | Financial Reports | 100 | 100 | 100 |
| PubTables-1M (Smock et al., ) | Scientific Papers | 100 | 100 | 100 |  |
| Key<br>Information<br>Extraction | SROIE (Huang et al., ) | Receipts | 100 | 303 | 303 |
| WildReceipt (Sun et al., ) | Receipts | 100 | 512 | 512 |  |
| CORD (Park et al., ) | Receipts | 100 | 372 | 372 |  |
| Doc Forgery<br>Detection | T-SROIE (Yuxin et al., ) | Receipts | 100 | 100 | 286 |
| DocTamper (Qu et al., ) | Cross-Domain Documents | 100 | 100 | 129 |  |
| Document<br>QA | DocVQA (Mathew et al., ) | Industry Documents | 100 | 262 | 262 |
| WTQ (Pasupat & Liang, ) | Wikipedia Tables | 100 | 351 | 351 |  |
| TAT-DQA (Zhu et al., ) | Financial Reports | 100 | 214 | 214 |  |
| Chart<br>QA | ChartQA (Masry et al., ) | Cross-Domain Charts | 100 | 104 | 104 |
| CharXiv (Wang et al., ) | Scientific Charts | 100 | 149 | 149 |  |
| Infographic<br>QA | InfographicVQA (Mathew et al., ) | Infographics | 100 | 281 | 281 |
| Fine-Grained Visual Reasoning |  |  |  |  |  |
| Arithmetic<br>Reasoning | DUDE (Van Landeghem et al., ) | General Documents | 13 | 15 | 34 |
| WTQ (Pasupat & Liang, ) | Wikipedia Tables | 54 | 55 | 159 |  |
| TAT-DQA (Zhu et al., ) | Financial Table-Text Documents | 98 | 217 | 453 |  |
| CharXiv (Wang et al., ) | Scientific Charts | 23 | 23 | 67 |  |
| InfographicVQA (Mathew et al., ) | Infographics | 34 | 53 | 90 |  |
| Logical<br>Reasoning | DUDE (Van Landeghem et al., ) | General Documents | 10 | 11 | 20 |
| WTQ (Pasupat & Liang, ) | Wikipedia Tables | 11 | 11 | 41 |  |
| TAT-DQA (Zhu et al., ) | Financial Table-Text Documents | 1 | 1 | 2 |  |
| CharXiv (Wang et al., ) | Scientific Charts | 7 | 7 | 12 |  |
| InfographicVQA (Mathew et al., ) | Infographics | 2 | 2 | 3 |  |
| Spatial<br>Reasoning | DUDE (Van Landeghem et al., ) | General Documents | 38 | 41 | 43 |
| WTQ (Pasupat & Liang, ) | Wikipedia Tables | 4 | 4 | 8 |  |
| CharXiv (Wang et al., ) | Scientific Charts | 7 | 7 | 12 |  |
| InfographicVQA (Mathew et al., ) | Infographics | 17 | 23 | 54 |  |
| Comparison | DUDE (Van Landeghem et al., ) | Cross-Domain Documents | 3 | 3 | 6 |
| WTQ (Pasupat & Liang, ) | Wikipedia Tables | 33 | 34 | 74 |  |
| TAT-DQA (Zhu et al., ) | Financial Table-Text Documents | 10 | 10 | 30 |  |
| CharXiv (Wang et al., ) | Scientific Charts | 16 | 16 | 44 |  |
| InfographicVQA (Mathew et al., ) | Infographics | 13 | 15 | 44 |  |
| Sorting | DUDE (Van Landeghem et al., ) | General Documents | 3 | 3 | 6 |
| WTQ (Pasupat & Liang, ) | Wikipedia Tables | 6 | 12 | 23 |  |
| TAT-DQA (Zhu et al., ) | Financial Table-Text Documents | 7 | 7 | 14 |  |
| CharXiv (Wang et al., ) | Scientific Charts | 15 | 15 | 29 |  |
| InfographicVQA (Mathew et al., ) | Infographics | 20 | 29 | 57 |  |
| Counting | DUDE (Van Landeghem et al., ) | General Documents | 51 | 55 | 244 |
| WTQ (Pasupat & Liang, ) | Wikipedia Tables | 15 | 15 | 76 |  |
| TAT-DQA (Zhu et al., ) | Financial Table-Text Documents | 14 | 14 | 26 |  |
| CharXiv (Wang et al., ) | Scientific Charts | 38 | 40 | 149 |  |
| InfographicVQA (Mathew et al., ) | Infographics | 44 | 52 | 248 |  |

Step 2: Document Image & QA Pair Preparation. As shown in Table 1, we create BookOCR, Bbox2Text, and Text2Bbox by ourselves and use original task settings for other sub-tasks.
In particular, we build BookOCR, a text recognition dataset, based on selected document images (i.e., book covers) from OCR-VQA (Mishra et al., ).
After collecting document images, we use a pre-defined template for text recognition as the question and automatically identify all the OCR content from the image as the ground-truth answer.
We build Text2Bbox and Bbox2Text with the same document images, which are selected from DocILE (Šimsa et al., ), RVL-CDIP (Harley et al., ), DocBank (Li et al., ), PubLayNet (Zhong et al., ) and PubTabNet (Zhong et al., ) to cover a great diversity of document types.
The former task requires an LVLM to find the region in the document image given a piece of textual content, while the latter needs the model to identify corresponding text in the document image based on a specified bounding box.
For the three sub-tasks, our annotators (6 undergraduate or graduate students majored in computer science) manually create one QA pair for each selected document image.

For other sub-tasks, our annotators manually analyze and select appropriate document images with annotated input-output pairs from the source dataset.
After that, the input-output pairs are transformed into QA pairs following the pre-defined templates.
Note that all the document images and QA pairs are selected from the test set of the source dataset except CORD (Park et al., ), DUDE (Van Landeghem et al., ) and CharXiv (Wang et al., ).
CORD has an insufficient number of high-quality document images in its test set, so we select some from its validation set.
DUDE and CharXiv have not yet released their test sets, and thus we utilize their validation sets instead.
For each sub-task, we at most select $100$ document images.

It is worth mentioning that for preparing sub-tasks of fine-grained visual reasoning, we purposely select five existing datasets to ensure our MMDocBench includes a great diversity of document image types.
These datasets include DUDE (Van Landeghem et al., ) containing general documents from various industries,
WTQ (Pasupat & Liang, ) containing table-based documents from Wikipedia,
TAT-DQA (Zhu et al., ) containing table-text documents from financial reports,
ChartXiv (Wang et al., ) containing chart-based documents from scientific papers,
and InfographicVQA (Mathew et al., ) containing infographic-based documents.

Step 3: Region Generation.
We generate ground truth regions for each QA pair in MMDocBench to facilitate evaluation.
Similar to (Xuan et al., ; Chen et al., ), we normalize the coordinates used to represent the bounding box to the range [0, 1000] w.r.t. the image dimensions.

For the tasks regarding fine-grained visual perception, we set the answer’s location on the image as the region to be annotated, while for those regarding fine-grained visual reasoning, we annotate the locations of all supporting evidences used to infer the final answer.
Specifically, we first obtain the OCR results using Google OCR service for each document image in MMDocBench.
For fine-grained visual perception tasks, we automatically identify the corresponding value and its bounding box in the OCR result based on the answer.
If only one value matches, we use the region of this value as the correct one; otherwise, our annotators manually review and select the appropriate region for the answer.
For fine-grained visual reasoning tasks, we search for the regions for each supporting evidence if the source dataset already provides the annotation of supporting evidence, like TAT-DQA (Zhu et al., ); for the rest, our annotators manually check supporting evidence and annotate the appropriate region for each one.

Step 4: QA & Region Verification.
To ensure high quality of MMDocBench, we further verify the collected data.
First, we develop a program to automatically highlight the answer or supporting evidence with its corresponding regions on the document image.
Then, different annotators review the rendered document image in three rounds to ensure that the answer and supporting regions to the question are correct.

### 3.3 Statistics and analysis

With the above construction pipeline, the resultant MMDocBench contains a total of $2,400$ document images and $4,338$ QA pairs with $11,353$ annotated supporting regions.
On average, each question has $10.1$ words and around $2.61$ supporting regions.
Refer to Table 4 in Appendix and Table 1 for more detailed statistics of our MMDocBench.

We analyze the position distribution and area distribution of the annotated supporting regions for each QA pair in our MMDocBench.
To compute the position distribution,
given a region [$x_{1},y_{1},x_{2},y_{2}$], we first calculate the center point by ($\frac{x_{1}+x_{2}}{2}$, $\frac{y_{1}+y_{2}}{2}$).
Then, we plot all the center points on an image with a dimension of $1000\times 1000$.
As shown in Figure 4, all points are scattered across the entire image, indicating no clear positional bias for supporting regions in MMDocBench.
We also analyze the area distribution of all regions to examine their granularity.
We first calculate the area of each region and then apply a logarithmic transformation with a base of 10 on the computed area value.
As shown in Figure 4, the granularity of regions shows a diversity and the majority of the region areas fall between $1,000$ and $10,000$, corresponding to regions sized between $10\times 100$ and $100\times 100$.
These analyses well highlight the high quality of our MMDocBench, which is crucial for accurately assessing the capabilities of LVLMs in fine-grained visual understanding.

Figure: Figure 3: Position distribution of all regions.
Refer to caption: extracted/5919359/figs/region_distribution.jpeg

## 4 Experiments

### 4.1 Experimental setup

Evaluated LVLMs.
We conduct evaluation experiments with 13 open-source LVLMs and 3 proprietary LVLMs on the proposed MMDocBench.
We select those open-source models with strong document understanding capabilities, including InternVL2-Llama3-76B (Chen et al., ),Llava-OV-Chat-72B (Li et al., ),mPLUG-DocOwl-1.5-Omni (Hu et al., ), LLaVA-V1.6-34B (Liu et al., ), YI-VL-34B AI et al. (), CogVLM2-Chat-19B Hong et al. (), Qwen2-VL-7B-Instruct (Wang et al., ), TextMonkey (Liu et al., ), MiniCPM-V2.6 (Yao et al., ), MiniCPM-Llama3-V2.5 (Yao et al., ), InternVL2-8B (Chen et al., ), mPLUG-Owl3 (Ye et al., ), and Ferret (You et al., ).
For proprietary LVLMs to be evaluated in our experiments, we use Qwen-VL-Max-0809 (Bai et al., ) and two versions of the latest OpenAI GPT (Achiam et al., ), i.e. GPT-4o-2024-08-06 and GPT-4-turbo-2024-04-09.
These models show a noticeable diversity in respective parameter sizes, visual encoders, and language models.

Instruction Design.
For each main task in MMDocBench, we manually design an instruction template to guide the LVLM to output the answer and supporting regions in JSON format, e.g., {“answer”:“{answer}”, “bbox”:[“{bbox1}”,“{bbox2}”]}.
For fine-grained visual perception tasks, we instruct the LVLM to output the results directly, while for fine-grained visual reasoning tasks, we enable chain-of-thought (CoT) in the instructions.
Since some LVLMs cannot follow the instructions well during the test, especially for the region predictions, we revise the instructions based on the setting of these LVLMs to improve their performance.
At inference, we apply zero-shot prompting on all the LVLMs to generate responses for each question.
After obtaining the response, we develop a program that uses strict regular expressions to extract the predicted answer and the supporting regions for evaluation.

Evaluation Metrics. For each question, we use Exact Match (EM) and $F1$-score to evaluate the predicted answer, and Intersection over Union (IOU) to assess the predicted region(s).
The EM is determined by matching every character of the model’s text prediction to the ground truth. If all characters are matched, the EM is 1, and otherwise 0.
For $F1$-score, we calculate the word-level F1 based on the number of words in model prediction, ground truth, and their intersection (Rajpurkar et al., ).
The Intersection over Union (IOU) is computed between the predicted region and the ground-truth region, taking into account their overlapping area and union area.
The scores on the three metrics for each sub-task are computed by taking the mean of corresponding metric scores for all the questions included in this sub-task.
To obtain metric scores per task or capability, and overall performance, we calculate the macro average across all corresponding lower-level metric scores.

**Table 2: The overall performance of different LVLMs on MMDocBench. Best results are marked in bold while the second-best are underlined in the category (open-source or close-source). Metric values below 1% are marked with ’-’.**
| Model | Size | Fine-Grained<br>Visual Perception | Fine-Grained<br>Visual Reasoning | Overall |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EM | F1 | IOU | EM | F1 | IOU | EM | F1 | IOU |  |  |
| Close-Source LVLMs |  |  |  |  |  |  |  |  |  |  |
| GPT-4o | - | 62.47 | 65.18 | 3.21 | 70.33 | 72.56 | 1.68 | 66.40 | 68.87 | 2.44 |
| GPT-4V | - | 52.37 | 56.95 | 2.48 | 50.97 | 52.22 | - | 51.67 | 54.58 | 1.54 |
| Qwen-VL-Max | - | 61.63 | 65.89 | 18.60 | 70.15 | 72.24 | 4.27 | 65.89 | 69.06 | 11.44 |
| Open-Source LVLMs |  |  |  |  |  |  |  |  |  |  |
| InternVL2-Llama3-76B | 76B | 54.63 | 58.84 | 2.30 | 61.76 | 64.46 | - | 58.20 | 61.65 | 1.54 |
| LLava-OV-Chat-72b | 72B | 52.59 | 57.49 | 2.28 | 65.26 | 67.55 | - | 58.93 | 62.52 | 1.57 |
| mPLUG-DocOwl1.5-Omni | 72B | 8.18 | 12.82 | 1.04 | 12.24 | 13.67 | - | 10.21 | 13.25 | - |
| LLaVA-V1.6-34B | 34B | 32.75 | 39.13 | 2.66 | 27.85 | 31.34 | - | 30.30 | 35.23 | 1.63 |
| YI-VL-34B | 34B | 23.22 | 27.95 | - | 19.06 | 20.54 | - | 21.14 | 24.24 | - |
| CogVLM2-Chat-19B | 19B | 6.99 | 10.81 | - | 7.87 | 10.39 | - | 7.43 | 10.60 | - |
| Qwen2-VL-7B-Instruct | 9B | 55.15 | 59.14 | 15.16 | 43.07 | 45.88 | 2.69 | 49.11 | 52.51 | 8.93 |
| TextMonkey | 9B | 31.41 | 35.88 | 19.22 | 13.73 | 14.74 | - | 22.57 | 25.31 | 9.87 |
| MiniCPM-V2.6 | 8B | 45.35 | 52.55 | 4.71 | 43.17 | 46.13 | - | 44.26 | 49.34 | 2.75 |
| MiniCPM-Llama3-V2.5 | 8B | 33.71 | 40.42 | 5.93 | 30.95 | 33.23 | 1.45 | 32.33 | 36.82 | 3.69 |
| InternVL2-8B | 8B | 45.72 | 51.42 | 2.01 | 45.16 | 48.57 | - | 45.44 | 50.00 | 1.29 |
| mPLUG-Owl3 | 7B | 15.63 | 19.97 | - | 10.84 | 12.65 | - | 13.24 | 16.31 | - |
| Ferret | 7B | 3.96 | 6.96 | 4.33 | - | - | - | 1.98 | 3.48 | 2.16 |

### 4.2 Main results

We conduct a comprehensive comparison of different LVLMs with our proposed MMDocBench, and show the results in Table 2.
We make below key findings.
1) The proposed MMDocBench poses significant challenges to current LVLMs in terms of both answer prediction and region prediction.
The best model GPT-4o achieves $66.40\%$ in EM for answer prediction, but only $2.44\%$ in IOU for region prediction.
Another close-source model, Qwen-VL-Max, which has comparable answer prediction performance to GPT-4o, is the best-performing model for region prediction, with an IOU score of $11.44\%$ only.
This highlights substantial challenges of region prediction in MMDocBench.
2) There is a large gap in answer prediction performance between open-source and close-source models, while that in their region prediction performance is minor.
For answer prediction, GPT-4o significantly outperforms the best open-source model Llava-OV-Chat-72b (with $58.93\%$ in EM) by over $12\%$ in EM.
For region prediction, open-source models like TextMonkey and Qwen2-VL-7B-Instruct, with IOU scores of $8.93\%$ and $9.87\%$ respectively, perform slightly worse than Qwen-VL-Max but outperform GPT-4o by around three times.
This could be explained by OpenAI using insufficient samples with visual grounding requirements when training GPT-4o and GPT-4v, leading to a lack of visual grounding capability.
Another possible reason is that visual encoders adopted in GPT-4o and GPT-4v do not effectively support fine-grained visual understanding, which we leave for future investigation.
3) LVLMs trained on document images with text-grounding requirements, such as Qwen-VL-Max, Qwen2-VL-7B-Instruct, and TextMonkey, show improvements in region prediction, while those trained with object-level grounding over natural images, like Ferret, exhibit no such improvements.
This highlights the necessity of establishing a benchmark that supports visual grounding at various and finer granularities on document images, such as our MMDocBench.

In the following, we further analyze model performance on answer prediction and region prediction regarding fine-grained visual perception and fine-grained visual reasoning, respectively.

Answer Prediction. We make below key findings.
1) GPT-4o consistently beats all other models on both fine-grained visual perception and fine-grained visual reasoning tasks in terms of EM, demonstrating its superior effectiveness.
2) Larger models, like GPT-4o, Qwen-VL-Max, InternVL2-Llama3-76B, and Llava-OV-Chat-72b tend to perform better on fine-grained visual reasoning tasks than fine-grained visual perception tasks,
while smaller models, conversely, excel in fine-grained visual perception tasks over reasoning tasks.
This might be because reasoning capabilities improve significantly as the model size increases.

Region Prediction. We make below key findings.
1) TextMonkey achieves the best results on fine-grained visual perception tasks with $19.22\%$ in IOU, but it fails to output supporting regions for fine-grained visual reasoning tasks.
One possible reason is that TextMonkey only involves perception-related samples and instructions for training its grounding ability, resulting in its inability to ground the supporting evidence for reasoning tasks.
2) All LVLMs face significant challenges in predicting supporting regions for fine-grained visual reasoning tasks.
The performance of most LVLMs on fine-grained visual reasoning tasks is notably worse than that on fine-grained visual perception tasks.
Qwen-VL-Max, which is the best model for region prediction on fine-grained visual reasoning tasks, earns only $4.27\%$ in IOU, indicating the remarkable challenges of this task.

Figure: Figure 5: Model performance comparison across all tasks with F1 score.
Refer to caption: extracted/5919359/figs/task_f1.jpeg

### 4.3 Analysis on different tasks

We compare the performance of various LVLMs across tasks and present the results in Figure 6 and Figure 6.
We make below key findings.
1) As shown in Figure 6, GPT-4o and Qwen-VL-Max are the best and second-best models across most tasks among all LVLMs, except for Logical Reasoning and Spatial Reasoning, where LLaVA-OV-Chat-72B outperforms all other models.
2) As shown in Figure 6, all LVLMs struggle with the prediction of supporting regions on almost all the tasks, except for Text Recognition, where the best model, Qwen-VL-Max, achieves around $71.2\%$ in IOU.
Qwen-VL-Max delivers the best results in region prediction across most tasks, except for Forgery Detection, Infographic QA, and Text Localization, where TextMonkey ranks the first.
3) Among all the tasks, Document Forgery Detection, a fine-grained visual perception task, is the most challenging for all LVLMs, with the best result being only around $20.6\%$ in EM for answer prediction and $1.8\%$ in IOU for region prediction;
As illustrated in Figure 1, this task requires the model to identify the inconsistent word(s) against other words within the image.
In addition, the answer prediction performance of all LVLMs on Text Localization and Table Cell Localization is poor, underscoring the challenges posed by both tasks.
Refer to Section A.2 in Appendix for a more detailed performance comparison of all LVLMs across different tasks.

**Table 3: The performance comparison of LVLMs across different document types. Best results are marked in bold while the second-best are underlined in the category (open-source or close-source). Metric values below 1% are marked with a ’-’.**
| Model | General | Table-Based | Table-Text | Chart-Based | Infographic-Based |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| F1 | IOU | F1 | IOU | F1 | IOU | F1 | IOU | F1 | IOU |  |
| Close-Source LVLMs |  |  |  |  |  |  |  |  |  |  |
| GPT-4o | 71.21 | 3.25 | 71.31 | 1.39 | 77.97 | 2.90 | 71.53 | 3.06 | 74.41 | 1.38 |
| GPT-4V | 58.65 | 2.43 | 56.84 | - | 62.76 | 2.17 | 54.89 | 1.64 | 64.27 | 1.14 |
| Qwen-VL-Max | 68.45 | 19.38 | 67.35 | 5.23 | 80.89 | 9.52 | 74.18 | 11.53 | 79.41 | 4.39 |
| Open-Source LVLMs |  |  |  |  |  |  |  |  |  |  |
| InternVL2-Llama3-76B | 65.72 | 2.57 | 60.28 | - | 68.37 | - | 68.39 | 1.38 | 68.67 | - |
| LLaVA-OV-Chat-72B | 61.83 | 2.65 | 65.34 | - | 69.28 | 1.17 | 68.39 | 1.10 | 69.53 | - |
| mPLUG-DocOwl1.5-Omni | 17.06 | - | 13.35 | - | 11.84 | - | 12.99 | - | 19.69 | 1.25 |
| LLaVA-V1.6-34B | 42.53 | 2.57 | 33.42 | - | 36.65 | - | 40.62 | - | 33.96 | - |
| Yi-VL-34B | 13.28 | 1.10 | 8.55 | - | 10.87 | - | 12.37 | - | 13.94 | - |
| CogVLM2-Chat-19B | 27.81 | - | 17.33 | - | 32.11 | - | 33.27 | - | 35.39 | - |
| Qwen2-VL-7B-Instruct | 55.64 | 16.57 | 50.14 | 2.67 | 64.12 | 5.17 | 55.92 | 7.57 | 59.74 | 2.81 |
| TextMonkey | 36.11 | 15.44 | 20.17 | 4.01 | 24.70 | 3.34 | 23.62 | 5.51 | 27.35 | 4.43 |
| MiniCPM-V2.6 | 48.33 | 5.03 | 51.73 | - | 54.18 | 3.45 | 52.01 | - | 54.64 | 1.78 |
| MiniCPM-Llama3-V2.5 | 40.90 | 6.67 | 34.97 | 1.59 | 41.64 | 3.26 | 42.61 | - | 39.49 | 3.83 |
| InternVL2-8B | 55.24 | 2.04 | 52.17 | - | 53.01 | - | 57.93 | - | 52.49 | - |
| mPLUG-Owl3 | 21.27 | - | 10.89 | - | 18.76 | - | 22.02 | - | 15.53 | - |
| Ferret | 8.61 | 7.17 | 3.08 | - | 4.70 | - | 5.81 | - | 5.63 | - |

### 4.4 Analysis on different document types

Based on the majority of content included, we categorize all the document images in MMDocBench into five types: General Document, Table-Based Document, Table-Text Document, Chart-Based Document, and Infographic-Based Document, and ensure document images per sub-task belong to the same one category.
Refer to Table 9 in Appendix for detailed strategies.

We present LVLMs’ results across different document types in Table 3, from which we make below findings.
1) For answer prediction, GPT-4o obtains the best performance on General Document and Table-Based Document, while Qwen-VL-Max ranks the first across the other three document types.
2) For region prediction, TextMonkey outperforms all other models on Infographic-Based Document, while Qwen-VL-Max leads on the remaining four document types.
3) In comparison, region prediction on other document types is significantly more challenging than on General Documents, while there is no notable difference in answer prediction across different document types.

## 5 Conclusion

In this work, we introduce the MMDocBench benchmark to comprehensively evaluate LVLMs’ fine-grained visual perception and reasoning capabilities via various OCR-free document understanding tasks.
In MMDocBench, we carefully design 15 tasks and 48 sub-tasks that require LVLMs to perform deep, fine-grained interpretation of image details to answer each question.
To enable a more comprehensive evaluation, we provide annotations of the supporting regions for each question-answer pair to assess whether LVLMs have the abilities to correctly ground their predictions on the associated regions in the image.
With MMDocBench, we evaluate various open-source and proprietary LVLMs, analyzing their performance in fine-grained visual document image understanding.
We observe that our MMDocBench presents significant challenges to current LVLMs in both answer and region prediction, with GPT-4o achieving the highest answer prediction score of $66.40\%$ in EM and Qwen-VL-Max achieving the best region prediction score of $11.44\%$ in IOU.
Moreover, we find that open-source LVLMs demonstrate competitive performance in region prediction compared to proprietary models, despite a significant gap in answer prediction.
We believe MMDocBench can enable a thorough and multi-faceted evaluation of fine-grained visual document understanding of LVLMs, thereby facilitating LVLMs’ future advancement.

## Appendix A Appendix

### A.1 Key Statistics

We present some key statistic about MMDocBench  in Table 4.

**Table 4: Key statistics of MMDocBench.**
| Statistic | Number |
| --- | --- |
| Total Number of Tasks | 15 |
| - # Visual Perception Tasks | 9 |
| - # Visual Reasoning Tasks | 6 |
| Total Number of Sub Tasks | 48 |
| - # Visual Perception Sub-Tasks | 19 |
| - # Visual Reasoning Sub-Tasks | 29 |
| Number of Existing Datasets Involved | 21 |
| Total Number of Images | 2,400 |
| Total Number of QA Pairs | 4,338 |
| Total Number of Regions | 11,353 |
| Avg. Number of Words per Question | 16.14 |
| Avg. Number of Words per Answer | 4.08 |
| Avg. Number of Regions per Question | 2.61 |

### A.2 Detailed performance of LVLMs across different tasks

**Table 5: Model performance comparison across different fine-grained visual perception tasks using F1 score. TXR:Text Recognition, TBR:Table Recognition, TXL:Text Localization, TCL:Table Cell Localization, KIE:Key Information Extraction, DFD:Document Forgery Detection, DQA:Document Question Answering, CQA:Chart Question Answering, IQA:Infographic Question Answering.**
| Model | Fine-Grained Visual Perception |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TXR | TBR | TXL | TCL | KIE | DFD | DQA | CQA | IQA |  |
| Close-Source LVLMs |  |  |  |  |  |  |  |  |  |
| GPT-4o | 95.66 | 73.53 | 10.81 | 42.14 | 89.23 | 20.44 | 88.76 | 85.18 | 80.90 |
| GPT-4V | 88.28 | 73.33 | 14.12 | 30.04 | 82.45 | 1.99 | 79.08 | 72.31 | 70.93 |
| Qwen-VL-Max | 91.85 | 70.05 | 29.09 | 39.59 | 90.66 | 6.74 | 90.23 | 88.77 | 86.00 |
| Open-Source LVLMs |  |  |  |  |  |  |  |  |  |
| InternVL2-Llama3-76B | 90.82 | 75.18 | 12.90 | 33.25 | 85.97 | 9.24 | 74.06 | 80.53 | 67.64 |
| LLaVA-OV-Chat-72B | 90.58 | 68.12 | 5.74 | 26.55 | 79.67 | 4.49 | 81.09 | 84.16 | 76.99 |
| mPLUG-DocOwl1.5-Omni | 33.90 | - | 0.80 | 2.19 | 7.37 | 2.25 | 15.03 | 16.73 | 24.32 |
| LLaVA-V1.6-34B | 77.68 |  | 4.43 | 11.97 | 67.14 | 2.92 | 54.88 | 55.27 | 38.72 |
| Yi-VL-34B | 39.44 | 5.77 | - | - | 8.80 | 1.07 | 11.84 | 12.56 | 16.91 |
| CogVLM2-Chat-19B | 50.15 | 7.75 | 0.97 | 1.70 | 50.11 | - | 45.37 | 49.27 | 46.02 |
| Qwen2-VL-7B-Instruct | 89.04 | 76.11 | 24.57 | 28.00 | 81.63 | - | 79.23 | 78.21 | 74.76 |
| TextMonkey | 88.13 | - | 26.10 | 8.70 | 59.98 | 5.50 | 53.12 | 42.66 | 38.71 |
| MiniCPM-V2.6 | 82.69 | 59.87 | 17.10 | 35.16 | 69.11 | 6.72 | 69.60 | 73.56 | 59.11 |
| MiniCPM-Llama3-V2.5 | 78.40 | 47.40 | 11.91 | 18.27 | 55.52 | 1.01 | 51.40 | 56.01 | 43.83 |
| InternVL2-8B | 79.09 | 74.93 | 6.51 | 21.70 | 78.53 | 5.76 | 67.21 | 72.26 | 56.81 |
| mPLUG-Owl3 | 45.73 | 5.31 | 0.12 | 7.67 | 38.20 | 1.59 | 28.07 | 36.01 | 17.04 |
| Ferret | 29.19 | - | 5.32 | - | - | - | 7.56 | 5.81 | 5.63 |

**Table 6: Model performance comparison across different fine-grained visual perception tasks using IOU. TXR:Text Recognition, TBR:Table Recognition, TXL:Text Localization, TCL:Table Cell Localization, KIE:Key Information Extraction, DFD:Document Forgery Detection, DQA:Document Question Answering, CQA:Chart Question Answering, IQA:Infographic Question Answering.**
| Model | Fine-Grained Visual Perception |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TXR | TBR | TXL | TCL | KIE | DFD | DQA | CQA | IQA |  |
| Close-Source LVLMs |  |  |  |  |  |  |  |  |  |
| GPT-4o | 14.64 | - | 2.09 | 2.47 | 2.04 | - | 2.06 | 3.20 | 1.18 |
| GPT-4V | 8.66 | - | 1.55 | 1.30 | 4.42 | - | 1.66 | 2.40 | 1.57 |
| Qwen-VL-Max | 70.80 | 5.59 | 10.91 | 6.40 | 33.22 | 1.12 | 16.79 | 16.96 | 5.66 |
| Open-Source LVLMs |  |  |  |  |  |  |  |  |  |
| InternVL2-Llama3-76B | 14.48 | - | 1.20 | - | 1.45 | - | - | - | - |
| LLaVA-OV-Chat-72B | 13.48 | - | 1.18 | - | 2.60 | - | 1.03 | - | - |
| mPLUG-DocOwl1.5-Omni | 3.49 | - | 1.01 | - | - | - | - | - | 2.27 |
| LLaVA-V1.6-34B | 11.01 |  | - | 1.49 | 5.45 | - | - | - | 1.13 |
| Yi-VL-34B | 4.89 | - | - | - | 1.36 | - | - | - | - |
| CogVLM2-Chat-19B | 2.79 | - | - | - | - | - | - | - | - |
| Qwen2-VL-7B-Instruct | 65.55 | 2.23 | 7.08 | 3.12 | 30.63 | - | 12.27 | 11.28 | 4.28 |
| TextMonkey | 67.54 | - | 43.64 | 2.39 | 24.30 | 1.80 | 14.72 | 10.48 | 8.15 |
| MiniCPM-V2.6 | 17.69 | 1.00 | 7.40 | - | 6.89 | - | 5.69 | - | 2.67 |
| MiniCPM-Llama3-V2.5 | 12.94 | 1.09 | 9.18 | - | 13.03 | - | 9.26 | - | 6.74 |
| InternVL2-8B | 11.86 | - | 1.03 | 1.00 | 2.34 | - | - | - | - |
| mPLUG-Owl3 | 2.03 | - | - | - | - | - | - | - | - |
| Ferret | 29.12 | - | 1.47 | - | - | - | - | - | - |

**Table 7: Model performance comparison across different fine-grained visual reasoning tasks using F1.**
| Model | Fine-Grained Visual Reasoning |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Arithmetic Reasoning | Logical Reasoning | Spatial Reasoning | Comparison | Sorting | Counting |  |
| Close-Source LVLMs |  |  |  |  |  |  |
| GPT-4o | 76.52 | 73.55 | 67.50 | 74.64 | 73.86 | 69.28 |
| GPT-4V | 53.36 | 49.25 | 54.08 | 49.18 | 66.57 | 40.89 |
| Qwen-VL-Max | 82.37 | 70.27 | 66.80 | 69.84 | 76.70 | 67.45 |
| Open-Source LVLMs |  |  |  |  |  |  |
| InternVL2-Llama3-76B | 65.77 | 70.28 | 65.44 | 62.27 | 64.83 | 58.17 |
| LLaVA-OV-Chat-72B | 65.91 | 75.09 | 77.18 | 71.51 | 62.88 | 52.74 |
| mPLUG-DocOwl1.5-Omni | 4.92 | 11.17 | 22.14 | 18.76 | 17.30 | 7.72 |
| LLaVA-V1.6-34B | 28.37 | 44.61 | 43.46 | 37.12 | 18.10 | 16.36 |
| Yi-VL-34B | 1.69 | 15.59 | 17.48 | 14.14 | 8.97 | 4.47 |
| CogVLM2-Chat-19B | 12.48 | 24.76 | 24.63 | 29.24 | 20.43 | 11.68 |
| Qwen2-VL-7B-Instruct | 50.42 | 43.61 | 53.42 | 48.93 | 38.26 | 40.62 |
| TextMonkey | 2.11 | 10.91 | 20.38 | 25.68 | 14.37 | 14.99 |
| MiniCPM-V2.6 | 40.64 | 57.57 | 60.04 | 44.38 | 42.70 | 31.47 |
| MiniCPM-Llama3-V2.5 | 26.73 | 32.37 | 44.40 | 44.54 | 39.57 | 11.78 |
| InternVL2-8B | 48.19 | 56.64 | 60.08 | 49.73 | 41.30 | 35.50 |
| mPLUG-Owl3 | 2.81 | 25.34 | 16.79 | 22.23 | 5.39 | 3.36 |
| Ferret | - | - | - | - | - | - |

**Table 8: Model performance comparison across different fine-grained visual reasoning tasks using IOU.**
| Model | Fine-Grained Visual Reasoning |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Arithmetic Reasoning | Logical Reasoning | Spatial Reasoning | Comparison | Sorting | Counting |  |
| Close-Source LVLMs |  |  |  |  |  |  |
| GPT-4o | - | 1.70 | 2.81 | - | 1.85 | 2.37 |
| GPT-4V | - | - | - | - | - | - |
| Qwen-VL-Max | 3.14 | 4.68 | 7.15 | 4.90 | 2.63 | 3.14 |
| Open-Source LVLMs |  |  |  |  |  |  |
| InternVL2-Llama3-76B | - | 1.73 | 1.43 | - | - | - |
| LLaVA-OV-Chat-72B | - | - | 1.92 | - | - | 1.08 |
| mPLUG-DocOwl1.5-Omni | - | - | - | - | - | - |
| LLaVA-V1.6-34B | - | 1.39 | 1.02 | - | - | - |
| Yi-VL-34B | - | - | - | - | - | - |
| CogVLM2-Chat-19B | - | - | - | - | - | - |
| Qwen2-VL-7B-Instruct | 1.74 | 4.66 | 4.99 | 1.92 | 1.18 | 1.66 |
| TextMonkey | - | - | 1.12 | - | 1.41 | - |
| MiniCPM-V2.6 | - | 1.60 | 1.75 | - | - | - |
| MiniCPM-Llama3-V2.5 | 1.16 | 1.49 | 2.37 | 2.64 | - | - |
| InternVL2-8B | - | 1.06 | 1.43 | - | - | - |
| mPLUG-Owl3 | - | - | - | - | - | - |
| Ferret | - | - | - | - | - | - |

### A.3 Document Category

We follow the strategy in Table 9 to divide the document images in each sub-task into five categories, i.e., General Document, Table-Based Document, Table-Text Document, Chart-Based Document and Infographic-Based Document.

**Table 9: The document image category for each sub task**
| Task | Sub Task | Category |
| --- | --- | --- |
| Text Recognition | TextOCR | General Document |
| OCR-VQA | General Document |  |
| Table Recognition | FinTabNet | Table-Based Document |
| PubTables-1M | Table-Based Document |  |
| Text Localization | Text2Bbox | General Document |
| Bbox2Text | General Document |  |
| Table Cell Localization | FinTabNet | Table-Based Document |
| PubTables-1M | Table-Based Document |  |
| Key Information Extraction | SROIE | General Document |
| WildReceipt | General Document |  |
| CORD | General Document |  |
| Document Forgery Detection | T-SROIE | General Document |
| DocTamper | General Document |  |
| Document Question Answering | DocVQA | General Document |
| WTQ | Table-Based Document |  |
| TAT-DQA | Table-Text Document |  |
| Chart Question Answering | ChartQA | Chart-Based Document |
| CharXiv | Chart-Based Document |  |
| Infographic Question Answering | InfographicVQA | Inforgraphic-Based Document |
| Arithmetic Reasoning | DUDE | General Document |
| WTQ | Table-Based Document |  |
| TAT-DQA | Table-Text Document |  |
| CharXiv | Chart-Based Document |  |
| InfographicVQA | Inforgraphic-Based Document |  |
| Logical Reasoning | DUDE | General Document |
| WTQ | Table-Based Document |  |
| TAT-DQA | Table-Text Document |  |
| CharXiv | Chart-Based Document |  |
| InfographicVQA | Inforgraphic-Based Document |  |
| Spatial Reasoning | DUDE | General Document |
| WTQ | Table-Based Document |  |
| CharXiv | Chart-Based Document |  |
| InfographicVQA | Inforgraphic-Based Documen |  |
| Comparison | DUDE | General Document |
| WTQ | Table-Based Document |  |
| TAT-DQA | Table-Text Document |  |
| CharXiv | Chart-Based Document |  |
| InfographicVQA | Inforgraphic-Based Document |  |
| Sorting | DUDE | General Document |
| WTQ | Table-Based Document |  |
| TAT-DQA | Table-Text Document |  |
| CharXiv | Chart-Based Document |  |
| InfographicVQA | Inforgraphic-Based Document |  |
| Counting | DUDE | General Document |
| WTQ | Table-Based Document |  |
| TAT-DQA | Table-Text Document |  |
| CharXiv | Chart-Based Document |  |
| InfographicVQA | Inforgraphic-Based Document |  |

### A.4 Samples for each Task in MMDocBench

#### A.4.1 Text Recognition

[Uncaptioned image]: extracted/5919359/samples/text_recognition.jpeg

- •
Question: Could you identify and read the text present in the provided image?
- •
Answer: Chaos Computer Club
- •
Bounding Box: [339, 362, 856, 498]

#### A.4.2 Text Localization

[Uncaptioned image]: extracted/5919359/samples/text_localization.png

- •
Text2Bbox Question: Can you find where ”Miami, Florida.” appears in the image and send back its position?
- •
Bounding Box: [406, 644, 576, 658]
- •
Bbox2Text Question: For the provided image and bounding box [406, 644, 576, 658], can you extract and return the text contained within the specified area?
- •
Text: Miami, Florida.

#### A.4.3 Table Recognition

[Uncaptioned image]: extracted/5919359/samples/table_recognition.jpeg

- •
Question: Could you break down the table into its cell components?
- •
Answers: [[Total genes, Total\nCGIs, Promoter\nCGIs, Inside\nCGIs, Downstream\nCGIs], [42, 49, 17, 31, 1], [Unvaried, 29, 10, 18, 1], [Unmet wave, 16, 6, 10, 0], [Met wave, 4, 1, 3, 0]]
- •
Bounding Box: [[[100,252,236,347], [320,217,385,382], [445,217,559,382], [614,217,687,382], [739,217,896,382]], [[100,394,130,482], [339,394,369,482], [489,394,513,482], [638,394,663,482], [812,394,823,482]], [[100,494,211,582], [339,494,366,582], [489,494,516,582], [638,494,663,582], [812,494,823,582]], [[100,588,252,676], [339,588,366,676], [494,588,510,676], [638,588,663,676], [809,588,823,676]], [[100,688,214,776], [345,688,361,776], [497,688,508,776], [644,688,657,776], [809,688,823,776]]]

#### A.4.4 Table Cell Localization

[Uncaptioned image]: extracted/5919359/samples/table_cell_localization.png

- •
Question: What is stored in the cell at the intersection of row 1 and column 3?
- •
Answer: Normal
- •
Bounding Box: [594, 283, 684, 373]

#### A.4.5 Key Information Extraction

[Uncaptioned image]: extracted/5919359/samples/key_information_extraction.png

- •
Question: Please perform detection and recognition of the text that has been categorized under the ”total amount” label.
- •
Answer: 50.000
- •
Bounding Box: [685, 740, 799, 762]

#### A.4.6 Document Forgery Detection

[Uncaptioned image]: extracted/5919359/samples/document_forgery_detection.png

- •
Question: Can you spot the words that have been falsified in the photo?
- •
Answer: results
- •
Bounding Box: [220, 707, 380, 755]

#### A.4.7 Document Question Answering

[Uncaptioned image]: extracted/5919359/samples/document_question_answering.png

- •
Question: When was the document prepared?
- •
Answer: December 31, 1943
- •
Bounding Box: [428, 448, 579, 464]

#### A.4.8 Chart Question Answering

[Uncaptioned image]: extracted/5919359/samples/chart_question_answering.png

- •
Question: How many app publishers were in Apple’s App Store in 2017?
- •
Answer: 143
- •
Bounding Box: [226, 222, 250, 235]

#### A.4.9 Infographic Question Answering

[Uncaptioned image]: extracted/5919359/samples/infographic_question_answering.png

- •
Question: What is the contact number for children and teens suffering from mental crisis, 1-833-456-4566, 1-800-668-6868, or 1-866-633-4220?
- •
Answer: 1-800-668-6868
- •
Bounding Box: [147, 829, 331, 845]

#### A.4.10 Counting

[Uncaptioned image]: x3.png

- •
Question: How many years did Shares issued under the ESPP exceed $2 million?
- •
Reasoning Type: counting
- •
Answer: 3
- •
Bounding Box: []
- •
Supporting Evidence:
–
Text: [4, 3, 4]
–
Bounding Box: [[795, 225, 807, 236], [673, 225, 685, 236], [918, 225, 929, 236]]

#### A.4.11 Arithmetic Reasoning

[Uncaptioned image]: x4.jpeg

- •
Question: how many years after the levanger church was built was the bamberg church built?
- •
Reasoning Type: arithmetic
- •
Answer: 96
- •
Bounding Box: []
- •
Supporting Evidence:
–
Text: [1998, 1902]
–
Bounding Box: [[843, 506, 918, 551], [843, 407, 917, 452]]

#### A.4.12 Logical Reasoning

[Uncaptioned image]: extracted/5919359/samples/logical_reasoning.jpg

- •
Question: Which step denotes the quality assurance part of the test performed for COVID-19?
- •
Reasoning Type: logical
- •
Answer: 8
- •
Bounding Box: [726,536,744,568]
- •
Supporting Evidence:
–
Text: [they pass Quality Control (QC)]
–
Bounding Box: [[712,685,811,737]]

#### A.4.13 Spatial Reasoning

[Uncaptioned image]: extracted/5919359/samples/spatial_reasoning.jpg

- •
Question: Where is the largest font word printed at the top or bottom of the document?
- •
Reasoning Type: spatial
- •
Answer: Top
- •
Bounding Box: []
- •
Supporting Evidence:
–
Text: [DIPLOMA]
–
Bounding Box: [[382, 222, 610,268]]

#### A.4.14 Comparison

[Uncaptioned image]: extracted/5919359/samples/comparison.jpg

- •
Question: was the finish in 1930 above/below 12?
- •
Reasoning Type: comparison
- •
Answer: above
- •
Bounding Box: []
- •
Supporting Evidence:
–
Text: [20]
–
Bounding Box: [[547, 358, 577, 388]]

#### A.4.15 Sorting

[Uncaptioned image]: extracted/5919359/samples/sorting.jpg

- •
Question:Which model has the least tie rate?\n * Your final answer must be grounded to some text that is explicitly written and relevant to the question in the chart.\n * If you need to answer multiple terms, separate them with commas.\n * Unless specified in the question (such as answering with a letter), you are required to answer the full names of subplots and/or labels by default.\n
- •
Reasoning Type: sorting
- •
Answer: GPT-3.5
- •
Bounding Box: [41,508,145,553]
- •
Supporting Evidence:
–
Text: [10.0%]
–
Bounding Box: [[536,493,642,575]]

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Missing details about the CoT on instruction.
soundness: 3
strengths: 1. Fine-grained visual and OCR-free document understanding is a good point for evaluating LVLMs.
2. The evaluated models are representative and SOTA.
summary: This paper introduces a benchmark named MMDocBench for OCR-free document understanding across 15 main tasks with 4,338 QA pairs and 11,353 supporting regions. Compared with existing benchmarks focusing on object-level recognition, fine-grained visual and OCR-free understanding is the main contribution. The paper evaluates 13 LVLMs on MMDocBench and provides some valuable findings.
weaknesses: 1. There are no experiments on the instruction sensitivity, while small modifications may cause large result differences.
2. The EM metric may not work in some scenarios since LVLMs's responses are diverse.
3. The analysis of experimental results should include the model architectures, training data, and model sizes to do a more in-depth analysis. How these factors affect the results.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. The density of document information is a major characteristic, but this paper does not discuss its impact on fine-grained VQA. Please investigate the impact of information density on fine-grained VQA performance in this benchmark.
2. In the cases presented in the appendix, some question areas are more obvious or structured. Are there any more dense and complex examples, such as a page of two-column paper?
3. I suggest calculating the proportion of question areas and the proportion of textual information on the image.
4. What impact does the output of the box have on hallucination problems?
5. For model design, does the author think there are any methods to improve LVLM's ability to handle fine-grained tasks？
6. Why not discuss the real fine-grained question, such as PointQA and other document-level referring tasks.
soundness: 3
strengths: 1. This paper is well-written and clear.
2. The grounding-like task is challenging and the authors explore further in the dense document-level scenario.
summary: This paper proposes a Fine-Grained Visual Document Understanding benchmark to evaluate LVLMs further. The comprehensive benchmark contains 15 main tasks across various document-level data, such as tables, figures, and so on. The authors are committed to exploring the capabilities of LVLMs for capturing visual details under more complex fine-grained visual document perception and reasoning tasks.
weaknesses: The differences between some document images and natural scene images require further analysis, as mentioned in the introduction. 
What does ‘’fine-grained‘’ mean? If it only refers to the answer in a certain part of the image, it seems to be no different from the existing evaluation benchmark (Chart QA, DocVQA, etc).

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. Line 236: `To solve the problem in MMDocBench` -> I am wondering what the problem refers to.

2. Can the authors provide more details about the chain-of-thought (CoT) instruction in Line 402?

3. While the paper utilizes human annotators to create QA pairs (line 257), it remains unclear whether all QAs were sourced exclusively from human annotation or if some were derived from existing benchmarks like DUDE.
soundness: 3
strengths: 1. The authors provide annotations of supporting regions, which increases the interpretability when analysis of the evaluation results. I think previous benchmarks rarely include the supporting regions.

2. The paper is easy to understand and the proposed benchmark is easy to follow.

3. The authors provide a comprehensive benchmark results of different LVLM models.
summary: The authors provide the MMDocBench benchmark with fine-grained document understanding questions and annotations. The authors collect the images from existing benchmarks, with human-annotated QA pairs. The authors also provide the benchmark results with recent open-source and proprietary LVLMs.
weaknesses: 1. **Lack of discussion with existing document understanding benchmarks**: The motivation of this paper is fine-grained visual understanding, with a specific focus on document understanding. However, I think the existing document understanding benchmarks such as DocVQA, InfoVQA [1], DUDE, TextVQA [2], ChartQA, MMLongBench-Doc [3] have already provided the fine-grained QAs. The QAs in these benchmarks have already required the fine-grained visual understanding ability of a specific region of the whole image. Many existing LVLM models, e.g., QWen2-VL also report their results on these existing document understanding benchmarks. The authors miss the discussion/citation with [1,2,3] and do not discuss the previous document understanding benchmarks in the related work section. What's the difference between MMDocBench and these existing document understanding benchmarks? I suggest the authors provide a comparative analysis table highlighting the key differences between MMDocBench and these existing benchmarks. This could include aspects like task types, annotation methods, evaluation metrics, and any novel features of MMDocBench.
    - [1] InfographicVQA, WCAV 2022
    - [2] Towards VQA models that can read, CVPR 2019
    - [3] MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations, NeurIPS 2024

2. **About the LVLM baselines**: I think document understanding and OCR are hot topics in LVLM research. The related work discussed in this manuscript is mainly 2023, and many new LVLM models are not discussed. In addition, I think some of the baseline models are not chosen properly. Please refer to 2.1 - 2.3 below for details:
   - 2.1: The authors lack the discussion or benchmark results with some existing LVLMs that are specially designed for document/figure/chart/OCR understanding, such as:
     - [4] TableLlama: Towards Open Large Generalist Models for Tables, NAACL 2024
     - [5] Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models, ECCV 2024
   - 2.2: In the related work section, the authors mention the LVLMs such as UReader and LLVAR that are focused on visual document understanding. Why do not report their results in Table 2?
   - 2.3: The selected proprietary models such as GPT-4o and GPT-4v are homogeneous, and the authors may consider other proprietary models such as Gemini-1.5 and Claude-3.5.

3. **The IoU score results:** According to Table 2 and lines 464-472, the authors report notably low IoU scores for all models, with GPT-4 achieving a mere 2.44% and the best-performing model reaching only 11.44%. This finding is perplexing, given the strong performance of these models on other metrics. Specifically, GPT-4, despite its low IoU score, achieves the highest EM score (66.40%) and the second-highest F1 score (68.87%). This discrepancy suggests that the IoU metric may not be the most suitable measure for evaluating region prediction in this context. I think It is unlikely that a model can consistently produce accurate answers without correctly identifying relevant regions.
    - Another possibility is that the author measured the IoU metric incorrectly. For example, the normalization of coordinates to the range [0, 1000] (lines 314-315) raises questions about the input provided to the LVLMs. But how did the models account for this coordinate normalization during region prediction? The varying image processing strategies employed by different LVLMs, such as the use of different tile sizes and resizing techniques, can impact the accuracy of coordinate transformation and region predictions. For example, 1) GPT-4o has two different image-processing strategies with different numbers of 512px tiles. 2) Some LVLM instruction following datasets (e.g., LLAVA-665k) use a different coordinate normalization method to the range [0, 1).
    - Additionally, Some LVLMs design special tokens (e.g., 
 and 
 for QWen2-VL) for referring grounding, do the authors consider that different LVLMs require different visual grounding prompt templates?
    - The authors claim that `Since some LVLMs cannot follow the instructions well during the test, especially for the region predictions, we revise the instructions based on the setting of these LVLMs to improve their performance` (Line 402), But the detailed prompts are missing and the implementation details are not clear. Do the authors think this step is fair for comparison?
    - These implementation details (input prompts, image processing strategies, special tokens, instruction following ability, and other facts that may impact the IoU score), while potentially significant, are not thoroughly discussed in the paper. The authors just claim that `All LVLMs face significant challenges in predicting supporting regions`. I think this claim is insufficient and lacks evidence.
    - I suggest the authors provide the following explanation:
      - Consider analyzing the relationship between IoU scores and answer accuracy (EM/F1-Score) to better understand this discrepancy.
      - Evaluate whether the IoU metric is the most appropriate measure for this task, or if alternative metrics might better capture region prediction performance. Does it is necessary to require the LVLMs to predict the box coordinates with the answers at the same time?
      - How the IoU metric was calculated and applied across different models.
      - Provide more implementation details, such as 1) how the coordinate normalization was communicated to the models during inference. 2) the potential impact of different image processing strategies on region prediction accuracy. 3) The details of the instruction following ability with different LVLM models and other variables that influence the results.

4. **The box prediction results are not easier for readers to assess**: The authors present the bounding box prediction results in Appendix A.4. It is hard for the reviewers to be aware of whether the box coordinates are correct or not. Can the authors provide visual examples with bounding boxes plotted on the images for a subset of representative cases? This would make it easier for readers to assess the accuracy of the predicted regions.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: (1) The novelty of the paper is limited, as similar fine-grained document understanding benchmarks have already been introduced, such as MMLongBench-Doc. Compared to MMLongBench-Doc, what advantages does your MMDocBench offer?

(2) In the experimental section, it would be beneficial to include human performance results on MMDocBench. This could provide an additional perspective on the benchmark's difficulty.

(3) Error Analysis. Could the authors conduct a more in-depth analysis of the models' performance on MMDocBench by examining specific cases? For example, analyzing the reasons for LVLMs' reasoning errors or why different models show performance variations. Understanding the root causes behind error cases would be valuable for further improving model performance.
soundness: 3
strengths: (1) A new fine-grained document understanding benchmark is proposed, covering 15 main tasks with 4,338 QA pairs and 11,353 supporting regions.

(2) Focusing on fine-grained image information and local regions, this benchmark increases the difficulty compared to previous benchmarks that emphasize global image information.

(3) The performance of multiple open-source and proprietary LVLMs is evaluated on this benchmark.
summary: This paper proposes a new benchmark for Fine-Grained Visual Document Understanding. Unlike traditional benchmarks that focus on overall image and object-level understanding, MMDocBench emphasizes evaluating the MLLM's comprehension of fine-grained image details and localized regions. MMDocBench includes a variety of sub-tasks to assess the LVLM's capability in fine-grained document understanding from multiple perspectives. Results show that both existing open-source LVLMs and proprietary LVLMs have considerable room for improvement on MMDocBench.
weaknesses: The paper lacks sufficient novelty, as there are already earlier works with similar approaches, such as MMLongBench-Doc. In the experimental section, the authors could conduct additional experiments for deeper analysis, such as incorporating human evaluation results and providing specific error case analysis.