mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding
Authors: Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, Jingren Zhou
Accepted at: acl25

mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding

Anwen Hu1*, Haiyang Xu1*†, Liang Zhang2*‡, Jiabo Ye1, Ming Yan1†
Ji Zhang1, Qin Jin2, Fei Huang1, Jingren Zhou1
1Alibaba Group, 2Renmin University of China
{huanwen.haw,shuofeng.xhy,ym119608}@alibaba-inc.com

(a) The efficiency and performance comparison.

(b) Detailed explanation with evidence.

(c) Overall structure parsing for a document.

Figure 1: (a) mPLUG-DocOwl2 achieves state-of-the-art Multi-page Document Understanding performance with faster inference speed and less GPU memory; (b-c) mPLUG-DocOwl2 is able to provide a detailed explanation containing the evidence page as well as the overall structure parsing of the document.

Abstract

Multimodel Large Language Models(MLLMs) have achieved promising OCR-free Document Understanding performance by increasing the supported resolution of document images. However, this comes at the cost of generating thousands of visual tokens for a single document image, leading to excessive GPU memory and slower inference times, particularly in multi-page document comprehension. In this work, to address these challenges, we propose a High-resolution Doc-Compressor module to compress each high-resolution document image into 324 tokens, guided by low-resolution global visual features. With this compression module, to strengthen multi-page document comprehension ability and balance both token efficiency and question-answering performance, we develop the DocOwl2 under a three-stage training framework: Single-image Pretraining, Multi-

*Equal contribution.
†Corresponding authors.
‡Work done during internship at Alibaba Group.
image Continue-pretraining, and Multi-task Finetuning. DocOwl2 sets a new state-of-the-art across multi-page document understanding benchmarks and reduces first token latency by more than 50%. Compared to single-image MLLMs trained on similar data, our DocOwl2 achieves comparable single-page understanding performance with less than 20% of the visual tokens. Our codes, models, and data are released at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2.

1 Introduction

Understanding a multi-page document or news video is common in human daily life. To tackle such scenarios, Multimodal Large Language Models (MLLMs) (Ye et al., 2023c,d, 2024; Bai et al., 2023; Liu et al., 2023b) should be equipped with the ability to understand multiple images with rich visually-situated text information. Different from natural images mainly comprising of objects, comprehending document images asks for a more fine-grained perception to recognize all texts. By encoding high-resolution document images with thousands of tokens, state-of-the-art Multimodal LLMs (Ye et al., 2023b; Hu et al., 2024; Chen et al., 2024; Dong et al., 2024a,b) achieves promising OCR-free document understanding performance, e.g., InternVL 2 (Chen et al., 2024) costs a average of 3k visual tokens for a A4-sized document page. However, as shown in Fig. 1(a), such long visual tokens not only result in long inference time but also occupy too much GPU memory, making it difficult to understand a complete document or video.

In this work, we argue that visual tokens of document images can be further compressed while maintaining both layout and most textual information. Existing compressing architecture in MLLMs are hard to balance information retention and token efficiency during document image encoding. As shown in Fig. 2(a), independently compressing each crop of a document image (Li et al., 2024b; Hu et al., 2024) could reduce visual tokens of each sub-image but still results in a long sequence of visual tokens after concatenating all sub-images. Leveraging learnable queries (Bai et al., 2023; Li et al., 2023a; Ye et al., 2023c) or selected tokens (Liu et al., 2024) as compressing guidance could produce an identical length of tokens for any resolution but overlook the overall layout information, as shown in Fig. 2(b). Layout-aware guidance is important for compressing visual features of document images because texts within a layout region are semantic-coherent and easier to summarize. For example, in a two-column paper, texts belonging to the ‘Related Work’ section are difficult to summarize with texts on the same line but belonging to the ‘Method’ section.

In this work, as shown in Fig. 2(c), we propose a layout-aware compressing architecture High-resolution DocCompressor based on cross-attention. Considering that a global low-resolution image can well capture the overall layout information, we utilize visual features of a global low-resolution image as the compressing guidance (query). Each visual feature in the global feature map just captures the layout information of partial regions. Therefore, each query attending to all high-resolution features will not only make information compression more difficult but also increase computation complexity. To summarize text information within a layout region, for each query from the global feature map, a group of high-resolution features with identical relative positions in the raw image is collected as compressing objects, sometimes spanning multiple sub-images. Besides, since the vision-to-text (V2T) module of MLLMs could convert visual features into textual feature space, we argue that compressing visual features after the vision-to-text module could better maintain textual semantics in document images. Therefore, based on the architecture of DocOwl 1.5 (Hu et al., 2024), we propose mPLUG-DocOwl2 by placing the High-resolution DocCompressor after its V2T module: H-Reducer. To take full advantage of the compressing method, our model DocOwl2 is trained with a three-stage framework: Single-image Pretraining, Multi-image Continue-Pretaining, and Multi-task Finetuning to support both single-image and multi-image/frame understanding.

Our contributions in this work are three-fold:

• We propose a novel layout-aware compressing architecture to greatly reduce visual tokens of high-resolution document images.

• We design a three-stage training framework to empower DocOwl2 with both single-page and multi-page document understanding abilities.

• DocOwl2 achieves state-of-the-art performance on Multi-page Document understanding benchmarks with < 50% First Token Latency. Compared with state-of-the-art MLLMs with similar model size
and training data, DocOwl2 achieves comparable performance with < 20% visual tokens on 10 single-image document benchmarks.

2 Related Work

OCR-free Visual Document Understanding. Visual Document Understanding aims to comprehend images with rich text information, including scans of document pages (Mathew et al., 2021; Tito et al., 2022; Landeghem et al., 2023; Zhang et al., 2023; Wei et al., 2023), infographics (Mathew et al., 2022), charts (Masry et al., 2022; Kafle et al., 2018; Methani et al., 2020; Kahou et al., 2018), tables images (Pasupat and Liang, 2015; Chen et al., 2020; Zhong et al., 2020), webpage screenshots (Tanaka et al., 2021; Chen et al., 2021) and natural images with scene texts (Singh et al., 2019; Sidorov et al., 2020; Hu et al., 2021). Recently, many Multimodal Large Language Models have been proposed to perform visual document understanding in an OCR-free manner. mPLUG-DocOwl (Ye et al., 2023a) and UReader (Ye et al., 2023b) first propose to unify different tasks across 5 types of document images in the seq-to-seq format. To encode rich text information in high-resolution images, UReader (Ye et al., 2023b) proposes a Shape-adaptive Cropping Module to cut the raw image into multiple low-resolution sub-images and utilizes an identical low-resolution encoder to encode both sub-images and a global image. Monkey (Li et al., 2023b) proposes to employ a sliding window to partition high-resolution images and a resampler to reduce redundant information of each sub-image. mPLUG-DocOwl15 (Hu et al., 2024) increases the basic resolution of the low-resolution encoder and replaces the Visual Abstractor (Ye et al., 2023c) with 1 simple convolution layer to better maintain the structure information. DocPedia (Feng et al., 2023) directly processes high-resolution images in the frequency domain. CoGAgent (Hong et al., 2023) proposes to utilize a high-resolution encoder to encode high-resolution visual features and a low-resolution encoder to encode low-resolution global features. Series work of InternLM-XComposer (Dong et al., 2024b,a) and InternVL (Chen et al., 2024) further optimize the cropping method or increase the cropping number and greatly improves the OCR-free Document Understanding performance. These works achieve promising performance but suffer from too many visual tokens for a high-resolution image (always > 1k tokens for a common A4-sized document page), which hinders the development of OCR-free multi-page document understanding. Recent works explore enhancing the document understanding abilities of general-purpose MLLMs. However, they are either resource-intensive (Li et al., 2025) or can not achieve adorable performance (Kim and Seo, 2024). In contrast, we explore a more efficient model structure and training paradigm for multi-page document understanding. We believe these efforts can help to build models with more comprehensive capabilities.

Visual Feature Compressing. Reducing visual tokens of a single image enables a Multimodal Large Language Model with limited maximum sequence length to leverage more images as contexts to perform complex multimodal tasks, such as video understanding, embodied interaction, or multi-page document understanding. Some works (Zhang et al., 2024b; Shi et al., 2024; Li et al., 2024c) propose to ensemble and compress visual features from multiple vision encoders. For example, Eagle compresses visual features of 5 vision encoders to identical lengths of visual tokens and then fuses them by channel-level concatena-

![Illustrations of different compressing methods for OCR-free document understanding.](page_172_120_1102_312.png)

Figure 2: Illustrations of different compressing methods for OCR-free document understanding.
In the flowchart shown on the slide titled “Introduction to Markov Processes,” the two steps that come between “Stop” and “Prepare to stop” are “Prepare to go” and “Proceed.” The sequence of traffic lights changes from Stop (red light) to Prepare to go (amber light), then to Proceed (light), and finally cycles back to Prepare to stop (yellow or amber light), completing the loop.

![The architecture of DocOwl2. Each image is independently encoded by the pipeline of Shape-adaptive Cropping, High-resolution Visual Encoding and High-resolution DocCompressor.](page_184_153_1079_627.png)

Figure 3: The architecture of DocOwl2. Each image is independently encoded by the pipeline of Shape-adaptive Cropping, High-resolution Visual Encoding and High-resolution DocCompressor.

tion. Besides, there are also explorations to compress visual features of general images with fewer learnable queries, such as the Resampler (Alayrac et al., 2022; Bai et al., 2023), Abstractor (Ye et al., 2023c,d) and Q-former (Li et al., 2023a). Randomly initialized learnable queries can ensemble object information in general images but is hard to summarize rich text information in high-resolution document images. As a compromise solution, TokenPacker (Li et al., 2024b) proposes to compress each sub-image with its downsampled visual features as the query to perform cross-attention. TokenPacker just reduces each sub-image’s visual tokens, thus still creates more than 1k visual tokens when processing high-resolution document images. TextMonkey (Liu et al., 2024) first filters valuable visual tokens and then uses them as guidance to aggregate all visual tokens. Due to that valuable visual tokens are selected by measuring the token similarity, visual information of partial regions may not be covered and thus not well compressed during following cross-attention. In this work, our High-resolution DocCompressor leverages visual features from the row-resolution global images as the query, the ensembled feature map of sub-images as key and value. This not only produces a fixed number of visual tokens for images of any resolution but also covers all areas during compression. Compared to Mini-Gemini (Li et al., 2024c) which compresses general visual features, there are major two differences. Firstly, we make full use of global visual features and sub-image features produced by an identical low-resolution vision encoder and don’t need to add an extra high-resolution encoder. Secondly, for better summarizing textual information in document images, our cross-attention is applied based on visual features that have been aligned with textual features of LLM. We argue that directly compressing outputs of the vision encoder loses semantic information while comprising features aligned with LLM is like summarizing texts and can better maintain textual semantics in document images. We conduct fair comparisons to support this hypothesis.

3 mPLUG-DocOwl2

As shown in Fig. 3, for multiple document images, DocOwl2 leverages a High-resolution Visual Encoding module and a High-resolution DocCompressor to encode each image independently. After that, a LLM is utilized for multimodal understanding.

3.1 High Resolution Vision Encoding

Following UReader (Ye et al., 2023b) and DocOwl 1.5 (Hu et al., 2024), DocOwl2 utilizes a
parameter-free Shape-adaptive Cropping Module to preprocess high-resolution images. Concretely, it cuts each high-resolution image \( I \) into \( R \times C \) size-fixed sub-images \( I^s = \{I_{xy}^s\}, 1 \leq x \leq R, 1 \leq y \leq C \), where cropping rows \( R \) and columns \( C \) are flexibly decided based on the raw resolution of \( I \). Besides, to maintain the overall layout information, the raw image is also directly resized to a global image \( I^g \).

After the cropping module, a low-resolution transformer-based vision encoder ViT (Dosovitskiy et al., 2021) is utilized to independently extract vision features of each sub-image and the global image as follows:

\[
V^g = \mathrm{ViT}(I^g),
\]
\[
V_{xy}^s = \mathrm{ViT}(I_{xy}^s), 1 \leq x \leq R, 1 \leq y \leq C,
\]

where both \( V^g \) and \( V_{xy}^s \) are visual features with the shape of \( h \times w \times d \), \( d \) is the feature dimension and \( w, h \) are the width and height of the feature map.

Following DocOwl 1.5, after the ViT, for each sub-image or global image, we apply a vision-to-text module H-Reducer to ensemble horizontal 4 features by a convolution layer and align the feature dimension with the Large Language Model with a fully connected layer. The calculation of H-Reducer is represented as follows:

\[
\hat{V} = \mathrm{FC}(\mathrm{Conv}(V)),
\]
\[
V \in \{V^g, V_{xy}^s\}, 1 \leq x \leq R, 1 \leq y \leq C,
\]

where the shape of the visual feature map \( \hat{V} \) is \( h \times \frac{w}{4} \times \hat{d} \), \( \hat{d} \) is the dimension of hidden states of the large language model.

3.2 High-resolution DocCompressor

A sentence/paragraph/document of text tokens can be compressed into fewer summary vectors while maintaining most semantics (Cheng et al., 2024; Ge et al., 2024; Chevalier et al., 2023). Besides, since visual features have been aligned with the textual feature space of large language models, the visual tokens of document images after the vision-to-text module can also be treated as textual tokens encoding different parts of textual information in the image. Thus, taking into account these two points, in this work, we argue that visually situated textual information of document images can also be further compressed into fewer tokens, especially after the vision-to-text alignment.

Texts from the same layout region are more appropriate to be fused into fewer tokens. After the vision-to-text module H-Reducer, the global visual feature \( \hat{V}^g \) mainly encodes the overall text layout information while visual features of sub-images \( \{\hat{V}_{xy}^s\} \) capture detailed textual information. Besides, due to both the global image and cropped sub-images come from an identical image, there is a clear mapping between the visual tokens of \( \hat{V}^g \) and \( \{\hat{V}_{xy}^s\} \). As shown in Fig. 3, each visual token in \( \hat{V}^g \) can be aligned with \( R \times C \) visual tokens in \( \{\hat{V}_{xy}^s\} \). Therefore, we first re-organize feature maps of cropping images (\( \{\hat{V}_{xy}^s\}, 1 \leq x \leq R, 1 \leq y \leq C \)) to a complete feature map \( \hat{V}^s \) according to their positions in the raw high-resolution image. Then, for each visual token in the feature map \( \hat{V}^g \) of the global image, we collect its corresponding \( R \times C \) visual tokens from \( \hat{V}^s \) as the key and value, the cross-attention layer is calculated as follows:

\[
\bar{v}_{ij} = \sigma(\frac{W^q \hat{v}_{ij}^g W^k \hat{v}_{ij}^{s T}}{\sqrt{d_k}})W^v \hat{v}_{ij}^g + \hat{v}_{ij}^g,
\]
\[
\hat{v}_{ij}^g \in \hat{V}^g, 1 \leq i \leq h, 1 \leq j \leq w/4,
\]
\[
\hat{v}_{ij}^s = [\hat{v}_{i'j'}^s] \subset \hat{V}^s,
\]
\[
(i - 1)R + 1 \leq i' \leq iR,
\]
\[
(j - 1)C + 1 \leq j' \leq jC,
\]

where \( \hat{v}_{ij}^g \) is a visual token from the global feature map and \( \hat{v}_{ij}^s \) are visual tokens from the re-organized feature map of cropping images. \( \hat{v}_{ij}^g \) and \( \hat{v}_{ij}^s \) correspond to the same area in the raw image. \( W^* \) are learnable matrices. \( \sigma \) refers to softmax.

After high-resolution compressing, the compressed feature map of each image is organized into a sequence \( \bar{V} = [\bar{v}_1, \bar{v}_2, ..., \bar{v}_{h \times \frac{w}{4}}] \) for subsequent understanding of the large language model.

3.3 Multi-image Modeling with LLM

Through the high-resolution compressing, the number of visual tokens for each high-resolution image is reduced from \( (R \times C + 1) \times h \times \frac{w}{4} \) to \( h \times \frac{w}{4} \). Such efficient vision encoding allows joint understanding of multiple document images with Large Language Models. To help the LLM better distinguish visual features from different images and understand the ordinal number of images, we add a textual ordinal token '<img x>' before the visual features of each image, where \( x \) is the ordinal number. Overall, the decoding of the decoder for multiple images is as follows:

\[
Y = \mathrm{LLM}([P_1; \bar{V}_1; P_2; \bar{V}_2, ..., P_n; \bar{V}_n; T])
\]
where \( [\cdot] \) means the concatenation operation, \( n \) is the number of images, \( P_x, 1 \leq x \leq n \) is the textual embedding of the ordinal token `<img x>`, \( V_x \) is the visual features for each image, \( T \) is the textual instruction and \( Y \) is the predicted answer.

3.4 Model Training

DocOwl2 is trained with three stages: Single-image Pre-training, Multi-image Continue Pretraining, and Multi-task Finetuning.

At the first stage, to ensure the compressed visual tokens can encode most visual information, especially visually situated texts, we first perform Unified Structure Learning as DocOwl 1.5, which covers the learning of struct-aware document parsing, table parsing, chart parsing and natural image parsing of a single image.

After Single-image Pretraining, to empower our model with the ability to correlate multiple images, we further perform Multi-image Continue Pretraining with a struct-aware multi-page document parsing dataset MP-DocStruct1M. We design two symmetrical tasks of multi-image understanding: Multi-page Text Parsing and Multi-page Text Lookup. Given successive page images in a document, the Multi-page Text Parsing instructs the model to parse texts of specified one or two pages, such as ‘Recognize texts in image 2 and image 10.’. As for the Multi-page Text Lookup task, with texts from 1-2 pages as input, the model is required to predict the concrete ordinal number of images containing these texts, for example, ‘Looking for the image with text <doc> ...</doc> and <doc> ...</doc>’. Besides multi-image tasks, during this stage, we also randomly chose partial training samples from the first stage to avoid the catastrophic forgetting of structure parsing across different types of images.

Finally, we ensemble both single-page and multi-page instruction tuning datasets of document understanding to perform multi-task tuning. The task format includes concise question answering and detailed explanations.

The detailed introduction of training datasets of DocOwl2 can be found in Appendix A.1. More training details are introduced in Appendix A.2.

4 Experiments

4.1 Main Results

We compare DocOwl2 with state-of-the-art MLLMs on 10 single-image document understanding benchmarks, 2 Multi-page document Understanding benchmarks, and 1 text-rich video understanding benchmark. Both question-answering performance and the First Token Latency (seconds) are considered to show the effectiveness of our model.

Single-image Document Understanding Compared with MLLMs (Ye et al., 2023b; Liu et al., 2024; Li et al., 2024b; Bai et al., 2023) with < 1k visual tokens, our DocOwl2 achieves better or comparable performance on 10 benchmarks. Especially, with fewer visual tokens, our model outperforms both TextMonkey (Liu et al., 2024) and TokenPacker (Li et al., 2024b) which also aim to compress visual tokens, showing that our layout-aware architecture High-resolution DocCompressor is better at summarizing and maintaining textual information in high-resolution document images. Besides, compared with state-of-the-art MLLMs (Dong et al., 2024b; Chen et al., 2024; Hu et al., 2024) with > 1k visual tokens, DocOwl2 achieves > 80% performance on 7/10 benchmarks while with < 20% visual tokens. A more comprehensive comparison with existing OCR-free models can be found in Appendix B.1.

Furthermore, we compare the First Token Latency (seconds) on the most frequently compared dataset DocVQA (Mathew et al., 2021). As shown in Table 2, the far greater number of visual tokens enable InternVL 2 (Chen et al., 2024) and IXC 2.5 (Dong et al., 2024b) to achieve better performance but also result in higher inference time. Considering the model architecture and training data, it’s most fair to compare DocOwl2 with DocOwl 1.5. After adding the High-resolution DocCompressor, with similar training data of OCR learning, DocOwl2 achieves 98% performance of DocOwl 1.5 while reducing 50% First Token Latency with just 20% visual tokens, validating the effectiveness of our compressor for compressing visually-situated text information. Similar comparisons on more benchmarks can be found in Appendix B.1.

Multi-page/Video Document Understanding In such benchmarks, we choose recently proposed Multimodal LLMs (Zhang et al., 2024a; Laurençon et al., 2024; Li et al., 2024a) with multi-page OCR-free document understanding abilities and can be fed into more than 10 images under a single A100-80G as baselines. As shown in Table 3, with fewer visual tokens for a single image/frame, DocOwl2 achieves better question-answering performance and much less First Token Latency, validating the good balance of DocOwl2 between the
Table 1: Comparison with OCR-free methods on single-image document understanding tasks. ‘Token^V’ means the average number of visual tokens of a single image. ‘Bold’ means SOTA performance within the group and ‘Underline’ means achieving 80% SOTA performance among all baselines.

<table>
  <tr>
    <th>Model</th>
    <th>Domain</th>
    <th>Size</th>
    <th>Token^V</th>
    <th colspan="2">Doc VQA</th>
    <th colspan="2">Info VQA</th>
    <th colspan="2">Deep Form</th>
    <th>KLC</th>
    <th>WTQ</th>
    <th colspan="2">Tab Fact</th>
    <th>Chart QA</th>
    <th colspan="2">Text VQA</th>
    <th>Text Caps</th>
    <th>Visual MRC</th>
  </tr>
  <tr>
    <td>IXC 2.5</td>
    <td>General</td>
    <td>7B</td>
    <td>~5,118</td>
    <td>90.9</td>
    <td>69.9</td>
    <td>71.2</td>
    <td>-</td>
    <td>53.6</td>
    <td>85.2</td>
    <td>82.2</td>
    <td>78.2</td>
    <td>-</td>
    <td>307.5</td>
  </tr>
  <tr>
    <td>InternVL2</td>
    <td>General</td>
    <td>8B</td>
    <td>~3,133</td>
    <td><b>91.6</b></td>
    <td><b>74.8</b></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td><u>83.3</u></td>
    <td><u>77.4</u></td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DocOwl1.5</td>
    <td>Document</td>
    <td>8B</td>
    <td>~1,698</td>
    <td>82.2</td>
    <td>50.7</td>
    <td>68.8</td>
    <td><u>38.7</u></td>
    <td>40.6</td>
    <td>80.2</td>
    <td>70.2</td>
    <td>68.6</td>
    <td><u>131.6</u></td>
    <td>246.4</td>
  </tr>
  <tr>
    <td>UReader</td>
    <td>Document</td>
    <td>7B</td>
    <td>~841</td>
    <td>65.4</td>
    <td>42.2</td>
    <td>49.5</td>
    <td>32.8</td>
    <td>29.4</td>
    <td>67.6</td>
    <td>59.3</td>
    <td>57.6</td>
    <td>118.4</td>
    <td>221.7</td>
  </tr>
  <tr>
    <td>TextMonkey</td>
    <td>Document</td>
    <td>9B</td>
    <td>768</td>
    <td>73.0</td>
    <td>28.6</td>
    <td>59.7</td>
    <td><u>37.8</u></td>
    <td>31.9</td>
    <td>-</td>
    <td>66.9</td>
    <td>65.9</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>TokenPacker</td>
    <td>Document</td>
    <td>13B</td>
    <td>~467</td>
    <td>58.0</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>OwenVL</td>
    <td>General</td>
    <td>9B</td>
    <td>256</td>
    <td>65.1</td>
    <td>35.4</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>65.7</td>
    <td>63.8</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DocOwl2</td>
    <td>Document</td>
    <td>8B</td>
    <td>324</td>
    <td><b>80.7</b></td>
    <td><b>46.4</b></td>
    <td><u>66.8</u></td>
    <td><u>37.5</u></td>
    <td><u>36.5</u></td>
    <td><u>78.2</u></td>
    <td><u>70.0</u></td>
    <td><u>66.7</u></td>
    <td><u>131.8</u></td>
    <td>217.4</td>
  </tr>
</table>

Table 2: Comparison of performance and inference speed on DocVQA. ‘FTL(s)’ refers to the First Token Latency (seconds). ‘IL(s)’ refers to Instance Latency.

<table>
  <tr>
    <th>Model</th>
    <th>Size</th>
    <th>Token^V</th>
    <th>FTL(s)</th>
    <th>IL(s)</th>
    <th>ANLS†</th>
  </tr>
  <tr>
    <td>InternVL 2</td>
    <td>8B</td>
    <td>~3,198</td>
    <td>0.94</td>
    <td>2.46</td>
    <td>91.6</td>
  </tr>
  <tr>
    <td>IXC 2.5</td>
    <td>7B</td>
    <td>~7,395</td>
    <td>3.73</td>
    <td>7.57</td>
    <td>90.9</td>
  </tr>
  <tr>
    <td>DocOwl 1.5</td>
    <td>8B</td>
    <td>~1,806</td>
    <td>0.58</td>
    <td>1.84</td>
    <td>82.2</td>
  </tr>
  <tr>
    <td>Idefics2</td>
    <td>8B</td>
    <td>64</td>
    <td><b>0.21</b></td>
    <td><b>0.62</b></td>
    <td>67.3</td>
  </tr>
  <tr>
    <td>Idefics2</td>
    <td>8B</td>
    <td>320</td>
    <td>0.89</td>
    <td>2.15</td>
    <td>74.0</td>
  </tr>
  <tr>
    <td>TextMonkey</td>
    <td>9B</td>
    <td>768</td>
    <td>0.58</td>
    <td>1.74</td>
    <td>73.0</td>
  </tr>
  <tr>
    <td>DocOwl2</td>
    <td>8B</td>
    <td>324</td>
    <td>0.26</td>
    <td>0.66</td>
    <td>80.7</td>
  </tr>
</table>

OCR-free document understanding performance and token efficiency.

4.2 Ablation Study

Compressor Architecture. We compare different compressing architectures with an identical training pipeline of Single-image Pretraining and Single-image Document Understanding Finetuning, keeping both training data and setting consistent.

As shown in Table 4, compared with CABstractor (Cha et al., 2023), the Resampler (Bai et al., 2023) achieves worse document understanding performance (r2 vs r1). This shows that due to no prior knowledge, such as spatial relationship, is leveraged as compressing guidance, utilizing queries learned from scratch to compress rich visually-situated text information is more challenging than simple adaptive mean pooling. Our High-resolution DocCompressor outperforms CABstractor (r3 vs r2), validating that leveraging global visual features as layout-aware guidance can better distinguish the information density of each fine-grained visual feature and therefore maintain more visually-situated text information.

Instead of placing the compressor after the vision-to-text module H-Reducer, we also try inserting it between the vision encoder and the vision-to-text module. Such a setting results in performance decreases across three datasets (r4 vs r3), validating our hypothesis that compressing features after the vision-to-text module is like summarizing textual features and can maintain more textual semantics while compressing visual features after the visual encoder loses more visually situated text information. Besides, without aligning each query token in the global feature map with \( R \times C \) fine-grained visual tokens from the re-organized feature map to perform attention within a group as Eq. (9), we try utilizing each query token to attend all visual tokens of sub-images. Such complete attention not only brings higher computational complexity but also causes performance decreases (r5 vs r3), showing that the positional correspondence between the global visual map and the re-organized fine-grained visual map is a reliable prior knowledge for compressing visual features efficiently. Furthermore, directly performing mean pooling on each group of \( R \times C \) fine-grained visual features underperforms utilizing global visual features as the query (r6 vs r3), proving the importance of reliable guidance during compressing.

Compared with 2 layers of cross-attention, decreasing cross-attention layers bring a slight performance increase on DocVQA (Mathew et al., 2021) but more performance decrease on WikiTablesQA (WTQ) (Pasupat and Liang, 2015) (r7 vs r3). Further increasing to 4 layers doesn’t significantly improve performance (r8 vs r3). This shows that compressing high-resolution visual features doesn’t require a deep neural network. Finally, increasing the maximum number of crops and the base resolution of the global image or each sub-image are two main strategies to increase the supported input resolution. Our experiments show that increasing the cropping number (r9 vs r3) or basic resolution (r10 vs r9) benefits the document understanding performance. Increasing basic resolution brings more improvement because of more visual
Table 3: The OCR-free performance comparison on multi-page/video document understanding benchmarks. ‘FTL(s)’ refers to the First Token Latency. ‘Token^V’ means the average number of visual tokens of a single page/frame. LLaVA-Next-Interleave-7B*: fine-tuned with the same data of DocOwl2 for held-in evaluation.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Token^V</th>
    <th colspan="2">MP-DocVQA</th>
    <th colspan="2">DUDE</th>
    <th colspan="2">NewsVideoQA</th>
  </tr>
  <tr>
    <th>FTL(s)↓</th>
    <th>ANLS↑</th>
    <th>FTL(s)↓</th>
    <th>ANLS↑</th>
    <th>FTL(s)↓</th>
    <th>ANLS↑</th>
  </tr>
  <tr>
    <td>LongVA-7B</td>
    <td>~2,029</td>
    <td>2.13</td>
    <td>60.80</td>
    <td>2.26</td>
    <td>38.37</td>
    <td>4.29</td>
    <td>50.61</td>
  </tr>
  <tr>
    <td>Idefics-3B</td>
    <td>~838</td>
    <td>2.26</td>
    <td>67.15</td>
    <td>2.29</td>
    <td>38.65</td>
    <td>6.39</td>
    <td>60.16</td>
  </tr>
  <tr>
    <td>LLaVA-Next-Interleave-7B</td>
    <td>729</td>
    <td>1.56</td>
    <td>44.87</td>
    <td>1.47</td>
    <td>28.03</td>
    <td>4.35</td>
    <td>56.66</td>
  </tr>
  <tr>
    <td>LLaVA-Next-Interleave-7B*</td>
    <td>729</td>
    <td>1.56</td>
    <td>49.99</td>
    <td>1.47</td>
    <td>39.02</td>
    <td>4.35</td>
    <td>62.38</td>
  </tr>
  <tr>
    <td>DocOwl2-8B</td>
    <td>324</td>
    <td><b>0.95</b></td>
    <td><b>69.42</b></td>
    <td><b>0.94</b></td>
    <td><b>46.77</b></td>
    <td><b>1.17</b></td>
    <td><b>64.09</b></td>
  </tr>
</table>

Table 4: Ablation study about the architecture of the compressor on single-image document benchmarks. ‘Img<sup>base</sup>’ refers to the basic resolution of the global image and each sub-image.

<table>
  <tr>
    <th rowspan="2">Img<sup>base</sup></th>
    <th rowspan="2">Crop</th>
    <th colspan="5">Compressor</th>
    <th colspan="3">DocVQA</th>
    <th colspan="2">WTQ</th>
    <th colspan="2">ChartQA</th>
  </tr>
  <tr>
    <th>Name</th>
    <th>Compressing</th>
    <th>Layer</th>
    <th>Position</th>
    <th>Token^V</th>
    <th>DocVQA</th>
    <th>WTQ</th>
    <th>ChartQA</th>
    <th>DocVQA</th>
    <th>WTQ</th>
    <th>ChartQA</th>
  </tr>
  <tr>
    <td>r1</td>
    <td>448</td>
    <td>9</td>
    <td>Resampler</td>
    <td>learnable query</td>
    <td>-</td>
    <td>after H-Reducer</td>
    <td>256</td>
    <td>69.0</td>
    <td>29.4</td>
    <td>66.6</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>r2</td>
    <td>448</td>
    <td>9</td>
    <td>CABstractor</td>
    <td>Adaptive Mean</td>
    <td>-</td>
    <td>after H-Reducer</td>
    <td>256</td>
    <td>73.0</td>
    <td>32.6</td>
    <td>67.6</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>r3</td>
    <td>448</td>
    <td>9</td>
    <td>DocCompressor</td>
    <td>Group Att</td>
    <td>2</td>
    <td>after H-Reducer</td>
    <td>256</td>
    <td>76.1</td>
    <td>35.1</td>
    <td>69.2</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>r4</td>
    <td>448</td>
    <td>9</td>
    <td>DocCompressor</td>
    <td>Group Att</td>
    <td>2</td>
    <td>after ViT</td>
    <td>256</td>
    <td>75.7</td>
    <td>33.3</td>
    <td>68.7</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>r5</td>
    <td>448</td>
    <td>9</td>
    <td>DocCompressor</td>
    <td>Complete Att</td>
    <td>2</td>
    <td>after H-Reducer</td>
    <td>256</td>
    <td>74.4</td>
    <td>33.7</td>
    <td>68.2</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>r6</td>
    <td>448</td>
    <td>9</td>
    <td>DocCompressor</td>
    <td>Group Mean</td>
    <td>-</td>
    <td>after H-Reducer</td>
    <td>256</td>
    <td>74.6</td>
    <td>31.9</td>
    <td>68.2</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>r7</td>
    <td>448</td>
    <td>9</td>
    <td>DocCompressor</td>
    <td>Group Att</td>
    <td>1</td>
    <td>after H-Reducer</td>
    <td>256</td>
    <td>76.4</td>
    <td>34.2</td>
    <td>69.2</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>r8</td>
    <td>448</td>
    <td>9</td>
    <td>DocCompressor</td>
    <td>Group Att</td>
    <td>4</td>
    <td>after H-Reducer</td>
    <td>256</td>
    <td>75.9</td>
    <td>35.8</td>
    <td>70.1</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>r9</td>
    <td>448</td>
    <td>12</td>
    <td>DocCompressor</td>
    <td>Group Att</td>
    <td>2</td>
    <td>after H-Reducer</td>
    <td>256</td>
    <td>76.8</td>
    <td>35.6</td>
    <td>69.5</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>r10</td>
    <td>504</td>
    <td>12</td>
    <td>DocCompressor</td>
    <td>Group Att</td>
    <td>2</td>
    <td>after H-Reducer</td>
    <td>324</td>
    <td>78.7</td>
    <td>36.7</td>
    <td>69.4</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 5: Ablation study of DocCompressor with LLaVA-Next-Interleaves. ‘LLaVA-NI’ refers to LLaVA-Next-Interleaves. ‘DC’ refers to DocCompressor. ‘R.Acc’ refers to Relaxed Accuracy.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">DocVQA</th>
    <th colspan="2">ChartQA</th>
    <th colspan="2">MP-DocVQA</th>
    <th colspan="2">DUDE</th>
    <th colspan="2">NewsVideoQA</th>
  </tr>
  <tr>
    <th>Token^V</th>
    <th>ANLS↑</th>
    <th>Token^V</th>
    <th>R.Acc↑</th>
    <th>Token^V</th>
    <th>ANLS↑</th>
    <th>Token^V</th>
    <th>ANLS↑</th>
    <th>Token^V</th>
    <th>ANLS↑</th>
  </tr>
  <tr>
    <td>LLaVA-NI</td>
    <td>~3,061</td>
    <td><b>76.0</b></td>
    <td>~1,677</td>
    <td>69.5</td>
    <td>729</td>
    <td>50.0</td>
    <td>729</td>
    <td>39.0</td>
    <td>729</td>
    <td>62.4</td>
  </tr>
  <tr>
    <td>LLaVA-NI w/ DC</td>
    <td>729</td>
    <td>73.8</td>
    <td>729</td>
    <td><b>73.2</b></td>
    <td>729</td>
    <td><b>54.2</b></td>
    <td>729</td>
    <td><b>41.7</b></td>
    <td>729</td>
    <td><b>64.7</b></td>
  </tr>
</table>

tokens after compressing.

DocCompressor with different models. Our proposed DocCompressor is theoretically compatible with most MLLMs that have a vision-to-text module. To verify this, we insert DocCompressor into the LLaVA-Next-Interleave between its vision-to-text MLP and LLM. We finetune both the original model and the model with DocCompressor with the same data of DocOwl2 and evaluate them across both single-page and multi-page document understanding benchmarks. As shown in Table 5, LLaVA-Next-Interleave (w/ doccompressor) achieves comparable performance with LLaVA-Next-Interleave with fewer visual tokens. It validates that our compression module can be applied with a different backbone model.

Three-stage Training. DocOwl2 is trained with three stages: Single-image Pretraining, Multi-image Continue-pretraining, and Multi-task Fine-tuning. Table 6 shows the influence of each stage for OCR-free single-page and multi-page document understanding. With the Single-image Pretraining and Single-image finetuning (r1), the model achieves promising performance on single-page benchmark DocVQA and documents from MP-DocVQA with only 1 page. Although only trained with 1 image as the input, the model can also achieve around 50% accuracy when fed into 2-10 page images. However, the model struggles to understand documents with more than 10 pages, which greatly exceeds the number of input images during training and brings great difficulty in correlating images and finding answers. Performing Multi-image Fintuing could greatly improve the model’s ability to understand multiple images (r2 vs r1). Furthermore, adding the Multi-image Continue-pretraining could also improve the question-answering performance on downstream datasets, especially for documents with more than 10 pages (r3 vs r2). This demonstrates that parsing texts of the specified page or judging which pages contain specified texts among multi-page documents is a basic ability for multi-page document understanding. Finally, by ensembling both single-
Table 6: Ablation study about the training stages of DocOwl2. ‘Page Num’ and ‘Evidence Page’ refer to the number of input page images and the page ordinal number with the ground-truth answer.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="4">Pretraining</th>
    <th colspan="4">SFT</th>
    <th colspan="5">MP-DocVQA</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>Single Image</th>
    <th>Multi Image</th>
    <th>Single Image</th>
    <th>Multi Image</th>
    <th>DocVQA</th>
    <th>Page Num</th>
    <th>1</th>
    <th>2-10</th>
    <th>>10</th>
    <th>Evidence Page</th>
    <th>1</th>
    <th>2-10</th>
    <th>>10</th>
  </tr>
  <tr>
    <td>r1</td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>78.7</td>
    <td>81.3</td>
    <td>55.0</td>
    <td>5.8</td>
    <td>67.7</td>
    <td>45.9</td>
    <td>6.2</td>
    <td>54.2</td>
  </tr>
  <tr>
    <td>r2</td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>75.2</td>
    <td>78.7</td>
    <td>65.2</td>
    <td>34.6</td>
    <td>74.3</td>
    <td>54.9</td>
    <td>40.9</td>
    <td>63.8</td>
  </tr>
  <tr>
    <td>r3</td>
    <td>✓</td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td>74.2</td>
    <td>78.9</td>
    <td>65.7</td>
    <td>37.9</td>
    <td>74.2</td>
    <td>56.8</td>
    <td>43.4</td>
    <td>64.7</td>
  </tr>
  <tr>
    <td>r4</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td></td>
    <td><b>80.7</b></td>
    <td><b>83.3</b></td>
    <td><b>70.2</b></td>
    <td><b>42.5</b></td>
    <td><b>78.6</b></td>
    <td><b>60.9</b></td>
    <td><b>53.6</b></td>
    <td><b>69.4</b></td>
  </tr>
</table>

Table 7: VRAM usage comparison between DocCompressor and MLP (input size: 595 × 842, unit: MB).

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Module (i.)</th>
    <th colspan="2">LLM VRAM(i.)</th>
    <th colspan="2">Total VRAM (i.)</th>
  </tr>
  <tr>
    <th>Param.</th>
    <th>Act.</th>
    <th>Param.</th>
    <th>Act.</th>
    <th>Param. + Act.</th>
  </tr>
  <tr>
    <td>DocOwl2 w/ MLP</td>
    <td>96.02</td>
    <td>89.55</td>
    <td>16179.72</td>
    <td>45043.14</td>
    <td>61408.43</td>
  </tr>
  <tr>
    <td>DocOwl2 w/ DC</td>
    <td>804.36</td>
    <td>191.89</td>
    <td>16179.72</td>
    <td>2840.34</td>
    <td>20016.31</td>
  </tr>
</table>

image and multi-image instruction tuning sets (r4), DocOwl2 achieves the best performance on both single-page and multi-page document benchmarks, showing the cross-improvement between single-image and multi-image comprehension.

Qualitative results of multi-page text parsing, text lookup, question answering with detailed explanations can be found in Appendix B.3.

4.3 Analyses

Text Capacity of the Visual Embedding. To analyze the text capacity of the visual embedding, we further synthesize several A4-sized document images with different font sizes and numbers of characters to examine the parsing performance of DocOwl2 with 324 visual tokens. Concretely, we create an A4-sized document page with the resolution of 595 × 842 through the PyMuPDF and fill it with font sizes from 10 to 20 of English texts collected from a Wikipedia to synthesize multiple document pages. The number of characters ranges from 1,540 to 6,104. We let DocOwl2 parse the word inside these images and evaluate the result by ANLS score. Fig. 4 shows that DocOwl2 could almost perfectly parse a document with a document with less than 5,000 characters. We show a declined ANLS score when the character number exceeds 5,000, but it still maintains an ANLS score of 92.56% given a font size of 10, which contains 6,104 characters or 1,502 tokens inside an A4-sized page. This result demonstrated that our model has strong text capacity with only 324 visual tokens.

VRAM Consumption of DocCompressor. We compare the VRAM consumption of DocCompressor and the two-layer MLP projector (Liu et al., 2023a). We insert DocCompressor and MLP into DocOwl2 and measure the VRAM consumption in training mode when processing a fixed input resolution of 595 × 842. We report the VRAM consumption of both the activation and parameter produced by the module and LLM, and ignore the VRAM consumption of ViT since it is not related to the module. It can be seen that the DocCompressor requires more VRAM in terms of both parameter and activation. However, since it compresses the input length to LLM, it can largely reduce the activation costs of LLM, and finally requires much less VRAM for training with the same resolution.

![Parsing performance on A4-sized document image.](page_1012_1012_420_312.png)

Figure 4: Parsing performance on A4-sized document image.

5 Conclusion

We propose DocOwl2, a Multimodal LLM for efficient OCR-free Multi-page Document Understanding. The novel architecture High-resolution DocCompressor compresses each high-resolution document image into 324 tokens through cross-attention with the global visual feature as guidance, and reorganized features of cropped images as keys and values. A carefully designed three-stage training framework empowers the model with multi-page understanding ability and maintains single-page performance after compressing visual tokens. With fewer visual tokens, DocOwl2 outperforms existing compressing methods on single-page document understanding benchmarks, and achieves OCR-free state-of-the-art performance on two multi-page document understanding benchmarks and 1 text-rich video understanding benchmark.
Limitation

In this work, we propose a compressing architecture High-resolution DocCompressor for reducing visual tokens of high-resolution document images. Due to the compressor being placed between the vision-to-text module and the LLM, extra training for compressing visual tokens and re-aligning with LLM is indispensable. A more efficient method of compressing visual tokens and reduce training costs for re-aligning with LLMs can better leverage existing MLLMs, which is left as future work.

Ethics Statement

Initialized from a general Multimodal Large Language Model trained with massive web data, DocOwl2 may also suffer from issues of LLMs such as toxic language and bias (Bender et al., 2021). However, the three-stage training in this work focuses on parsing texts or questioning answering for publicly available document images. This introduces few biases relevant to ethical issues.

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Rimg, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022. Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204.14198.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966.

Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623.

Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. 2023. Honeybee: Locality-enhanced projector for multimodal LLM. CoRR, abs/2312.06742.

Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. Tabfact : A large-scale dataset for table-based fact verification. In International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia.

Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021. Websrc: A dataset for web-based structural reading comprehension. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4173–4185.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. 2024. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. CoRR, abs/2404.16821.

Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. 2024. xrag: Extreme context compression for retrieval-augmented generation with one token. CoRR, abs/2405.13792.

Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023. Adapting language models to compress contexts. In EMNLP, pages 3829–3846. Association for Computational Linguistics.

Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. 2024a. Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k HD. CoRR, abs/2404.06512.

Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. 2024b. Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k HD. CoRR, abs/2404.06512.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net.
Hao Feng, Qi Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang. 2023. Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding. CoRR, abs/2311.11810.

Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2024. In-context autoencoder for context compression in a large language model. In ICLR. OpenReview.net.

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. 2023. Cogagent: A visual language model for GUI agents. CoRR, abs/2312.08914.

Anwen Hu, Shizhe Chen, and Qin Jin. 2021. Question-controlled text-aware image captioning. In ACM Multimedia, pages 3097–3105. ACM.

Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. 2024. mplug-docow1 l5: Unified structure learning for ocr-free document understanding. CoRR, abs/2403.12895.

Soumya Jahagirdar, Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. 2023. Watching the news: Towards videoqa models that can read. In WACV, pages 4430–4439. IEEE.

Kushal Kafle, Brian L. Price, Scott Cohen, and Christopher Kanan. 2018. DVQA: understanding data visualizations via question answering. In CVPR, pages 5648–5656. Computer Vision Foundation / IEEE Computer Society.

Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2018. Figureqa: An annotated figure dataset for visual reasoning. In ICLR (Workshop). OpenReview.net.

Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdo Yun, Dongyoon Han, and Seunghyun Park. 2022. Ocr-free document understanding transformer. In ECCV (28), volume 13688 of Lecture Notes in Computer Science, pages 498–517. Springer.

Geewook Kim and Minjoon Seo. 2024. On efficient language and vision assistants for visually-situated natural language understanding: What matters in reading and reasoning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16978–17000, Miami, Florida, USA. Association for Computational Linguistics.

Jordy Van Landeghem, Rafal Powalski, Rubén Tito, Dawid Jurkiewicz, Matthew B. Blaschko, Lukasz Borchmann, Mickael Coustaty, Sien Moens, Michal Pietruszka, Bertrand Anckaert, Tomasz Stanislawek, Pawel Józiak, and Ernest Valveny. 2023. Document understanding dataset and evaluation (DUDE). In ICCV, pages 19471–19483. IEEE.

Hugo Laurençon, Leo Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? In Advances in Neural Information Processing Systems, volume 37, pages 87874–87907. Curran Associates, Inc.

Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisen schllos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 18893–18912. PMLR.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zwei Liu, and Chunyuan Li. 2025. LLava-onevision: Easy visual task transfer. Transactions on Machine Learning Research.

Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024a. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. CoRR, abs/2407.07895.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023a. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. CoRR, abs/2301.12597.

Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, and Lei Zhang. 2024b. Tokenpacker: Efficient visual projector for multimodal LLM. CoRR, abs/2407.02392.

Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. 2024c. Mini-gemini: Mining the potential of multi-modality vision language models. CoRR, abs/2403.18814.

Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. 2023b. Monkey: Image resolution and text label are important things for large multi-modal models. CoRR, abs/2311.06607.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning. CoRR, abs/2310.03744.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. CoRR, abs/2304.08485.

Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. 2024. Textmonkey: An ocr-free large multimodal model for understanding document. CoRR, abs/2403.04473.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACL (Findings), pages 2263–2279. Association for Computational Linguistics.

Minesh Mathew, Viraj Bagal, Ruben Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V. Jawahar. 2022. Infographicvqa. In WACV, pages 2582–2591. IEEE.

Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. 2021. Docvqa: A dataset for VQA on document images. In WACV, pages 2199–2208. IEEE.

Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. In WACV, pages 1516–1525. IEEE.

Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In ACL (1), pages 1470–1480. The Association for Computer Linguistics.

Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu. 2024. Eagle: Exploring the design space for multimodal llms with mixture of encoders. Preprint, arXiv:2408.15998.

Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. 2020. Textcaps: A dataset for image captioning with reading comprehension. In ECCV (2), volume 12347 of Lecture Notes in Computer Science, pages 742–758. Springer.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA models that can read. In CVPR, pages 8317–8326. Computer Vision Foundation / IEEE.

Tomasz Stanislawek, Filip Gralinski, Anna Wróblewska-Dawid Lipinski, Agnieszka Kaliska, Paulina Rosalska, Bartosz Topolski, and Przemyslaw Biecek. 2021. Kleister: Key information extraction datasets involving long documents with complex layouts. In ICDAR (1), volume 12821 of Lecture Notes in Computer Science, pages 564–579. Springer.

S Svetlichnaya. 2020. Deepform: Understand structured documents at scale.

Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. 2021. Visualmrc: Machine reading comprehension on document images. In AAAI, pages 13878–13888. AAAI Press.

Rubén Tito, Dimosthenis Karatzas, and Ernest Valveny. 2022. Hierarchical multimodal transformers for multi-page docvqa. CoRR, abs/2212.05935.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. 2023. Vary: Scaling up the vision vocabulary for large vision-language models. CoRR, abs/2312.06109.

Renqiu Xia, Song Mao, Xiangchao Yan, Hongbin Zhou, Bo Zhang, Haoyang Peng, Jiahao Pi, Daocheng Fu, Wenjie Wu, Hancheng Ye, Shiyang Feng, Bin Wang, Chao Xu, Conghui He, Pinlong Cai, Min Dou, Botian Shi, Sheng Zhou, Yongwei Wang, Bin Wang, Junchi Yan, Fei Wu, and Yu Qiao. 2024. Docgenome: An open large-scale scientific document benchmark for training and testing multi-modal large language models. CoRR, abs/2406.11633.

Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. 2023a. mplug-docowl: Modularized multimodal large language model for document understanding. CoRR, abs/2307.02499.

Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Lin, and Fei Huang. 2023b. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. In EMNLP (Findings), pages 2841–2858. Association for Computational Linguistics.

Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2024. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. Preprint, arXiv:2408.04840.

Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. 2023c. mplug-owl1: Modularization empowers large language models with multimodality. CoRR, abs/2304.14178.

Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023d. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. CoRR, abs/2311.04257.

Liang Zhang, Anwen Hu, Jing Zhang, Shuo Hu, and Qin Jin. 2023. Mpqa: multimodal question answering on product manuals. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13958–13966.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. 2024a. Long context transfer from language to vision. CoRR, abs/2406.16852.

Ruiyi Zhang, Yufan Zhou, Jian Chen, Jiuxiang Gu, Changyou Chen, and Tong Sun. 2024b. Llava-read: Enhancing reading ability of multimodal language models. Preprint, arXiv:2407.19185.

Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno-Yepes. 2020. Image-based table recognition: Data, model, and evaluation. In ECCV (21), volume 12366 of Lecture Notes in Computer Science, pages 564–580. Springer.

A Model Training

A.1 Training Data
We utilize DocStruct4M (Hu et al., 2024) as the training data of the first stage.

In the second stage, we construct training samples of Multi-page Text Parsing and Multi-page Text Lookup based on partial documents from two datasets of PixParse\(^{1,2}\).

As for the third Multi-task Tuning stage, we leverage DocDownstream-1.0 (Hu et al., 2024) and DocReason25K (Hu et al., 2024) as single-image datasets. DocDownstream-1.0 is an ensembled dataset comprising of DocVQA (Mathew et al., 2021), InfoVQA (Mathew et al., 2022), DeepForm (Svetlichnaya, 2020), KLC (Stanislawek et al., 2021), WTQ (Pasupat and Liang, 2015), TabFact (Chen et al., 2020), ChartQA (Masry et al., 2022), TextVQA (Singh et al., 2019), TextCaps (Sidorov et al., 2020) and VisualMRC (Tanaka et al., 2021). DocReason25K is a question-answering dataset with detailed explanations. As for multi-image understanding, we ensemble 2 document datasets, MP-DocVQA (Tito et al., 2022) and DUDE (Landeghem et al., 2023), and 1 news video dataset NewsVideoQA (Jahagirdar et al., 2023) as concise question-answering datasets. MP-DocVQA contains 46k question-answering pairs on 60k page images scanned from 6k industry documents with rich tables, diagrams, pictures, and both handwritten and printed texts. DUDE covers more domains of documents, including medical, legal, technical, financial, etc. It contains 41k question-answering pairs on 5k documents. NewsVideoQA collects news videos with rich visually-situated texts from diverse English news channels around the world, such as BBC, CNN, etc. It contains 8k question-answering pairs framed on 3k videos. Besides, to trigger the ability of detailed explanations with evidence pages, we built MP-DocReason51K based on DocReason25K. Concretely, for each single-image sample from DocReason25K, we construct two multi-image samples with noisy images randomly chosen from the same or different categories. After randomly inserting the evidence image into noisy images, we add an extra evidence description (e.g., ‘According to the 5th image,’) into the raw detailed explanation to get the target of multi-image samples. Most question-answering samples just focus on 1-2 pages of a document, to further strengthen the ability of a comprehensive understanding of a document, we leverage a small part of annotations from DocGenome (Xia et al., 2024) to construct text sequences in the JSON format, which represents the hierarchical structure of a scientific paper and partial detailed texts.

Table 8 shows the detailed statistics of training data at each stage.

A.2 Implementation Details
The maximum number of crops is set to 12. The resolution of each sub-image or the global image is 504x504. The High-resolution DocCompressor comprises of 2 layers of cross attention. Initialized from mPLUG-Owl2 (Ye et al., 2023d), the vision encoder (ViT/L-14 (Dosovitskiy et al., 2021)), H-Reducer and High-resolution DocCompressor are trained during the Sinlge-image Pretraining. Besides, the main parameters of the Large Language Model (Touvron et al., 2023) are frozen while a Modality Adaptive Module (MAM) (Ye et al., 2023d) used to distinguish visual and textual features in the LLM is tuned. The first stage takes 12k steps on 32 A100 GPUs for 84 hours with a batch size of 1,024 and a learning rate of 1e-4. During the Multi-image Continue-pretraining, the vision encoder is further frozen and the H-Reducer, High-resolution DocCompressor and MAM is tuned. The second stage takes 2.4k steps on 32 A100 GPUs in 130 hours with a batch size of 1,024 and the learning rate set as 2e-5. At the final Multi-task Finetuning stage, all parameters except the vision encoder are optimized. The batch size, training step, and learning rate at this stage are set as 256, 9k, 2e-5 respectively. This
Table 8: Detailed statistic of training datasets of DocOwl2.

<table>
  <tr>
    <th>Training Stage</th>
    <th>Input Image</th>
    <th>Dataset</th>
    <th>Num</th>
  </tr>
  <tr>
    <td>Single-image Pretraining</td>
    <td>Single</td>
    <td>DocStruct4M</td>
    <td>4,036,402</td>
  </tr>
  <tr>
    <td rowspan="2">Multi-image Continue Pretraining</td>
    <td>Single</td>
    <td>DocStruct4M</td>
    <td>501,781</td>
  </tr>
  <tr>
    <td>Multiple</td>
    <td>MP-DocStruct1M</td>
    <td>1,113,259</td>
  </tr>
  <tr>
    <td rowspan="2">Multi-task Finetuning</td>
    <td>Single</td>
    <td>DocVQA, InfoVQA, DeepForm, KLC, WTQ, TabFact, ChartQA, TextVQA, TextCaps, VisualMRC</td>
    <td>552,315</td>
  </tr>
  <tr>
    <td></td>
    <td>DocReason25K</td>
    <td>25,877</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td></td>
    <td>MP-DocVQA</td>
    <td>70,154</td>
  </tr>
  <tr>
    <td></td>
    <td>DUDE</td>
    <td>35,438</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>Multiple</td>
    <td>NewsVideoQA</td>
    <td>8,619</td>
  </tr>
  <tr>
    <td></td>
    <td>MP-DocReason51K</td>
    <td>51,754</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>DocGenome12K</td>
    <td>12,010</td>
  </tr>
</table>

![Two radar charts comparing performance and average number of visual tokens for various datasets](page_101_670_1242_312.png)

Figure 5: The comparison of our DocOwl2 with state-of-the-art Multimodal Large Language Models on (a) OCR-free performance and (b) the average number of visual tokens on 10 Visual Document Understanding benchmarks.

training stage takes 125 hours to converge with 32 A100 GPUs.

B Experiments

B.1 Single-image Document Understanding

We divide baselines into three groups: (a) models without Large Language Models as decoders (Kim et al., 2022; Lee et al., 2023), (b) Multimodal LLMs (Hong et al., 2023; Dong et al., 2024b; Chen et al., 2024; Li et al., 2024b; Hu et al., 2024; Feng et al., 2023; Li et al., 2023b) with an average number of visual tokens over 1k for a single document image and (c) Multimodal LLMs (Ye et al., 2023a,b; Liu et al., 2024; Li et al., 2024b; Bai et al., 2023) with an average number of visual tokens less than 1k. As shown in Table 9, although specifically fine-tuned on each downstream dataset, Donut (Kim et al., 2022) or PixsStruct (Lee et al., 2023) are not as good as Multimodal LLMs, showing the potential of MLLMs for generalized OCR-free document understanding. Among models with <1k visual tokens, DocOwl2 achieves state-of-the-art performance. Compared with MLLMs with >1k visual tokens, DocOwl2 achieves >80% performance on 7 benchmarks while with <20% visual tokens. Fig. 5 visualizes the comparison with SOTA in terms of question-answering performance and the number of visual tokens.

Table 10 further shows the performance and inference speed comparison on the 3 most frequently compared benchmarks, representing document, chart, and natural images.

B.2 DocCompressor versus Mini-Gemini

Though Mini-Gemini (Li et al., 2024c) also explore to mixing low and high-resolution features via cross-attention, there are two major differences between DocCompressor and Mini-Gemini. First, DocCompressor uses a single vision encoder combined with image cropping to encode high-resolution images, and Mini-Gemini relies on an additional high-resolution encoder. Second, our Doc-
Table 9: Comparison with OCR-free methods on single-image document understanding tasks. The ‘*’ refers to models without LLMs and separately fine-tuned on each downstream task. ‘Token^V’ means the average number of visual tokens of a single image. ‘Bold’ means SOTA performance within the group and ‘Underline’ means achieving 80% SOTA performance among all baselines.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Size</th>
    <th rowspan="2">Token^V</th>
    <th colspan="3">Doc VQA</th>
    <th colspan="2">Info VQA</th>
    <th colspan="2">Deep Form</th>
    <th>KLC</th>
    <th>WTQ</th>
    <th>Tab Fact</th>
    <th>Chart QA</th>
    <th>Text VQA</th>
    <th>Text Caps</th>
    <th>Visual MRC</th>
  </tr>
  <tr>
    <th>VQA</th>
    <th>VQA</th>
    <th>Form</th>
    <th>VQA</th>
    <th>Form</th>
    <th>VQA</th>
    <th>Form</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td>Donut*</td>
    <td>&lt;1B</td>
    <td>4,800</td>
    <td>67.5</td>
    <td>11.6</td>
    <td>61.6</td>
    <td>30.0</td>
    <td>18.8</td>
    <td>54.6</td>
    <td>41.8</td>
    <td>43.5</td>
    <td>74.4</td>
    <td>93.91</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Pix2Struct<sub>base</sub></td>
    <td>&lt;1B</td>
    <td>2,048</td>
    <td>72.1</td>
    <td>38.2</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>56.0</td>
    <td>-</td>
    <td>88.0</td>
    <td>-</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Pix2Struct<sub>large</sub></td>
    <td>1B</td>
    <td>2,048</td>
    <td>76.6</td>
    <td>40.0</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>58.6</td>
    <td>-</td>
    <td>95.5</td>
    <td>-</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="7">Token^V &lt; 1K</td>
    <td>CogAgent</td>
    <td>17B</td>
    <td>6,656</td>
    <td>81.6</td>
    <td>44.5</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>68.4</td>
    <td>76.1</td>
    <td>-</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>IXC 2.5</td>
    <td>7B</td>
    <td>~5,118</td>
    <td>90.9</td>
    <td>69.9</td>
    <td><b>71.2</b></td>
    <td>-</td>
    <td><b>53.6</b></td>
    <td><b>85.2</b></td>
    <td>82.2</td>
    <td>78.2</td>
    <td>-</td>
    <td><b>307.5</b></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>InternVL 2</td>
    <td>8B</td>
    <td>~3,133</td>
    <td><b>91.6</b></td>
    <td><b>74.8</b></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td><b>83.3</b></td>
    <td><b>77.4</b></td>
    <td>-</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TokenPacker</td>
    <td>13B</td>
    <td>~1,855</td>
    <td>70.0</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>DocOwl 1.5</td>
    <td>8B</td>
    <td>~1,698</td>
    <td>82.2</td>
    <td>50.7</td>
    <td>68.8</td>
    <td><b>38.7</b></td>
    <td>40.6</td>
    <td>80.2</td>
    <td>70.2</td>
    <td>68.6</td>
    <td><b>131.6</b></td>
    <td>246.4</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>DocPedia</td>
    <td>7B</td>
    <td>1,600</td>
    <td>47.1</td>
    <td>15.2</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>46.9</td>
    <td>60.2</td>
    <td>-</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Monkey</td>
    <td>9B</td>
    <td>1,280</td>
    <td>66.5</td>
    <td>36.1</td>
    <td>40.6</td>
    <td>32.8</td>
    <td>25.3</td>
    <td>-</td>
    <td>-</td>
    <td>64.3</td>
    <td>93.2</td>
    <td>-</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="7">Token^V &lt; 1K</td>
    <td>DocOwl</td>
    <td>7B</td>
    <td>~841</td>
    <td>62.2</td>
    <td>38.2</td>
    <td>42.6</td>
    <td>30.3</td>
    <td>26.9</td>
    <td>60.2</td>
    <td>57.4</td>
    <td>52.6</td>
    <td>111.9</td>
    <td>188.8</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>URReader</td>
    <td>7B</td>
    <td>~841</td>
    <td>65.4</td>
    <td>42.2</td>
    <td>49.5</td>
    <td>32.8</td>
    <td>29.4</td>
    <td>67.6</td>
    <td>59.3</td>
    <td>57.6</td>
    <td>118.4</td>
    <td><b>221.7</b></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TextMonkey</td>
    <td>9B</td>
    <td>768</td>
    <td>73.0</td>
    <td>28.6</td>
    <td>59.7</td>
    <td><b>37.8</b></td>
    <td>31.9</td>
    <td>-</td>
    <td>66.9</td>
    <td>65.9</td>
    <td>-</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TokenPacker</td>
    <td>13B</td>
    <td>~467</td>
    <td>58.0</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>QwenVL</td>
    <td>9B</td>
    <td>256</td>
    <td>65.1</td>
    <td>35.4</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>65.7</td>
    <td>63.8</td>
    <td>-</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Vary</td>
    <td>7B</td>
    <td>256</td>
    <td>76.3</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>66.1</td>
    <td>-</td>
    <td>-</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>DocOwl2</td>
    <td>8B</td>
    <td>324</td>
    <td><b>80.7</b></td>
    <td><b>46.4</b></td>
    <td><b>66.8</b></td>
    <td><b>37.5</b></td>
    <td><b>36.5</b></td>
    <td><b>78.2</b></td>
    <td><b>70.0</b></td>
    <td><b>66.7</b></td>
    <td><b>131.8</b></td>
    <td>217.4</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 10: Comparison with OCR-free Multimodal Large Language Models on single-image document understanding benchmarks. ‘FTL(s)’ refers to the First Token Latency (seconds)

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Size</th>
    <th colspan="3">DocVQA</th>
    <th colspan="3">ChartQA</th>
    <th colspan="3">TextVQA</th>
  </tr>
  <tr>
    <th>Token^V</th>
    <th>FTL(s)↓</th>
    <th>ANLS↑</th>
    <th>Token^V</th>
    <th>FTL(s)↓</th>
    <th>ANLS↑</th>
    <th>Token^V</th>
    <th>FTL(s)↓</th>
    <th>ANLS↑</th>
  </tr>
  <tr>
    <td>InternVL 2</td>
    <td>8B</td>
    <td>~3,198</td>
    <td>0.94</td>
    <td>91.6</td>
    <td>~1,827</td>
    <td>0.56</td>
    <td>83.3</td>
    <td>~2,864</td>
    <td>1.01</td>
    <td>77.4</td>
  </tr>
  <tr>
    <td>IXC 2.5</td>
    <td>7B</td>
    <td>~7,395</td>
    <td>3.73</td>
    <td>90.9</td>
    <td>~1,971</td>
    <td>1.05</td>
    <td>82.2</td>
    <td>~2,075</td>
    <td>1.11</td>
    <td>78.2</td>
  </tr>
  <tr>
    <td>DocOwl 1.5</td>
    <td>8B</td>
    <td>~1,806</td>
    <td>0.58</td>
    <td>82.2</td>
    <td>~1,713</td>
    <td>0.53</td>
    <td>70.2</td>
    <td>~1,664</td>
    <td>0.56</td>
    <td>68.6</td>
  </tr>
  <tr>
    <td>TextMonkey</td>
    <td>9B</td>
    <td>768</td>
    <td>0.58</td>
    <td>73.0</td>
    <td>768</td>
    <td>0.51</td>
    <td>66.9</td>
    <td>768</td>
    <td>0.50</td>
    <td>65.9</td>
  </tr>
  <tr>
    <td>DocOwl2</td>
    <td>8B</td>
    <td>324</td>
    <td><b>0.26</b></td>
    <td>80.7</td>
    <td>324</td>
    <td><b>0.21</b></td>
    <td>70.0</td>
    <td>324</td>
    <td><b>0.23</b></td>
    <td>66.7</td>
  </tr>
</table>

Table 11: Comparison between Mini-Gemini and DocCompressor over document understanding benchmarks.

<table>
  <tr>
    <th>Structure</th>
    <th>DocVQA</th>
    <th>WTQ</th>
    <th>ChartQA</th>
    <th>InfoVQA</th>
    <th>DeepForm</th>
    <th>KLC</th>
  </tr>
  <tr>
    <td>Mini-Gemini</td>
    <td>75.7</td>
    <td>33.3</td>
    <td>68.7</td>
    <td>41.6</td>
    <td>58.4</td>
    <td>37.0</td>
  </tr>
  <tr>
    <td>DocCompressor</td>
    <td>76.1</td>
    <td>35.1</td>
    <td>69.2</td>
    <td>41.7</td>
    <td>59.5</td>
    <td>37.5</td>
  </tr>
</table>

Compressor merges high-resolution information after the vision-to-text module, and Mini-Gemini does it before. To show the advantages of our framework, we train both structures with the same training recipe to make a fair comparison. As shown in Table 11, our DocCompressor outperforms Mini-Gemini on all document understanding benchmarks, which verifies the effectiveness of the design of our DocCompressor.

B.3 Qualitative Results

As shown in Fig. 6, after the Multi-image Continue Pretraining stage, DocOwl2 is able to locate the corresponding image of the given texts accurately. Besides, although representing each high-resolution image with just 324 tokens, DocOwl2 is still capable of parsing detailed texts of specified two images, validating the promising OCR-free multi-page document understanding performance of DocOwl2 . It also demonstrates our proposal that 324 tokens are enough to encode detailed text information in common A4-sized document pages and the effectiveness of our High-resolution DocCompressor.

After the Multi-task Finetuning, given multiple images and a question, DocOwl2 can give a simple answer first and then provide a detailed explanation with the evidence, as shown in Fig. 7. DocOwl2 can comprehend not only page images rendered from PDF files (Fig. 7(c)) but also scan images of a document (Fig. 7(a-b)). When a question is unanswerable, DocOwl2 can also tell and give corresponding reasons (Fig. 7(c)).

Besides multi-page documents, DocOwl2 is also capable of understanding text-rich videos. As shown in Fig. 8, among similar frames within a video, DocOwl2 can distinguish fine-grained textual differences, locate relevant frames, and give accurate answers.
Identify the image that features text <doc> (vii) the provisions of this Agreement governing payments with respect to Certificates and related notices, including Sections 2.4, 3.1 and 3.2
...
SECTION 10.1. Termination of Intercreditor Agreement. Following payment of Final Distributions with respect to each Class of Certificates and the payment in full of all Liquidity Obligations to the Liquidity Providers and provided that there shall then be no other amounts due to the Certificateholders, the Trustees, the Liquidity Providers and the 53 </doc> ?

ARTICLE X
MISCELLANEOUS
SECTION 10.1. Termination of Intercreditor Agreement. Following payment of Final Distributions with respect to each Class of Certificates and the payment in full of all Liquidity Obligations to the Liquidity Providers and provided that there shall then be no other amounts due to the Certificateholders, the Trustees, the Liquidity Providers and the

DocOwl 2:
<img 7>

(a)

Extract words from the 14th picture and 17th picture.

Georgia Garden Railway Society Sep 2020
The sound modules I purchased operate with less than 5 volts. Some use three 'button cells' for a total of about 4.5 volts. Buttons do not last very long. AAA cells also do not have to replace periodically. The modules also operate on a single Li-Ion rechargeable cell (voltage nominally 3.7 volts). These modules use very few milliamperes so I used a 240 milliAmpHour (mAh) Li-Ion battery purchased on AliExpress. It rarely require recharge. It is shown in Figure 2.

DocOwl 2:
<doc 14> Georgia Garden Railway Society Sep 2020
Atlanta Senior Life: Big Fun with Little Trains
The Atlanta Senior Life newspaper carried an article in its July 2020 Vol. 5 No. 7 edition featured a couple of couples from the GGRS ... .
Later in the article, another GGRS pair, Russ and Leslie Ann Bundy were also interviewed. Maybe we can pick up a couple of new members from this coverage. The Atlanta Senior Life is available online at at atlantaseniorlife.com or on facebook.com/atlantaseniorlife .
2020 Piedmont Pilgrimage -- An Online Tour of the Atlanta Area's Great Model Railroads
By Russ Bundy
The Piedmont Pilgrimage is sponsored each year by the Piedmont Division ...
The 18th annual pilgrimage, 2020 is proving to be quite a challenging year.
Social distancing to minimize chances of contracting the COVID - 19 virus has affected a lot of activities, including the Piedmont Pilgrimage.
Continued page 10 </doc 14>
<doc 17> Page 17 Georgia Garden Railway Society Sep 2020
The sound module is operated with less than 5 volts. Some use three 'button cells' for a total of about 4.5 volts. Buttons do not last very long. AAA cells also do not have to replace periodically.
The modules also operate on a single Li-Ion rechargeable cell (voltage nominal current modules). These modules use very few milliamperes so I used a 240 AmpHour (mAh) Li-Ion battery purchased on AliExpress. It rarely require recharge. It is shown in Figure 2.
With some soldering I replaced the pushbutton switches provided with the sound Magnet Trigger modules with magnetic switches. The magnetic switch was hot-glued to existing structure on the bottom of the flat car as shown in Figure 4. </doc 17>

(b)

Figure 6: Qualitative results of the Multi-page Text Lookup (a) and Multi-page Text Parsing (b) given by DocOwl2 after the Multi-image Continue Pretraining.
What is the "required filing frequency"?
Answer the question with detailed explanation?

DocOwl 2:
Required filing frequency is once per month. In accordance with the 7th image, the text states that the filing requirement for the NYS Income Tax withholdings is once per month. This means that taxpayers must file their tax returns once a month to report their income and pay any applicable taxes.

Mention any 2 Expected principle source of payments?
Answer the question with "detailed explanation."

DocOwl 2:
Self pay and Workmen's compensation.
As shown in the 18th image, the text states that the expected principal source of payment for the hospital ID numbered 1014 is self pay and workmen's compensation. This means that the patient will be responsible for paying for their own medical expenses and will also receive payment from Workers' Compensation for any work-related injuries they may have sustained while working in the hospital.

Is there more than 5 tables in chapter 18 of page 2?
Answer the question with detailed explanation

DocOwl 2:
There is no information about chapter 18 of page 2 in the given texts. According to the second image, the texts only provide information about the U.S. Senate Committee on Banking, the Ranking Members, and the Senators who cosponsored the Secure and Fair Enforcement (SAFE) Banking Act in the 117th Congress. There is no mention of chapter 18 of page 2 or any information related to it. Therefore, the answer to the question is that there is no information provided about the number of tables in chapter 18 of page 2 in the given texts.

Figure 7: Qualitative results of the Multi-page Question Answering with detailed explanation.
Which year birthday is NFT celebrating? Answer the question with detailed explanation.

DocOwl 2:
Stan Lee's 99th birthday.
According to the 16th image, the text in the image mentions "NFT to celebrate Stan Lee's 99th birthday", indicating that the NFT collection is being celebrated on Stan Lee's birthday. Therefore, Stan Lee's 99th birthday is the year being referred to in this context.

(a)

Who are being responded in supreme court? Answer the question with detailed explanation.

DocOwl 2:
States and Centre.
In accordance with the image 3, the text mentions that the air crisis is a centre-stage issue and states have a responsibility to respond to it. Therefore, it can be inferred that the Supreme Court is responding to the crisis by directing states and the Centre to take appropriate actions.

(b)

Are there physical coins or bills in cryptocurrency? Answer the question with detailed explanation.

DocOwl 2:
No physical coins or bills in cryptocurrency.
As shown in the image 4, the text clearly states that there are no physical coins or bills in cryptocurrency. This means that the digital currency only exists in the form of electronic coins or tokens, and there is no physical currency associated with it.

(c)

Figure 8: Qualitative results of the Text-rich Video Understanding.

Reviewer paper summaries: This paper presents DocOwl2, a High-resolution DocCompressor module designed to enable efficient OCR-free document understanding within Multimodal Large Language Models (MLLMs). DocOwl2 follows a three-stage training approach (Single-image Pretraining, Multi-image Continue-pretraining, Multi-task Finetuning) and demonstrates state-of-the-art performance on multi-page document understanding tasks. The proposed DocCompressor significantly reduces visual tokens without compromising accuracy, improving computational efficiency and latency. The code, data, and models will be publicly available.

This paper proposes a DocComposer module that compresses high-resolution document image encoding from a visual encoder into a fixed-length sequence of language embeddings.
The authors design a multimodal large language model DocOwl2 that includes this module, and apply pre-training on single and multiple images and fine-tuning on multitasking.
The final model achieves competitive scores on benchmarks on document images while keeping the number of input tokens small.
This paper also provides a thorough validation of the architecture choices.

This paper addresses the inefficiencies of Multimodal Large Language Models (MLLMs) in document understanding. To solve this, the authors propose a High-resolution DocCompressor module, which compresses high-resolution document images into just 324 tokens, using global visual features. They introduce DocOwl2, a model trained in three stages—Single-image Pretraining, Multi-image Continue-pretraining, and Multi-task Finetuning—to enhance multi-page document comprehension. DocOwl2 achieves state-of-the-art performance in multi-page document benchmarks, reducing first-token latency by over 50%, while maintaining comparable single-page performance with 80% fewer visual tokens.
Summary of strengths: - The paper is clearly structured, and easy to follow.
- Comprehensive experiments are conducted.
- Authors have effectively addressed most of prior concerns, particularly with clearer contextualization of recent related work, clarification of model resource requirements, and improved categorization of model comparisons.
- The authors state that they will make their codes, models, and data publicly available.

- The proposed method reduces visual tokens in VLMs for document understanding and would be a candidate for balancing VLM performance and computational cost, especially in practical domains.
- Publication of relevant resources would be beneficial to the community. DocOwl2 model could be useful in related areas that require fast inference, such as vision-based GUI agents.
- The related work section summarizes recent progresses in VLMs for documents well.
- Thorough ablation studies validate the design choices, including the number of layers, the place of the compressor, the cross-attention region, etc. This analysis could be helpful for a wide range of readers who study the design of vision and language models.

1. It proposes a High-resolution DocCompressor module, which compresses high-resolution document images into just 324 tokens. 
2. It proposes three-stages-training, including Single-image Pretraining, Multi-image Continue-pretraining, and Multi-task Finetuning.
3. It achieves state-of-the-art performance in multi-page document benchmarks, reducing first-token latency by over 50%.
4. The codes, models, and data will be publicly available.
Summary of weaknesses: The authors have addressed the concerns raised in the previous round of review.

Points I added this time:

- Strictness of verification. L.468 "validating our hypothesis that compressing features after the vision-to-text module is like summarizing textual features and can maintain more textual semantics ..." : While Table 4 shows better performance (r3 vs r4), that might not validate the hypothesis directly because even if the dimensions are identical, they do not necessarily have the same semantic space. Strict validation would requires additional analysis on the similarity between the output features and the text embeddings.

Points from the previous review:

- ~I believe there is a limit to how much text can be packed into a fixed-length embedding sequence, but this paper lacks discussion of it.~ -> I am convinced by discussion in B4.
- ~I am concerned about the extent to which this method sacrifices the ability to recognize common images such as photographs. The low InfoVQA score in Table 1 may suggest this limitation.~ -> I am convinced by the additional result in Table 9.
- It has not been discussed whether this method is as effective in languages that use more complex characters, such as Chinese characters, as it is in languages that use the alphabet. -> No revisions made on this point. Since differences in performance by character type could be important in document image understanding, I retain this point.
- I understood that DocComposer performs cross-attention between hw/4 tokens and RChw/4 tokens. I am concerned that this process requires large VRAM capacity. -> It would be better to compare memory consumption although the impact could be limited compred to entire LLMs.

No significant drawbacks were observed.
Comments, suggestions and typos: It would be helpful if the authors could explicitly include anonymized information in the submission about the planned public release (e.g., brief descriptions of the datasets or code structure), as reliance solely on stated future plans provides reviewers limited concrete evidence for evaluation.

**Comments after Author Discussion**

I thank the authors for their response. I have carefully considered the comments and will keep my positive score as is.

- In text parsing tasks like Figure 1 (c), how does the model parse images? Is it just a simple summary such as "overall model structure", which can be extracted from captions, or a detailed description based on the analysis of an image?

- In figure1(b), why the model answered the compony name according to image 10 although the company name appears many times in the document? Is there any bias such as the model tends to use a page for a specific position?

- B.3 (comparison with mini-gemini): Does this comparison use the same components for mini-gemini and DocCompressor except for a compressor? A detailed description of the setup might be better.

- L.373: "we also randomly chose partial training samples": Random selection percentage would improve reproducibility.

- L.101: typo: afther -> after


**Comments after Author Discussion**

The author provides more insight about my question in their response.
It did not completely answer my question, but it did lead to a better understanding of their model.
So I decided to raise my Excitement score.

No significant drawbacks were observed.
Overall assessment scores: 4

4

4
Metareview score: 4
Metareview: This paper introduces DocOwl2, an improved Multimodal Large Language Model (MLLM) for efficient and accurate OCR-free document understanding. DocOwl2 addresses the computational bottleneck of processing high-resolution document images by incorporating a novel High-resolution DocCompressor. This module compresses image representations into a fixed-length sequence of embeddings, significantly reducing the number of visual tokens required for processing. Trained in three stages (single-image pretraining, multi-image continue-pretraining, and multi-task finetuning), DocOwl2 achieves state-of-the-art performance on multi-page document benchmarks while reducing latency and memory footprint. The authors commit to releasing code, models, and data. This work fits into sessions on document understanding, multimodal learning, efficient deep learning, and vision-language models.
Summary of reasons to publish: - Novel Architecture for Efficient Document Understanding: The proposed DocCompressor module offers a significant advancement in handling high-resolution documents within MLLMs. By drastically reducing the number of visual tokens, it addresses a key computational bottleneck, enabling faster processing and lower memory usage without sacrificing accuracy.
State-of-the-art Performance: DocOwl2 demonstrates superior performance on multiple multi-page document benchmarks, showcasing the effectiveness of the proposed approach. The reduction in latency and computational resources makes it particularly relevant for real-world applications.
- Thorough Ablation Studies: The paper includes detailed ablation studies that validate the design choices of the DocCompressor and other architectural components. These analyses provide valuable insights into the model's behavior and contribute to a deeper understanding of efficient multimodal learning.
- Reproducibility and Community Benefit: The authors' commitment to releasing code, models, and data fosters reproducibility and allows the research community to build upon their work, promoting further advancements in document understanding.
- Addresses Practical Concerns: The work directly tackles the practical challenges of applying MLLMs to real-world document processing scenarios, where efficiency and scalability are crucial.