Venue: NAACL 2024

# Visually Guided Generative Text-Layout Pre-training

for Document Intelligence

 Zhiming Mao\({}^{1,2}\), Haoli Bai\({}^{3*}\), Lu Hou\({}^{3}\),

**Jiansheng Wei\({}^{3}\), Xin Jiang\({}^{3}\), Qun Liu\({}^{3}\), Kam-Fai Wong\({}^{1,2}\)**

###### Abstract

Prior study shows that pre-training techniques can boost the performance of visual document understanding (VDU), which typically requires models to gain abilities to perceive and reason both document texts and layouts (e.g., locations of texts and table-cells). To this end, we propose visually guided generative text-layout pre-training, named ViTLP. Given a document image, the model optimizes hierarchical language and layout modeling objectives to generate the interleaved text and layout sequence. In addition, to address the limitation of processing long documents by Transformers, we introduce a straightforward yet effective multi-segment generative pre-training scheme, facilitating ViTLP to process word-intensive documents of any length. ViTLP can function as a native OCR model to localize and recognize texts of document images. Besides, ViTLP can be effectively applied to various downstream VDU tasks. Extensive experiments show that ViTLP achieves competitive performance over existing baselines on benchmark VDU tasks, including information extraction, document classification, and document question answering1.

Footnote 1: “Haoï is the corresponding author. Code and checkpoint will be at [https://github.com/Vesan-silverbulletViTLP](https://github.com/Vesan-silverbulletViTLP).

+The Chinese University of Hong Kong, Hong Kong, China

_[FOOTNOTE:1]_Correspondence of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4713–4730_

June 16-21, 2024 0204 Association for Computational Linguistics

## 1 Introduction

Processing and reasoning document images with dense texts (e.g., scanned PDF files, digital forms, and spreadsheets) is a persistent yet challenging task for the research community and industry Katti et al. (2018); Majumder et al. (2020); Li et al. (2021). Advances in multimodal pre-training substantially improve the performance of visual document understanding (VDU) Xu et al. (2020, 2021); Gu et al. (2021); Appalaraju et al. (2021); Wang et al. (2022). These pre-training methods typically take multimodal inputs of given document images including i) visual features, ii) pre-processed OCR texts, and iii) spatial layouts of document elements (e.g., \(2\)D coordinates of texts and table-cells). Among these inputs, spatial layout information plays an essential role in connecting visual and textual features, as well as developing thorough reasoning of document structures Chen et al. (2021); Lee et al. (2022).

Though effective, the performance of most existing VDU approaches relies heavily on the OCR pipelines, because the pre-processed OCR texts and corresponding \(2\)D coordinates are used as intermediate inputs to pre-trained VDU models. The external OCR pipelines may produce incorrect or incomplete recognition results, which cannot be jointly optimized by the gradient back from VDU models. Another research line Kim et al. (2022); Lee et al. (2023) explores pre-training VDU models solely based on image inputs. Despite no OCR errors introduced, these methods focus on understanding texts from raw document images but neglect layout information modeling. Since the spatial information contained in layout locations is not exploited, it may hinder the models from understanding complex document structures, especially for documents containing nested paragraphs, forms, and tables.

In this work, we propose **Vis**ually guided generative **T**ext-**L**ayout **P**re-training (ViTLP) to jointly model text and layout information from document images. As shown in Figure 1, ViTLP can localize,

Figure 1: An overview workflow of the proposed ViTLP. Given a document image as input, ViTLP can generate sequences of text and layout (i.e., word bounding boxes) for various VDU tasks with task-specific prefixes.



recognize, and understand visual document texts given the input document image and task prefixes. To achieve this goal, ViTLP is pre-trained to generate _unified text-layout sequences_ from document images. Since natively generating text and layout tokens in a flattened sequence is _token-inefficient_ (see Sec. 2.1), we introduce hierarchical generation modules to achieve both effective and efficient text-layout sequence generation. To the best of our knowledge, ViTLP is the first attempt to learn OCR (i.e., text localization and recognition) and VDU (i.e., document understanding) abilities in a unified generative text-layout pre-training framework.

Besides, ViTLP is designed to handle long documents with intensive texts. Long document processing is ubiquitous in real-world scenarios. However, existing pre-trained models are constrained to certain token limits of input sequences. For instance, LayoutLMv2 Xu et al. (2021) accepts the maximum inputs of \(512\) word tokens using a BERT-structure encoder. In both pre-training and fine-tuning, the exceeded text tokens are truncated, leading to incomplete document information modeling. To tackle this issue, we introduce a **multi-segment pre-training scheme** which divides the target text-layout sequence into consecutive segments to perform generative pre-training. Given that the full document information is already encoded in visual representations, ViTLP takes the suffix tokens from previous segments as prefix prompts to generate the next-segment tokens. This multi-segment pre-training scheme further enables ViTLP to process documents of arbitrary length in fine-tuning. Notably, our multi-segment generation scheme retains the intact transformer architecture. Thus, it is more feasible than other long-document modeling workarounds, e.g., _sparse attention_Beltagy et al. (2020) and _memory modules_Bulatov et al. (2022), which need to modify the Transformer architecture and may affect the capacity of pre-trained models.

We evaluate ViTLP on a variety of OCR and VDU tasks. Experiment results demonstrate that ViTLP can achieve superior overall performance on both OCR and VDU tasks. For instance, ViTLP achieves the \(95.59\%~{}\mathrm{F1}\) score on CORD information extraction and \(95.36\%\) accuracy on RVL-CDIP document classification, both of which outperform most previous approaches. Notably, ViTLP can intrinsically generate \(2\)D layout locations for visual grounding, which helps in certain generative VDU tasks (e.g., visual document question answering) to be more interpretable and reliable to humans.

## 2 Approach

### Problem Formulation

We study multimodal pre-training for visual document modeling. As widely studied Xu et al. (2020, 2021); Appalaraju et al. (2021); Li et al. (2021b); Powalski et al. (2021); Wang et al. (2022a); Huang et al. (2022); Wang et al. (2022b), document images \(\mathbf{V}\), texts \(\mathbf{T}\), and layouts \(\mathbf{L}\) are three fundamental modalities for visual document modeling.

Unified Text-Layout Generation.We cast the pre-training objective on visual documents as text-layout sequence (i.e., \(\{\mathbf{T};\mathbf{L}\}\)) generation conditioned on document images \(\mathbf{V}\). The document texts \(\mathbf{T}\) are represented as word-token sequences. The layouts \(\mathbf{L}\), following prior studies Xu et al. (2020, 2021), can be represented by _location bounding boxes_ of words. Instead of generating two separate sequences of \(\mathbf{T}\) and \(\mathbf{L}\), ViTLP generates the texts with corresponding layout locations in a sequence of interleaved text-layout tokens, which facilitates compact multimodal interaction between texts and layouts. For the \(i\)-th word of a document, its text-layout tokens \(\{\mathbf{T};\mathbf{L}\}_{i}\) are represented as

\[\{\mathbf{T};\mathbf{L}\}_{i}=\big{\{}\{\boldsymbol{w}\}_{i},\{z_{x_{1}},z_{ y_{1}},z_{x_{2}},z_{y_{2}}\}_{i}\big{\}}, \tag{1}\]

where \(\{\boldsymbol{w}\}_{i}\) denotes the BPE tokens Radford et al. (2019) of the \(i\)-th word, \(\{z_{x_{1}},z_{y_{1}},z_{x_{2}},z_{y_{2}}\}_{i}\in\mathbb{Z}^{4}_{+}\) are the corresponding left-top and right-bottom bounding box coordinates. Given a document with \(N\) words, the objective is to maximize the likelihood function \(\log p(\mathbf{T};\mathbf{L}|\mathbf{V})\) which can be decomposed as autoregressive text and layout modeling:

\[\log p(\mathbf{T};\mathbf{L}|\mathbf{V}) =\sum_{i=1}^{N}\big{(}\underbrace{\log p(\mathbf{T}_{i}|\mathbf{ T}_{<i},\mathbf{L}_{<i},\mathbf{V})}_{\text{Text-modeling}}\] \[+\underbrace{\log p(\mathbf{L}_{i}|\mathbf{T}_{\leq i},\mathbf{ L}_{<i},\mathbf{V})}_{\text{Layout-modeling}}\big{)}. \tag{2}\]

Note that Eq. (2) shares similar ideas with Chen et al. (2022), where word and bounding box generation can be formulated as language modeling on a unified text-layout sequence. However, it is in fact nontrivial to generate sequences as in Eq. (1), because real-world documents commonly contain intensive texts, generating each word followed by four coordinate tokens in a long flattened sequence is especially **token-inefficient**. This would bring prohibitive computational and space overhead2 to the Transformer-based text-layout decoder.

 



### Model Architecture

The architecture of ViTLP is shown in Figure 2. ViTLP employs an encoder-decoder framework to encode document images \({\bf V}\) and generate target text-layout sequences \(\{{\bf T};{\bf L}\}\). Specifically, given an input document image \({\bf V}\), ViTLP employs a vision transformer (ViT) [15] to learn visual representations \({\bf H}^{V}\in\mathbb{R}^{|V|\times d}\), where \(|V|\) is the ViT patch number and \(d\) is the hidden size. The decoder receives the visual representations \({\bf H}^{V}\) and generates the unified text-layout sequence \(\{{\bf T};{\bf L}\}\). To address the _token-inefficiency_ issue discussed in Sec. 2.1, we design the _global-to-local_ text-layout generation process as follows.

#### 2.2.1 Global Text-Layout Modeling

Instead of directly generating the text-layout sequence as in Eq. (1), we first replace the bounding box coordinates \(\{z_{x_{1}},z_{y_{1}},z_{x_{2}},z_{y_{2}}\}\) with a generic layout location token \(\hat{w}=\texttt{[LOC]}\). This integrates the mixed text-layout sequence \(\{{\bf T};{\bf L}\}\) to unified language modeling. Given the original vocabulary \(\mathcal{V}\), the **global text-layout sequence**\(\hat{\mathbf{T}}\) derives from the augmented vocabulary \(\hat{\mathcal{V}}=\mathcal{V}\cup\texttt{[LOC]}\). The layout token embeddings \(\mathrm{E}_{\texttt{[LOC]}}\) are computed as

\[\mathrm{E}_{\texttt{[LOC]}}=\big{[}\mathrm{E}_{x}(z_{x_{1}}),\mathrm{E}_{y}( z_{y_{1}}),\mathrm{E}_{x}(z_{x_{2}}),\mathrm{E}_{y}(z_{yz_{2}})\big{]},\]

where \(\mathrm{E}_{x}(\cdot)\in\mathbb{R}^{d}\) and \(\mathrm{E}_{y}(\cdot)\in\mathbb{R}^{\frac{d}{4}}\) denote the x- and y-axis spatial embeddings. Besides, the word tokens are embedded by \(\mathrm{E}_{w}(\cdot)\in\mathbb{R}^{d}\). Given a document of \(N\) words and the corresponding bounding boxes, the text-layout input embeddings are represented as \({\bf H}^{TL}=\{\mathrm{E}_{w},\mathrm{E}_{\texttt{[LOC]}}\}\in\mathbb{R}^{| \hat{\mathbf{T}}|\times d}\).

The ViTLP text-layout decoder performs multimodal interaction among _visual_, _textual_, and _layout_ information via the Transformer cross-attention

For the \(i\)-th target token \(\hat{\mathbf{T}}_{i}\), the multimodal decoder output \({\bf H}_{i}^{VTL}\) is fed to a linear language modeling (LM) head with the softmax function to compute the conditional generative probability

\[p(\hat{\mathbf{T}}_{i}|\hat{\mathbf{T}}_{<i},\mathbf{V})=\textrm{Softmax} \big{(}\textrm{Linear}({\bf H}_{i}^{VTL})\big{)}.\]

With the generic layout token [LOC] incorporated, the text-modeling term in Eq. (2) is expressed as

\[\mathcal{L}_{\textrm{global-text}}=-\frac{1}{|\hat{\mathbf{T}}|}\sum_{i=1}^{| \hat{\mathbf{T}}|}\log p(\hat{\mathbf{T}}_{i}|\hat{\mathbf{T}}_{<i},\mathbf{V}). \tag{3}\]

#### 2.2.2 Local Layout Modeling

Local layout modeling aims to generate specific layout locations for each generic layout token [LOC].

Figure 2: Overview of the ViTLP architecture. ViTLP is a generative pre-training model that performs autoregressive text-layout modeling conditioned on visual document inputs. ViTLP adopts hierarchical decoder heads to generate target text-layout sequences in a _global-to-local_ manner. The segment mode tokens \(\in\{\texttt{[BOS]},\texttt{[CONT]}\}\) prompt the beginning and continuous modes of generation, respectively.



To capture the spatial relation among coordinates, we employ a lightweight sequential MLP layout head (see details in Appendix B) to decode the short sequence of four layout coordinate tokens from the last hidden state of [LOC]. For notation simplicity, we denote as the corresponding layout coordinates of the [LOC] token at the -th position, and its generative probability is modeled as

where is selected from the learned multimodal representations where . Here, we denote the index set of [LOC] tokens as . The layout-modeling term in Eq. (2) is expressed as

In summary, with the global and local text-layout modeling in a hierarchy, the original pre-training objective in Eq. (2) evolves to

The _global-to-local_ generation process aims to be effective and efficient for text-layout modeling. On effectiveness, the interleaved text-layout sequence modeling enables compact interaction between text and layout inputs, which can effectively fuse the information of text and layout modalities. On efficiency, suppose that the average BPE tokens of a document word are , and the _compression ratio_ of the text-layout sequence is , i.e., four coordinate tokens are compressed to one. In our experiment datasets, the _compression ratio_ is .

### Multi-segment Pre-training Scheme

Documents are usually intensive in text and layout, and it would be computationally intractable to fit the entire sequence into a generative model. To process documents with arbitrary length, we propose a multi-segment pre-training scheme that divides the long sequence into multiple segments for generation. Since a document image already contains all necessary information of the text and layout, long document modeling is feasible based on the _visual representations_ and _localized generation-context_.

Given the maximum sequence length of the decoder as , we first divide the text-layout sequence into segmented sequences . The beginning segment contains to be generated, and the continuous segment contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contains contain 



#### 2.4.2 Downstream VDU Tasks

Information Extraction.The information extraction task is formulated as sequence labeling on the target texts given document image input. Following BART Lewis et al. (2020), we feed ViTLP decoder's final hidden states of a target word (with layout coordinate inputs) to a linear classifier which outputs the token-level semantic label.

Document Classification.Given an input document image to the encoder, we feed a task prefix token [DOC_CLS] as input to the decoder to output the document classification label.

Document Visual Question Answering.Unlike discriminative VDU models that perform extractive QA on pre-processed OCR results, ViTLP directly generates answers given a task prefix token [VQA] followed by the question. It is noteworthy that ViTLP can intrinsically generate interpretable grounding **regions of interest (ROI)**, i.e., layout coordinates of answers, to verify the generation.

## 3 Experiments

### Experiment Setup

Implementation Details.We implement ViTLP with a \(12\)-layer ViT Dosovitskiy et al. (2021) image encoder and a \(6\)-layer text-layout decoder. The Transformer hidden size is \(d=768\) with \(12\) attention heads. In pre-training, the input image height and width are \(1920{\times}1600\) with the \(32{\times}32\) ViT patch size, and the decoder segmented sequence length is \(M=1024\). Following LayoutLMv2 Xu et al. (2021), the layout location coordinates are normalized into discrete bins of \([0,1000]\), resulting that the vocabulary size of the layout head is \(1001\). The multi-segment prefix ratio is set as \(\alpha_{p}=0.25\). We use the AdamW optimizer Loshchilov and Hutter (2019) to train ViTLP in \(250\)K steps, with the batch size of \(384\) and initial learning rate of \(2e{-}4\) with cosine decay. More implementation details are provided in Appendix A.2.

Pre-training Data.Following prior work Xu et al. (2021), we use IIT-CDIP Test Collection \(1.0\)Lewis et al. (2006) containing \(11\)M document images for pre-training. Following DONUT Kim et al. (2022), we generate \(2\)M synthetic document images with text and layout annotations. Another four supplementary datasets with \(0.4\)M document images are also added to augment the diversity of pre-training data, including PubLayNet Zhong et al. (2019), DocBank Li et al. (2020), SciTSR Chi et al. (2019), and IAM Marti and Bunke (2002). We use our internal OCR tool to extract words with location coordinates from the IIT-CDIP and PubLayNet images. Words with locations are provided in IAM, SciTSR, and DocBank. Refer to Appendix A.1 for more detailed data statistics.

Evaluation Tasks.We highlight that ViTLP are capable of handling both \(1)\)_perception tasks_ of document OCR and \(2)\)_cognition tasks_ of visual document understanding (VDU). To evaluate the comprehensive capabilities of ViTLP, we compare to baselines on each task as summarized in Table 1.

For OCR evaluation, we conduct two benchmark OCR sub-tasks, i.e., document text _localization_ and _recognition_. We evaluate model performance on SROIE competition4 Task #1 for text localization and Task #2 for text recognition. The text localization task is evaluated by DetEval protocol Wolf and Jolion (2006) which calculates the precision, recall, and F1 based on the _area of overlapping regions_ between model predictions and ground-truth text coordinates. The text recognition task evaluates the word-level precision, recall, and F1 based on exact word match.

Footnote 4: [https://rrc.cvc.uab.es/?ch=13&com=tasks](https://rrc.cvc.uab.es/?ch=13&com=tasks)

For VDU evaluation, we conduct three document understanding tasks. \(1)\)_Form Understanding_. Given a document image and its word entities, it is a sequential labeling task to predict the BIO tags for each textual entity. We use FUNSD Jaume et al. (2019) which contains \(199\) scanned forms, and the entities are labeled in four categories: _Header_, _Question_, _Answer_, and _Other_. FUNSC is divided into \(149\) images for training and \(50\) for testing. We report entity-level F1 as the evaluation score. \(2)\)

\begin{table}
\begin{tabular}{l|c c c c c c} \hline Approach & \multicolumn{3}{c}{OCR Tasks} & \multicolumn{3}{c}{VIDUs Tasks} \\  & Text Local. & Text Recog. & Info. Extraction & Doc. Classification & Document VQA & VQA Grounding \\ \hline OCR Pipelines & ✓ & ✓ & & & & \\ Discriminative VDU Models & & & ✓ & ✓ & ✓ & \\ Generative VDU Models & & & ✓ & ✓ & ✓ & ✓ \\ ViTLP & ✓ & ✓ & ✓ & ✓ & ✓ \\ \hline \end{tabular}
\end{table}
Table 1: The comprehensive capabilities of ViTLP and its comparison with the associated baselines on each task.



_Receipt Understanding._ We use CORD Park et al. (2019) containing \(800\) training and \(100\) testing images of real-world receptions. The receipt entities are labeled in \(30\) categories. We use entity-level F1 for evaluation. \(3)\)_Document Classification._ We conduct experiments on the RVL-CDIP dataset Harley et al. (2015) containing \(400\)K scanned documents in \(16\) classes. We adopt classification accuracy as the evaluation metric. For the sequence labeling tasks on FUNSD, we perform multi-segment fine-tuning on those samples whose entity-word sequences exceed the maximum decoder sequence length. This differs from previous work that truncates the input sequences into certain tokens, e.g., \(512\) tokens in LayoutLMv\(2\)Xu et al. (2021).

Besides, we evaluate generative question answering tasks on the DocVQA Mathew et al. (2020) and InfographicVQA Mathew et al. (2022) datasets. DocVQA consists of \(12\)K document images with \(50\)K QA pairs, and InfographicVQA contains \(5.4\)K document images with \(30\)K QA pairs. Since the answer word locations are not provided in the training sets, we use an OCR tool to locate the coordinates of answer words with heuristic text matching. In this way, we feed the answers with grounding coordinates to ViTLP for document VQA fine-tuning.

### OCR Evaluation Results

We compare ViTLP with representative OCR baselines on SROIE \(2019\) benchmark Huang et al. (2019). The text localization baselines include CRAFT Baek et al. (2019), YOLO-v3 Redmon and Farhadi (2018), CTPN Tian et al. (2016), and EAST Zhou et al. (2017). The text recognition baselines include BiLSTM-ResNet, BiLSTM-CTC Lee and Osindero (2016), UNet-CRNN Ronneberger et al. (2015), and TrOCR Li et al. (2023). Unlike conventional OCR models that first perform text localization and then use the localized text-regions for text recognition, ViTLP performs text localization and recognition in unified text-layout sequence generation, which does not need ground truth text-region inputs in the recognition task.

Table 2 shows the OCR evaluation performance. ViTLP outperforms most baseline methods on both localization and recognition tasks. ViTLP underperforms TroCR, given that TrOCR is a strong pre-trained model for two-stage OCR text recognition, while ViTLP performs text localization and recognition in one stage. Note that the SROIE training samples are few, i.e., only \(626\) images, and the input text coordinates are at textline-level, which are different from our word-level pre-training input format and thus render it challenging to fine-tune our model. Nonetheless, ViTLP can still achieve competitive performance by fine-tuning on the limited samples without additional data augmentation Li et al. (2023), successfully adapting to output the textline coordinates that have never met in the pre-training phase. We also provide qualitative ViTLP zero-shot OCR examples in Appendix C.

### VDU Evaluation Results

We compare ViTLP with competitive pre-trained baselines including i) general method RoBERTa Liu et al. (2019), ii) discriminative VDU models: LayoutLM Xu et al. (2020), SPADE Hwang et al. (2021), SelfDoc Li et al. (2021), TITL Powalski et al. (2021), LayoutLMv\(2\)Xu et al. (2021), LiLT Wang et al. (2022), FormNet Lee et al. (2022) and iii) generative VDU model DONUT Kim et al. (2022). Table 3 shows the VDU task performance.

Information Extraction.According to Table 3, our model achieves better F\(1\) scores compared to most baselines on FUNSD and CORD. The results indicate that ViTLP can develop a thorough understanding of form/receipt structures from images. Nonetheless, ViTLP underperforms the best discriminative baselines, i.e., LiLT on FUNSD and FormNet on CORD. We believe this is because pretrained discriminative VDU models have natural advantages over generative models for the information extraction task, which is formulated as token-level classification. Besides, ViTLP outperforms DONUT, proving that layout modeling is as nec

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & \multicolumn{4}{c}{_Text Localization Task_} \\
**Method** & **Area-Precision** & **Area-Recall** & **Area-F1** \\ \hline CRAFT & 62.73 & 59.94 & 61.31 \\ YOLO-v3 & 77.29 & 79.32 & 78.29 \\ CTPN & 81.14 & 87.23 & 84.07 \\ EAST & 85.07 & 87.17 & 86.11 \\ ViTLP & 91.62 & 91.68 & 91.65 \\ \hline \hline \end{tabular} 
\begin{tabular}{l c c} \hline \hline  & \multicolumn{4}{c}{_Text Recognition Task_} \\
**Method** & **Word-Precision** & **Word-Recall** & **Word-F1** \\ \hline BiLSTM-ResNet & 74.05 & 77.81 & 75.88 \\ BiLSTM-CTC & 83.38 & 87.37 & 85.33 \\ UnCRFnn & 85.77 & 86.48 & 86.12 \\ TriOCR\({}^{\dagger}\) & 95.89 & 95.74 & 95.82 \\ ViTLP & 93.07 & 92.52 & 92.79 \\ \hline \hline \end{tabular}
\end{table}
Table 2: OCR text localization and recognition results on SROIE \(2019\) benchmark. \({}^{\dagger}\)TrOCR uses the ground-truth cropped image regions as inputs, whereas ViTLP performs text localization and recognition in a unified stage. All scores are reported in percentage.

 

essary as language modeling to generative VDU models. For example, for the CORD images, entities with the same semantic label <menu.price> are always located in the same rightmost column of the receipt, sharing adjacent layout coordinates. Layout modeling can help generative VDU models better extract such structural-aware information.

Document Classification.From Table 3, we can see that ViTLP achieves the second best performance on classification accuracy. We also observe that the performance among TILT, LayoutLMv2, DONUT, and ViTLP are quite close. This may be because document classification is a coarse-grained task, wherein the vision modality contributes the most to classification performance, and the OCR text modality brings an incremental gain. Though ViTLP is suboptimal compared to LiLT, OCR-free generative methods are more flexible and lightweight because no pre-processed OCR texts are needed for input.

### Further Discussion

#### 3.4.1 Ablation Study

We conduct ablation studies on the effect of hierarchical text-layout modeling and multi-segment pre-training scheme. We compare ViTLP with three variants: i) pre-training with the language modeling objective only, without the layout modeling objective; ii) truncating long input document sequences in pre-training, without the multi-segment strategy; iii) generating four layout coordinate tokens for each word in a long flatten sequence, without hierarchical text-layout modeling.

Table 4 displays the ablation performance. We can observe that discarding the layout modeling objective leads to a substantial performance drop, i.e., \(6.19\) and \(4.05~{}\mathrm{F}1\) drops on FUNSD and CORD. The results suggest that generative pre-training on the layout modality can enhance the document understanding capability of VDU models. Besides, truncating long document inputs without the multi-segment pre-training strategy leads to lower performance. We believe that the multi-segment pre-training scheme enables ViTLP to model complete text and layout tokens of the pre-training corpora, which benefits the pre-trained model performance. We can also see that removing hierarchical text-layout modeling causes performance descent. It validates that hierarchical modeling is effective for interleaved text-layout information fusion.

#### 3.4.2 Generative Document VQA

Results and Analysis.Table 5 presents the performance of generative VDU models on DocVQA and InfographicVQA datasets. We can see that ViTLP underperforms DONUT by a slight margin

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Ablation Variants** & **FUNSD (F1)** & **CORD (F1)** \\ \hline ViTLP & 87.61 & 95.59 \\ \hline w/o layout modeling & 81.42 & 91.54 \\ w/o multi-segment training & 86.73 & 95.01 \\ w/o hierarchical modeling & 86.28 & 94.86 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Ablation model performance on the information extraction tasks.

\begin{table}
\begin{tabular}{l l r r r r r} \hline \hline
**Method** & **Modeling Type** & **\# Param.** & \multicolumn{2}{c}{
\begin{tabular}{c} **Maximum** \\ **Doc-Length** \\ \end{tabular} } & \multicolumn{1}{r}{**FUNSD (F1)**} & \multicolumn{1}{r}{**CORD (F1)**} & \multicolumn{1}{r}{**RVL-CDIP (Acc)**} \\ \hline RoBERTa\({}_{base}\)Liu et al. (2019) & & 125M & 512 & 66.48 & 93.54 & 90.06 \\ LayoutLMv\({}_{base}\)Xu et al. (2020) & & 160M & 512 & 79.27 & – & 94.42 \\ SPADE Hwang et al. (2021) & & 110M & 512 & 70.50 & 91.50 & – \\ SelfDoc Li et al. (2021b) & Discriminative & 137M & 1024 & 83.36 & – & 93.81 \\ TILLv\({}_{base}\)Wowalski et al. (2021) & (w/ OCR Input) & 230M & 512 & – & 95.11 & 95.25 \\ LayoutLMv\({}_{base}\)Xu et al. (2021) & & 200M & 512 & 82.76 & 94.95 & 95.25 \\ LiLTv\({}_{base}\)Wang et al. (2022a) & & – & 512 & 88.41 & 96.07 & 95.68 \\ FormNet Lee et al. (2022) & & 217/345M\({}^{\dagger}\) & 1024 & 84.69 & 97.28 & – \\ \hline DONUT Kim et al. (2022) & Generative & 259M & 1536 & – & 84.10 & 95.30 \\ ViTLP & (w/o OCR Input) & 253M & Any-length & 87.61 & 95.59 & 95.36 \\ \hline \hline \end{tabular}
\end{table}
Table 3: YDU evaluation results on form understanding (FUNSD), receipt understanding (CORD), and document classification (RVL-CDIP). \({}^{\dagger}\)FormNet has different sizes of \(217\)M and \(345\)M for FUNSD and CORD Lee et al. (2022). “Maximum Doc-Length” denotes the maximum tokens of an input text sequence that the model can handle.

 

on DocVQA and surpasses DONUT by a significant margin on InfographicvQDA. As discussed in Kim et al. (2022), DocVQA images are similar to the pre-training IIT-CDIP images, pre-training data quality may have a considerable influence on the performance of DocVQA. The average results show that ViTLP develops better overall document VQA performance than the strong generative model DONUT, which validates the effectiveness of our generative pre-training approach.

**Document VQA with Interpretable Grounding.** Owing to the fine-grained _word-level_ grounding capability learned in the pre-training stage, ViTLP can be fine-tuned to predict the regions of interest (ROI) associated with the generated answers, which is unprecedented to prior document VQA models. As shown in Figure 3, the output ROI groundingboxes _as visual rationale_s can help humans easily verify the model-generated answers, making the answer generation process _interpretable to humans_ where the model output derives from. See more examples of grounding document VQA with ViTLP in Appendix D.

## 4 Related Work

Visual document processing with multimodal pre-training is widely studied. From the perspectives of the document processing pipelines and model architectures, existing works can be generally divided into strands of research as listed below.

**OCR-based Methods.** Most initial VDU efforts adopt OCR tools to localize and recognize document layouts and texts, and then feed them to the multimodal pre-trained models Xu et al. (2021); Appalaraju et al. (2021); Li et al. (2021a); Peng et al. (2022); Li et al. (2021b); Bai et al. (2023); Lee et al. (2023). These methods usually involve multiple pre-training objectives over the vision, text, and layout. For instance, document text location Xu et al. (2020, 2021), paragraph and table regions Li et al. (2021b); Wang et al. (2022) are rich in structural information to align visual features with text embeddings. Though promising, these pipeline models suffer from heavy OCR pre-processing overhead. Moreover, incorrect OCR results may propagate errors to downstream tasks like document question answering Kim et al. (2022).

**OCR-free Methods.** There appear recent studies Kim et al. (2022); Lee et al. (2023); Kil et al. (2023) that jointly consider text reading and understanding without external OCR pipelines. For instance, Kim et al. (2022) takes document images as input to the model without prerequisite OCR results and conducts visual language pre-training. Lee et al. (2023) further improves the pre-training objectives over large-scaled visual webpage corpora. Kil et al. (2023) employs multiple pre-training tasks jointly to encourage the pre-trained model to learn text recognition capability explicitly and spatial reasoning capability implicitly.

Our research falls within the OCR-free branch. Different from existing works, we first study generative joint text-layout modeling conditioned on input document images. Our empirical results also validate that layout information not only enhances the learned representations for downstream VDU tasks but also can make the generation outputs more interpretable with visual groundings.

Figure 3: Visualization of ViTLP generated answers on DocVQA. The ViTLP output answer sequences consist of answer words (in blue) and corresponding location coordinates (in red). For direct visualization, we draw the region of interest (ROI) **referring to the output layout coordinates** on the image.

 



**LLM-backbone Methods.** Most recent studies leverage large language models (LLMs) to tackle multimodal document tasks Zhang et al. (2023); Ye et al. (2023); Wang et al. (2023). LLaVAR Zhang et al. (2023) inherits LLAVA architecture Liu et al. (2023) which directly projects the visual features to LLM embeddings and performs instruction tuning on visual document data. DocLLM Wang et al. (2023) uses spatial attention to inject \(2D\) layout information into Llama \(2\)Touvron et al. (2023) with supervised fine-tuning and first enables LLMs to process document information extraction tasks. Thanks to LLMs' powerful reasoning and generation abilities, utilizing LLMs for visual document processing has become a prominent research trend.

## 5 Conclusion

We propose visually guided generative text-layout pre-training (ViTLP) to enhance visual document processing covering the OCR and VDU tasks. In the pre-training phase, ViTLP optimizes hierarchical language and layout modeling objectives to generate interleaved text-layout target sequences. Moreover, the proposed multi-segment pre-training scheme enables ViNLP to process long documents with arbitrary lengths. ViTLP can function as a native OCR model to locate and recognize texts of document images. Experiments also show that ViTLP achieves superior performance on various VDU tasks with document grounding capability.

### Limitations

Our community has entered the era of large language models with multimodal capabilities Dai et al. (2023); OpenAI (2023). However, regarding the model size, ViTLP is still a rather small-scale pre-trained model5, which limits its potential to become an interactive and generalized document AI assistant. In future work, we plan to explore two paths: i) scaling up ViTLP with more parameters and training data, extending it to a more powerful foundation document model; ii) integrating ViTLP's _document-specific_ text-layout image encoder with _generalized_ advanced LLMs Chiang et al. (2023); Touvron et al. (2023) and visual instruction tuning Liu et al. (2023); Zhu et al. (2024) to build up an interactive document AI assistant.

Footnote 5: It is because we commenced the ViTLP project in mid-2022 and finished pre-training in early 2023, see the first version at [https://openreview.net/forum?id=ARtBIBAmNR](https://openreview.net/forum?id=ARtBIBAmNR).

**Remarks and Future direction.** i) ViTLP processes document images already calibrated in angle. Hence, we use \(4\) coordinates to represent the localization of words. It is feasible to pre-train ViTLP to generate \(8\) coordinates which can represent the angle of words. We choose word-level segmentation for pre-training because _a word is the elementary unit of document texts_. Word-level segmentation is also beneficial to fine-grained grounding, e.g., VQA with answer-word grounding. ii) We propose a multi-segment processing scheme to permit long sequence lengths on the _decoder side_. However, the document pixel inputs are also constrained by the resolution on the ViT _encoder side_. For the problem of long document processing, ViTLP only tackles the half. Processing document images with high resolutions and multiple pages is an intriguing problem for future research.

### Acknowledgements

We appreciate constructive comments from anonymous ARR reviewers. We thank Bin Liang from CUHK for valuable discussion. This research work is partially supported by CUHK direct grant No. \(4055209\) and CUHK Knowledge Transfer Project Fund No. KPF23GWP20.

## References

* A. Appalaraju, B. Jasani, B. Urala Kota, Y. Xie, and R. Manmatha (2021)Docformer: end-to-end transformer for document understanding. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 973-983. Cited by: SS6.
* Y. Baek, B. Lee, D. Han, S. Yun, and H. Lee (2019)Character region awareness for text detection. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9357-9366. External Links: Document Cited by: SS6.
* Haoli Bai, Z. Liu, X. Meng, L. Wentao, S. Liu, Y. Luo, N. Xie, R. Zheng, L. Wang, L. Hou, J. Wei, X. Jiang, and Q. Liu (2023)Wukong-reader: multi-modal pre-training for fine-grained visual document understanding. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 13386-13401. External Links: Link, Document Cited by: SS6.
* I. Beltagy, M. E. Peters, and A. Cohan (2020)LongFormer: the long-document transformer. External Links: 2009.02786 Cited by: SS6.
* A. Bulatov, Y. Kuratov, and M. Burtsev (2022)Recurrent memory transformer. In Advances in Neural Information Processing Systems, Vol. 35, pp. 11079-11091. External Links: Link, Document Cited by: SS6.



* Chen et al. (2022) Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton. 2022. Pix2seq: A language modeling framework for object detection. In _International Conference on Learning Representations_.
* Chen et al. (2021) Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021. WebSRC: A dataset for web-based structural reading comprehension. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 4173-4185, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Chi et al. (2019) Zewen Chi, Heyan Huang, Heng-Da Xu, Houjin Yu, Wanxuan Yin, and Xian-Ling Mao. 2019. Complicated table structure recognition.
* Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
* Dai et al. (2023) Wenliang Dai, Junnan Li, DONGXU LI, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2023. Instructublip: Towards general-purpose vision-language models with instruction tuning. In _Advances in Neural Information Processing Systems_, volume 36, pages 49250-49267. Curran Associates, Inc.
* ECCV Workshops: Tel Aviv, Israel, Proceedings, Part IV_, page 280-296, Berlin, Heidelberg. Springer-Verlag.
* Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissnborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_.
* Gu et al. (2021) Jiuviang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmalios, Anisin Nekova, and Tong Sun. 2021. Unidoc: Unified pretraining framework for document understanding. In _Advances in Neural Information Processing Systems_, volume 34, pages 39-50. Curran Associates, Inc.
* Harley et al. (2015) Adam W. Harley, Alex Ufkes, and Konstantinos G. Derpanis. 2015. Evaluation of deep convolutional nets for document image classification and retrieval. ICDAR 2015, page 991-995.
* Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus).
* Huang et al. (2022) Yunpau Huang, Tengchao Lv, Lei Cui, Vutong Lu, and Furu Wei. 2022. Layoutlm3: Pre-training for document ai with unified text and image masking. In _Proceedings of the 30th ACM International Conference on Multimedia_, page 4083-4091, New York, NY, USA. Association for Computing Machinery.
* Huang et al. (2019) Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar. 2019. Icdar2019 competition on scanned receipt ocr and information extraction. In _2019 International Conference on Document Analysis and Recognition (ICDAR)_, pages 1516-1520.
* Hwang et al. (2021) Wonseok Hwang, Jinyeong Yim, Seunghyun Park, Sohee Yang, and Minjoon Seo. 2021. Spatial dependency parsing for semi-structured document information extraction. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 330-343, Online. Association for Computational Linguistics.
* Jaume et al. (2019)Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019. Funsd: A dataset for form understanding in noisy scanned documents. In _2019 International Conference on Document Analysis and Recognition Workshops (ICDARW)_, volume 2, pages 1-6.
* Katti et al. (2018) Anoop R Katti, Christian Reisswig, Cordula Guder, Sebastian Brada, Steffen Bickel, Johannes Hohne, and Jean Bapistte Faddoul. 2018. Chargird: Towards understanding 2D documents. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4459-4469, Brussels, Belgium. Association for Computational Linguistics.
* Kil et al. (2023) J. Kil, S. Changpinyo, X. Chen, H. Hu, S. Goodman, W. Chao, and R. Soricut. 2023. Prestu: Pre-training for scene-text understanding. In _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_, pages 15224-15234.
* ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVIII_, page 498-517, Berlin, Heidelberg. Springer-Verlag.
* Lee and Osindero (2016) C. Lee and S. Osindero. 2016. Recursive recurrent nets with attention modeling for ocr in the wild. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 2231-2239.
* Lee et al. (2022) Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, and Tomas Pfister. 2022. FormNet: Structural encoding beyond sequential modeling in form document information extraction. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3735-3754, Dublin, Ireland. Association for Computational Linguistics 

Cited by: SS2.
* C. Lee, C. Li, H. Zhang, T. Dozat, V. Perot, G. Su, X. Zhang, K. Sohn, N. Qih, I. Glushnev, F. Wang, Y. Ainslie, S. Long, S. Qin, Y. Fujii, N. Hua, and T. Pfister (2023)FormNetV2: multimodal graph contrastive learning for form document information extraction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 9011-9026. External Links: Link Cited by: SS4.1.
* K. Lee, M. Joshi, I. Raluca Turc, R. Hu, F. Liu, J. Martin Eisenschlos, U. Khanedelwal, P. Shaw, M. Chang, and K. Toutanova (2023)Pix2Struct: screenshot parsing as pretraining for visual language understanding. In Proceedings of the 40th International Conference on Machine Learning, Vol. , pp. 18893-18912. External Links: Link Cited by: SS2.
* D. Lewis, G. Agam, S. Argamon, O. Frieder, D. Grossman, and J. Heard (2006)Building a test collection for complex document information processing. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, NY, USA, pp. 665-666. External Links: ISBN 978-1-4503-3811-1, Document, Link, [https://doi.org/10.1146/S0378-1-4503-3811-1](https://doi.org/10.1146/S0378-1-4503-3811-1), Link, Document Cited by: SS2.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2020)BERT: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 7871-7880. External Links: Link, Document Cited by: SS2.
* C. Li, B. Bi, M. Yan, W. Wang, S. Huang, F. Huang, and L. Si (2021a)StructuralLM: structural pre-training for form understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 6309-6318. External Links: Link, Document Cited by: SS2.
* M. Li, T. Lv, J. Chen, L. Cui, Y. Lu, D. Florencio, C. Zhang, Z. Li, and F. Wei (2023)Trocr: transformer-based optical character recognition with pre-trained models. Proceedings of the AAAI Conference on Artificial Intelligence37 (11), pp. 13094-13102. External Links: Link, Document Cited by: SS2.
* M. Li, Y. Xu, L. Cui, S. Huang, F. Wei, Z. Li, and M. Zhou (2020)DocBank: a benchmark dataset for document layout analysis. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, pp. 949-960. External Links: Link, Document Cited by: SS2.
* P. Li, J. Gu, J. Kuen, V. I Morariu, H. Zhao, R. Jain, V. Manjunatha, and H. Liu (2021b)Selfdoc: self-supervised document representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, pp. 5652-5660. External Links: Link, Document Cited by: SS2.
* H. Liu (2021)Selfdoc: self-supervised document representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, pp. 5652-5660. External Links: Link, Document Cited by: SS2.
* H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023)Visual instruction tuning. In Advances in Neural Information Processing Systems, Vol. 36, pp. 34892-34916. External Links: Link, Document Cited by: SS2.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)Roberta: a robustly optimized bert pretraining approach. External Links: 1905.02177 Cited by: SS2.
* I. Loshchilov and F. Hutter (2019)Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, Cited by: SS2.
* B. Prasad Majumder, N. Potti, S. Tata, J. Bradley Wendt, Q. Zhao, and M. Najork (2020)Representation learning for information extraction from form-like documents. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 6495-6504. External Links: Link, Document Cited by: SS2.
* U. Marti and H. Bunke (2002)The iamdatabase: an english sentence database for offline handwriting recognition. International Journal on Document Analysis and Recognition5, pp. 39-46. External Links: Link, Document Cited by: SS2.
* M. Mathew, V. Bagal, R. Tito, Dimosthenis Karatzas, E. Valveny, and C. V. Jawahar (2022)Infographicva. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2582-2591. External Links: Link, Document Cited by: SS2.
* M. Mathew, D. Karatzas, R. Manmatha, and C. V. Jawahar (2020)Docvqa: a dataset for vqa on document images. In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 2199-2208. External Links: Link, Document Cited by: SS2.
* OpenAI (2023)Gpt-4 technical report. External Links: Link Cited by: SS2.
* Seunghyun Park, S. Shin, B. Lee, J. Lee, J. Surh, M. Seo, and H. Lee (2019)Cord: a consolidated receipt dataset for post-ocr parsing. External Links: 1905.02177 Cited by: SS2.
* Q. Peng, Y. Pan, W. Wang, B. Luo, Z. Zhang, Z. Huang, Y. Cao, W. Yin, Y. Chen, Y. Zhang, S. Feng, Y. Sun, H. Tian, H. Wu, and H. Wang (2022)ERNIE-layout: layout knowledge enhanced pre-training for visually-rich document understanding. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dasbi, United Arab Emirates, pp. 3744-3756. External Links: Link, Document Cited by: SS2.

 

Rafal Powalski, Lukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michal Pietruszka, and Gabriela Palka. 2021. Going full-tilt boogie on document understanding with text-image-layout transformer. In _Document Analysis and Recognition-ICDAR 2021: 16th International Conference, Lausanne, Switzerland, Proceedings, Part II 16_, pages 732-747. Springer.
* Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
* Redmon and Farhadi (2018) Joseph Redmon and Ali Farhadi. 2018. Yolov3: An incremental improvement.
* Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In _MICCAI 2015_. MICCAI.
* Tian et al. (2016) Zhi Tian, Weilin Huang, Tong He, Pan He, and Yu Qiao. 2016. Detecting text in natural image with connectionist text proposal network. In _European Conference on Computer Vision_.
* Touvron et al. (2016) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esibou, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kolumann, Artem Koreney, Punit Singh Koruar, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Dinana Liskovich, Yinghai Liu, Yuning Mao, Xavier Martinet, Todor Mihaloylov, Pushkar Mishra, Igor Molbyg, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Runeta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Pavxin Xu, Zheng Yan, Iiiyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. LLama 2: Open foundation and fine-tuned chat models.
* Wang et al. (2023) Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, and Xiaomo Liu. 2023. Docllm: A layout-aware generative language model for multimodal document understanding.
* Wang et al. (2022a) Jiapeng Wang, Lianwen Jin, and Kai Ding. 2022a. LiLT: A simple yet effective language-independent layout transformer for structured document understanding. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7747-7757, Dublin, Ireland. Association for Computational Linguistics.
* Wang et al. (2022b) Zilong Wang, Jiuxiang Gu, Chris Tensmeyer, Nikolaos Barmpalios, Ani Nenkova, Tong Sun, Jingbo Shang, and Vlad Morariu. 2022b. MgDoc: Pre-training with multi-granular hierarchy for document image understanding. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3984-3993, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Wolf and Jolion (2006) Christian Wolf and Jean-Michel Jolion. 2006. Object count/area graphs for the evaluation of object detection and segmentation algorithms. _Document Analysis and Recognition_, 8:280-296.
* Xu et al. (2021) Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxiang Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021. LayoutLMv2: Multi-modal pre-training for visually-rich document understanding. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 2579-2591, Online. Association for Computational Linguistics.
* Xu et al. (2020) Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. Layoutlm: Pre-training of text and layout for document image understanding. In _KDD 2020_, page 1192-1200, New York, NY, USA. Association for Computing Machinery.
* Ye et al. (2023) Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. 2023. mplug-docowl: Modularized multimodal large language model for document understanding.
* Zhang et al. (2023) Yanzhe Zhang, Ruiyi Zhang, Jiaxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. 2023. LIavar: Enhanced visual instruction tuning for text-rich image understanding.
* Zhong et al. (2019) X. Zhong, J. Tang, and A. Jimeno Yepes. 2019. Publaynet: Largest dataset ever for document layout analysis. In _2019 International Conference on Document Analysis and Recognition (ICDAR)_, pages 1015-1022.
* Zhou et al. (2017) Xingyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. 2017. East: An efficient and accurate scene text detector. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 2642-2651.
* Zhu et al. (2024) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoesity. 2024. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In _The Twelfth International Conference on Learning Representations_ 



## Appendix A Experiment Details

### Pre-training Data Statistics

Table 6 shows the pre-training data statistics. Following previous work, e.g., LayoutLMv2 Xu et al. (2021), we use \(11\)M IIT-CDIP document images as the main pre-training data. Besides, we follow Kim et al. (2022) and Davis et al. (2022) to include \(2\)M machine-rendered synthetic documents for generative pre-training. Specifically, we adapt the official SynthDog generator6 to generate synthetic document images with text and layout metadata. The other four corpora, i.e., PublayNet, DocBank, SciTSR, and IAM, account for only \(\sim 3\%\) pre-training data whereby we aim to improve the diversity of pre-training document types.

Footnote 6: [https://github.com/clovaai/donut/tree/master/synthdog](https://github.com/clovaai/donut/tree/master/synthdog)

The distribution of document sequence lengths is displayed in Figure 4. The number of text-layout sequence tokens follows a _long-tailed distribution_: there exist some long documents with the sequence lengths ranging from \(1024\) to \(3072\). This brings a trade-off to pre-training. With a relatively short sequence length (e.g., \(512\) tokens in LayoutLMv2), language modeling on long documents is incomplete, as the sequence tokens are truncated and wasted. However, with a relatively long sequence length (e.g., \(3072\)), the GPU computation and memory overload would become prohibitive, which also forbids large batch sizes for better performance.7 The proposed multi-segment pre-training scheme can circumvent this bitter trade-off. Notably, the multi-segment processing scheme can be directly applied to long document fine-tuning (and inference). For example in the OCR and sequence labeling tasks, ViTLP also employs the multi-segment scheme to process the long documents by multiple segments with prefix context tokens.

Footnote 7: Even assuming sufficient computation resources, the long-tailed distribution of document lengths would also cause enormous padding tokens in long sequence input to Transformers, leading to considerable waste of computational resources.

### Fine-tuning Hyperparameter Settings

OCR Text Localization and Recognition.Fine-tuning ViTLP for text localization and recognition follows the same objective Eq. (5) as pre-training. Since the SROIE \(2019\)Huang et al. (2019) training set is rather small containing only \(626\) images, we fine-tune ViTLP for \(10\) epochs with the batch size of \(1\). The used learning rate and weight decay are \(2e{-}5\) and \(1e{-}2\). The input image resolution remains the same as pre-training, i.e., \(1920{\times}1600\).

Information Extraction.For FUNSD Jaume et al. (2019), the selected learning rate and weight decay are \(1e{-}4\) and \(1e{-}2\). For CORD Park et al. (2019), the selected learning rate and weight decay8 are \(5e{-}5\) and \(1e{-}4\). For both datasets, we fine-tune ViTLP for \(75\) epochs with the batch size of \(8\), using the same input image resolution as pre-training. Following the practice of prior work Huang et al. (2022); Lee et al. (2023), we use the shared segment-level layout coordinates as input instead of word-level coordinates, which can benefit the token classification accuracy in sequence labeling.

Footnote 8: For CORD, we search the configuration of learning rate in \(\{2e{-}4,1e{-}4,5e{-}5,3e{-}5,2e{-}5,1e{-}5\}\) and weight decay in \(\{1e{-}2,1e{-}4\}\).

Document Classification.We use the learning rate of \(1e{-}4\) and weight decay of \(1e{-}2\) for the document classification task. We fine-tune ViTLP for \(100\) epochs with the global batch size of \(320\). The input image resolution is the same as pre-training.

Document VQA.Since the layout coordinates of answer words are not provided in the DocVQA Mathew et al. (2020) and InfographicVQA

\begin{table}
\begin{tabular}{c c c c} \hline \hline Dataset & Size & Proportion & Document Type \\ \hline IIT-CDIP & \(10,816,672\) & \(81.89\%\) & Scanned Document \\ SynthDog & \(2,000,000\) & \(15.14\%\) & Synthetic Document \\ PublayNet & \(261,076\) & \(1.98\%\) & Scientific Paper \\ DocBank & \(125,815\) & \(0.95\%\) & Arxiv Paper \\ SciTSR & \(3,536\) & \(0.03\%\) & Figure and Table \\ IAM & \(1,198\) & \(0.01\%\) & Hand Written \\ \hline \hline \end{tabular}
\end{table}
Table 6: Pre-training dataset statistics.

Figure 4: Distribution of document sequence lengths. The text sequences are tokenized by the standard BPE tokenizer Radford et al. (2019).



(Mathew et al., 2022) datasets, we first conduct OCR on the training document images to obtain the texts with bounding-box coordinates. Then we apply a heuristic text-matching method to assign corresponding bounding-box coordinates to the answer words. It is worth noting that for the "Yes/No" questions that have no grounding answers on the images, we train ViTLP to generate a special answer token [ANS_YES] or [ANS_NO] without layout coordinates. For both datasets, we fine-tune ViTLP for \(60\) epochs with a batch size of \(128\). We use a learning rate of \(3e{-5}\). Since the document images are high-resolution, for DocVQA, we set the fine-tuning image resolution as \(2304{\times}1920\) which is multiplied by \(1.2\) based on the pre-training resolution. For InfographicVQA, the fine-tuning image resolution is set as \(3200{\times}1600\). From our empirical experiments, we find that input image resolution is essential to document VQA performance, especially for InfographicVQA.

## Appendix B Implementation Details of Sequential Layout Head

Given that multimodal interaction is learned by the stacked Transformer text-layout decoder layers, the LM and layout heads hereby function as a proper to output the next word and coordinate predictions. As introduced in Sec 2.2.2, the layout head predicts output probability \(\mathrm{Prob}(\mathbf{L}_{i,j})\) of the four coordinates \(\{\mathbf{L}_{i,j}\}^{d}_{j=1}=\{z_{x_{1}},z_{y_{1}},z_{x_{2}},z_{yz_{2}}\}_{ i}\) based on the \(i\)-th global [LOC] token's final hidden state \(\mathbf{H}_{i,0}=\mathbf{H}_{i}^{UTL}\in\mathbb{R}^{d}\) as follows.

\[\begin{cases}\mathbf{H}_{i,1}=\text{GELU}\big{(}\mathbf{W}_{h}\mathbf{H}_{i,0 }\big{)}\\ \mathbf{H}_{i,2}=\text{GELU}\big{(}\mathbf{W}_{h}\mathbf{H}_{i,1}+\mathbf{E}_ {x}^{{}^{\prime}}(\mathbf{L}_{i,1})\big{)}\\ \mathbf{H}_{i,3}=\text{GELU}\big{(}\mathbf{W}_{h}\mathbf{H}_{i,2}+\mathbf{E}_ {y}^{{}^{\prime}}(\mathbf{L}_{i,2})\big{)}\\ \mathbf{H}_{i,4}=\text{GELU}\big{(}\mathbf{W}_{h}\mathbf{H}_{i,3}+\mathbf{E}_ {x}^{{}^{\prime}}(\mathbf{L}_{i,3})\big{)}\end{cases}\]

The coordinate tokens are quantized into a discrete range of \([0,1000]\), making the layout-token vocabulary size of \(|L|=1001\). The layout head's parameters are lightweight including a hidden matrix \(\mathbf{W}_{h}\in\mathbb{R}^{d\times d}\), two embeddings \(\mathbf{E}_{x}^{{}^{\prime}}(\cdot)\in\mathbb{R}^{d}\) and \(\mathbf{E}_{y}^{{}^{\prime}}(\cdot)\in\mathbb{R}^{d}\), and a linear projection \(\mathbf{W}_{L}\in\mathbb{R}^{|L|\times d}\). We use the same GELU activation (Hendrycks and Gimpel, 2016) as in the Transformer layers. The layout head works sequentially, which is similar to a vanilla RNN, as each coordinate decoding step also considers the information of previous coordinates. Compared with naively using four independent linear heads, the sequential layout head can capture the spatial relation among the output coordinates (e.g., \(x_{1}<x_{2}\) and \(y_{1}<y_{2}\)), bootstrapping more accurate coordinate prediction.

## Appendix C Qualitative Cases of ViTLP Document OCR Functionality

Figure 5 to 7 demonstrate ViTLP's functionality on zero-shot document OCR. ViTLP outputs the interleaved OCR sequence consisting of words and corresponding bounding boxes.

## Appendix D Qualitative Cases of ViTLP Document VQA with Grounding Capability

Figure 8 showcases the ViTLP's VQA outputs on DocVQA with grounding capability. The top two examples are successful cases, and the bottom two are failure cases.

 

* [145] M. C. D'Amico, A. J. L. Read, and R. S. Swanson, "The Large Scale Structure of the Universe: The Role of the Cosmic Microwave Background in the Early Universe," _Astrophys. J._**630**(2), 79-80
* [146] G. F. Hinton, "The Cosmological Constant Problem," _Ann. Rev. Astron. Astrophys._**53**(1), 103-114
* [147] G. F. Hinton, "The Cosmological Constant Problem," _Phys. Rev. Lett._**53**(1), 103-110
* [148] G. F. Hinton, "The Cosmological Constant Problem," _JCAP_**120**(1), 001-00

[MISSING_PAGE_POST]

 

Figure 6: ViTLP OCR results on a paper. For comprehensive visualization, we render the words and bounding boxes according to ViTLP’s interleaved output sequence. The shown generated OCR results comprise two segments, as the generated tokens reach the decoder sequence length (\(M=1024\)) in the first segment generation, and the generation process continues by the second segment. The bounding boxes of the first segment are in red, and the second are in green.

 



[MISSING_PAGE_POST]

 

* [42] C. M. R. Dickson, J. A. Harvey, and S. W. Kittel, "A new approach to the detection of a single-event event using a large number of events in the search space", _Journal of Physics B: Biological Sciences_**106**(3), 579-680, 2000.

[MISSING_PAGE_POST]