A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding

## Abstract

Abstract Recently, many studies have demonstrated that exclusively incorporating OCR-derived text and spatial layouts with large language models (LLMs) can be highly effective for document understanding tasks. However, existing methods that integrate spatial layouts with text have limitations, such as producing overly long text sequences or failing to fully leverage the autoregressive traits of LLMs. In this work, we introduce Interleaving Lay out and Text in a L arge L anguage M odel (LayTextLLM) for document understanding. In particular, LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. LayTextLLM not only streamlines the interaction of layout and textual data but also shows enhanced performance in Key Information Extraction (KIE) and Visual Question Answering (VQA). Comprehensive benchmark evaluations reveal significant improvements, with a 27.2% increase on KIE tasks and 12.0% on VQA tasks compared to previous state-of-the-art document understanding MLLMs, as well as a 15.1% improvement over other SOTA OCR-based LLMs on KIE tasks. Furthermore, we found that the spatial layout can be decoded back into coordinates, and inference requiring output bounding box coordinates can further alleviate the hallucination problem. All resources are available at https://github.com/LayTextLLM/LayTextLLM .

## 1 Introduction

Recent research has increasingly focused on applying Large Language Models (LLMs) [, , , , , , , , , , , , , , , , , , , , , , ] to document-oriented Visual Question Answering (VQA) and Key Information Extraction (KIE) scenarios. Efforts to build a text-sensitive MultiModal Large Language Models (MLLMs) based on existing LLMs, particularly aimed at enhancing Visually Rich Document Understanding (VRDU), have made significant progress [, , ]. Although existing MLLMs show promising results in document understanding, they often encounter challenges related to image resolution. When the input image is of low resolution, it is too blurry to extract visual features effectively. Conversely, high-resolution images require additional computational resources to capture detailed textual information [].

Concurrently, another line of research employs off-the-shelf OCR tools to extract text and spatial layouts, which are then combined with LLMs to address VRDU tasks. These approaches assume that most valuable information for document comprehension can be derived from the text and its spatial layouts, viewing spatial layouts as “lightweight visual information” []. Following this premise, several studies [, , , , ] have explored various approaches that integrate spatial layouts with text for LLMs, achieving results that are competitive with, or even surpass, those of MLLMs.

The most natural method to incorporate layout information is by treating spatial layouts as tokens, which allows for the seamless interleaving of text and layout into a unified text sequence [, , ]. For example,  Perot et al. [] employ format such as “HARRISBURG 78|09” to represent OCR text and corresponding layout, where “HARRISBURG” is OCR text and “78|09” indicates the mean of the horizontal and vertical coordinates, respectively. Similarly,  He et al. [] use “[x_min, y_min, x_max, y_max]” to represent layout information. These approaches can effectively take advantage of autoregressive characteristics of LLMs and is known as the “coordinate-as-tokens” scheme []. In contrast, DocLLM [] explores interacting spatial layouts with text through a disentangled spatial attention mechanism that captures cross-alignment between text and layout modalities.

Figure: Figure 1: The performance against input sequence length of different datasets across various OCR-based methods where data is from Tab. 2 and 5.
Refer to caption: x1.png

However, we argue that both of the previous approaches have limitations. As shown in Fig. 1, coordinate-as-tokens significantly increases the number of tokens. Additionally, to accurately comprehend coordinates and enhance zero-shot capabilities, this scheme often requires few-shot in-context demonstrations and large-scale language models, such as ChatGPT Davinci-003 (175B) [], which exacerbates issues related to sequence length and GPU resource demands. Meanwhile, although DocLLM does not increase sequence length and integrates spatial layouts through attention, its generalizability is limited. We believe that spatial cross attention and masked span tasks in DocLLM cannot fully utilize the autoregressive traits of LLMs.

To address these problems, this paper explores a simple yet effective approach to enhance the interaction between spatial layouts and text — Interleaving
Layout and Text in a Large Language Model (LayTextLLM) for document understanding. Adhering to the common practice of interleaving any modality with text [, , ], we specifically apply this principle to spatial layouts. In particular, we maps each bounding box to a single embedding, which is then interleaved with its corresponding text. Then we propose a tailored pre-training task—Layout-aware Next Token Prediction—a completely self-supervised task that enhances the alignment between layout and textual modalities without using synthetic data. Finally, through the proposed Shuffled-OCR Supervised Fine-tuning, LayTextLLM significantly improves performance on downstream document-related VQA and KIE tasks. As shown in Fig. 1, LayTextLLM significantly outperforms the 175B models, while only slightly increasing or even reducing the sequence length compared to DocLLM. Our contributions can be listed as follows:

- •
We propose LayTextLLM for document understanding. To the best of the authors’ knowledge, this is the first work to employ a unified embedding approach (Sec. 3.1.1) that interleaves spatial layouts directly with textual data within a LLM. By representing each bounding box with one token, LayTextLLM efficiently addresses sequence length issues brought by coordiante-as-tokens while fully leveraging autoregressive traits for enhanced document understanding.
- •
We propose two tailored training tasks: (1) Layout-aware Next Token Prediction (Sec. 3.2.1), a completely self-supervised training task to enhance the alignment between layout and textual modality; (2) Shuffled-OCR Supervised Fine-tuning task (Sec. 3.2.2) to better elicit the model generalizability in downstream tasks.
- •
Comprehensive experimental results demonstrate quantitatively that LayTextLLM significantly outperforms previous state-of-the-art (SOTA) OCR-free MLLMs by a large margin in zero-shot scenarios, particularly in KIE tasks with an improvement of 27.0%. Additionally, we illustrate that LayTextLLM competes effectively or even surpasses previous SOTA OCR-based methods in both zero-shot and SFT scenarios. Specifically, it surpasses DocLLM by 19.8% on VQA and 15.5% on KIE tasks (Sec. 4).
- •
Extensive ablations demonstrate the utility of the proposed component, with analysis showing that LayTextLLM not only improves performance but also reduces input sequence length compared to current OCR-based models.

## 2 Related Work

### 2.1 OCR-based LLMs for Document Understanding

Early document understanding methods [, , , , ] tend to solve the task in a two-stage manner, i.e., first reading texts from input document images using off-the-shelf OCR engines and then understanding the extracted texts. Considering the advantages of LLMs (e.g., high generalizability), some recent methods endeavor to combine LLMs with OCR-derived results to solve document understanding. For example, inspired by the “coordinate-as-tokens” scheme [],  He et al. [] propose to use “[x_min, y_min, x_max, y_max]” to introduce the layout information, which can fuse the layout information and texts into a unified text sequence and fully exploit the autoregressive merit of LLMs. To reinforce the layout information while avoiding increasing the number of tokens, DocLLM [] designs a disentangled spatial attention mechanism to capture cross-alignment between text and layout modalities. Recently, LayoutLLM [] utilizes the pre-trained layout-aware model [], to insert the visual information, layout information and text information. However, the aforementioned methods neither suffer from the computational overhead leading by the increasing tokens or hardly take advantage of autoregressive characteristics of LLMs. Thus, it is an urgent problem to address how to better incorporate layout information without significantly increasing the number of tokens.

### 2.2 OCR-free MLLMs for Document Understanding

Another approach to solve document understanding tasks is the OCR-free method. Benefiting from the end-to-end training framework, it involves processing the text content of documents directly, without relying on OCR engines. Donut [] first presents an OCR-free method through mapping a text-rich document image into the desired answers. Pix2Struct [] is trained to parse masked screenshots of web pages into simplified HTML, where variable resolution inputs are supported. While these approaches eliminate the need for OCR tools, they still necessitate task-specific fine-tuning. With the increasing popularity of LLMs/MLLMs [, , , , , , , ], various methods are proposed to solve the document understanding task through explicitly training models on visual text understanding datasets and fine-tuning them with instructions to perform a zero-shot prediction. LLaVAR [] and UniDoc [] are notable examples that expand upon the document-oriented VQA capabilities of LLaVA [] by incorporating document-based tasks. These models pioneer the use of MLLMs for predicting texts and coordinates from document images, enabling the development of OCR-free document understanding methods. Additionally, DocPedia [] operates document images in the frequency domain, allowing for higher input resolution without increasing the input sequence length. Recent advancements in this field, including mPLUG-DocOwl [], Qwen-VL [], and TextMonkey [], leverage publicly available document-related VQA datasets to further enhance the document understanding capability. Although these OCR-free methods have exhibited their advantages, they still struggle with the high-resolution input to reserve more text-related details.

## 3 Method

Figure: Figure 2: An overview of LayTextLLM incorporates interleaving bounding box tokens ($b^{i}$) with text tokens ($t^{i}$), where the superscripts represent the sequence positions of the tokens.
Refer to caption: x2.png

In this section, we present our LayTextLLM. First, we introduce a innovative Spatial Layout Projector (Sec. 3.1.1) converts four-dimensional layout coordinates into a single-token embedding. To reduce parameter overhead, we apply Partial Low-Rank Adaptation (Sec. 3.1.2). We also introduce two specific training tasks: Layout-aware Next Token Prediction (Sec. 3.2.1) to align layouts with text during pre-training, and Shuffled-OCR Supervised Fine-tuning (Sec. 3.2.2) to enhance the generalizability of the model. An illustration of our approach is shown in Fig. 2.

Figure: Figure 3: The illustration of P-LoRA, adapted from [].
Refer to caption: x3.png

### 3.1 Model Architecture

LayTextLLM is built on the Llama2-7B-base model, which was originally designed to accept only text inputs [, ]. To enable the model to interleave spatial layouts with text, we introduce a novel Spatial Layout Projector. This projector converts OCR-derived coordinates into bounding box tokens. We also adopt the Partial Low-Rank Adaptation, a minimally invasive method to incorporate additional modalities while preserving the LLM’s inherent knowledge intact.

#### 3.1.1 Spatial Layout Projector (SLP)

A key innovation in LayTextLLM is the Spatial Layout Projector (SLP), which transforms a spatial layout into a singular bounding box token. This enhancement enables the model to process both spatial layouts and textual inputs simultaneously. To be specifically, each OCR-derived spatial layout is represented by a bounding box defined by four-dimensional coordinates $[x_{1},y_{1},x_{2},y_{2}]$, these coordinates represent the normalized minimum and maximum horizontal and vertical extents of the box, respectively. The SLP maps these coordinates into a high-dimensional space that the language model can process as a single token. The process can be computed as $z=W\cdot c+b$, where $c\in\mathbb{R}^{4}$ is the vector of the bounding box coordinates. $W\in\mathbb{R}^{d\times 4}$ is a weight matrix with $d$ represents the dimension of the embedding, $b\in\mathbb{R}^{d\times 1}$ is a bias vector, $z$ is the resulting bounding box token represented as an $d$-dimensional embedding. As illustrated in Fig. 2, the resulting bounding box token $z$ will be interleaved with corresponding textual embeddings to put into LLMs. Note that the SLP is shared by all bounding box tokens so very limited number of parameters are introduced.

Compared to the coordinate-as-tokens scheme, the SLP represents each bounding box with a single token. This approach significantly reduces the number of input tokens and adheres to the practice of interleaving any modality with text, effectively integrating layout and textual information into a unified sequence. This allows the model to process both modalities simultaneously and coherently, fully leveraging the autoregressive traits of LLMs.

#### 3.1.2 Layout Partial Low-Rank Adaptation

After using the SLP to generate bounding box tokens and a tokenizer to produce text tokens, these two modalities are then communicated using a Layout Partial Low-Rank Adaptation (P-LoRA) module in LLMs. P-LoRA, introduced in InternLM-XComposer2 [], is originally used to adapt LLMs to visual modality. It applies plug-in low-rank modules specified to the visual tokens, which adds minimal parameters while preserving the LLMs inherent knowledge.

Formally, as shown in Fig. 3 for a linear layer in the LLM, the original weights $W_{O}\in\mathbb{R}^{C_{out}\times C_{in}}$ and bias $B_{O}\in\mathbb{R}^{C_{out}}$ are specified for input and output dimensions $C_{in}$ and $C_{out}$. P-LoRA modifies this setup by incorporating two additional matrices, $W_{A}\in\mathbb{R}^{C_{r}\times C_{in}}$ and $W_{B}\in\mathbb{R}^{C_{out}\times C_{r}}$. These matrices are lower-rank, with $C_{r}$ being considerably smaller than both $C_{in}$ and $C_{out}$, and are specifically designed to interact with new modality tokens, which in our case are bounding box tokens. For example, given an input $x=[x_{b},x_{t}]$ comprising of bounding box tokens ($x_{b}$) and textual tokens ($x_{t}$) is fed into the system, the forward process is as follows, where $\hat{x}_{t},\hat{x}_{b}$ and $\hat{x}$ are outputs:

$$ $\displaystyle\hat{x}_{t}=W_{0}x_{t}+B_{0}$ (1) $\displaystyle\hat{x}_{b}=W_{0}x_{b}+W_{B}W_{A}x_{b}+B_{0}$ $\displaystyle\hat{x}=\left[\hat{x}_{b},\hat{x}_{t}\right]$ $$

### 3.2 Training Procedure

LayTextLLM is trained with innovative layout-aware training procedure, which consists of two stages: Layout-aware Next Token Prediction pre-training and Shuffled-OCR Supervised Fine-tuning.

#### 3.2.1 Layout-aware Next Token Prediction

Figure: Figure 4: Comparison of Layout-aware Next Token Prediction and normal Next Token Prediction.
Refer to caption: x4.png

Inspired by the next token prediction commonly used in current LLM pre-training [, , , , , , ], we propose the Layout-aware Next Token Prediction (LNTP). Fig. 4 presents the contrast of the proposed Layout-aware Next Token Prediction and the conventional next token prediction task. The traditional next token prediction (Fig. 4(a)) relies solely on the textual content, predicting each subsequent token based on the prior sequence of tokens without considering their spatial layouts. Layout-aware next token prediction (Fig. 4(b)), however, interleaves the spatial information encoded by SLP (i.e., $b^{i}$) with the text tokens (i.e., $t^{i}$). This integration considers both the content and its layout within the document, leading to a richer, more precise understanding of both the structure and the content.

Figure: Figure 5: Receipt layout example.
Refer to caption: x5.png

Similarly, primary objective of LNTP is to maximize the likelihood of its predictions for the next token. Thus the loss function is defined as

$$ $\mathcal{L}=-\frac{1}{T}\sum_{i=1}^{T}\log P\left(t^{i}\mid t^{1},t^{2},\ldots,t^{i-1}\right)$ (2) $$

where $P\left(t^{i}\mid t^{1},t^{2},\ldots,t^{i-1}\right)$ represents the probability of $i^{th}$ token $t^{i}$ given the sequence of preceding tokens $t^{1},t^{2},\ldots,t^{i-1}$, as predicted by the
model. Note that we compute the loss only for text tokens, excluding bounding box tokens. During pre-training, our goal is to enhance the alignment between spatial layouts and textual modality, while preserving the LLM’s inherent knowledge as much as possible. Thus, we freeze the LLMs and only update the parameters of SLP and P-LoRA.

It is important to note that the proposed Layout-aware Next Token Prediction is a completely self-supervised pre-training procedure, unlike previous works that require human annotations of document structure data or synthetic data generated by larger LLMs such as GPT-4 []. Thus, LNTP facilitates the creation of large-scale, high-fidelity pre-training datasets at minimal cost.

#### 3.2.2 Shuffled-OCR Supervised Fine-tuning

OCR engines typically process text from top to bottom and left to right. This order is also adopted as the input sequence for current OCR-based LLMs [, ]. However, modern LLMs often exhibit a strong inductive bias toward the positions of input tokens, influenced by designs such as Rotary Position Embeddings (RoPE) []. Specifically, tokens that are close together in the input sequence are likely to receive higher attention scores, which is advantageous for processing standard text sequences. Such inductive bias brings cons and pros.

Consider the example illustrated in Fig. 5, where the OCR input text reads: “ … Change, 1.30, GST%, Amt(RM), GST(RM), Total(RM), SR, 6, 17.64, 1.06, 18.70 … ”. If the question posed is “What is the value of the field Change?” (highlighted in a blue box), the model easily identifies “1.30” as it is closely positioned to the word “Change” in the sequence. However, for a more challenging query like “What is the value of the field Total(RM)?” (highlighted in a red box), the model struggles to determine the correct answer due to the presence of multiple subsequent numbers closed to “Total(RM)”. LayTextLLM integrates spatial layouts with textual data, reducing reliance on input sequence order. Thus, we posit that shuffling the OCR input order could enhance the resilience of LayTextLLM in discerning relevant information irrespective of token proximity in the sequence.

Specifically, we propose Shuffled-OCR Supervised Fine-tuning (SSFT) that randomly shuffles the order of OCR-derived text in a certain proportion of examples. The range of exploration for the shuffling ratio can be found in Tab. 7 and 20% shuffled ratio is applied. The training objective is equivalent to predicting the next tokens, but in this scenario, only the tokens of the response are used to compute loss. During SSFT, we unfreeze all parameters including those of LLMs. Experimental results in Section 4.6 demonstrate that utilizing SSFT can further enhance model performance, making it more robust to disruptions in input token order.

## 4 Experiments

### 4.1 Datasets

##### Pre-training data

In our training process, we exclusively use open-source data to facilitate replication. We collect data from two datasets for pre-training: (1) IIT-CDIP Test Collection 1.0 [] and (2) DocBank []. The IIT-CDIP Test Collection 1.0 comprises an extensive repository of more than 16 million document pages. DocBank consists of 500K documents, each presenting distinct layouts with a single page per document. For training efficiency, we choose to utilize the entire DocBank dataset and only subsample 5 million pages from the IIT-CDIP collection 1.0.

##### SFT data

For document-oriented VQA, we select Document Dense Description (DDD) and Layout-aware SFT data used in Luo et al. [], which are two synthetic datasets generated by GPT-4. Besides, DocVQA [], InfoVQA [], ChartQA [], VisualMRC [] is included following []. For KIE task, we select SROIE [], CORD [], FUNSD [], POIE [] datasets following [, , ].

### 4.2 Implementation Detail

The LLM component of LayTextLLM is initialized from the Llama2-7B-base [], which is a widely-used backbone. Other parameters including SLP and P-LoRA are randomly initialized. During pre-training, the LLM is frozen, and the parameters of SLP and P-LoRA modules are updated. During SFT, all parameters are fine-tuned. Other detailed setup can be found in Appendix B.

We have configured the model with three versions of LayTextLLM for a side-by-side comparison under different settings. Aligned with Luo et al. [], the first version, LayTextLLMzero, is trained exclusively with DDD and Layout-aware SFT data. Building upon this, and in alignment with the setting of Liu et al. [], we introduce the DocVQA and InfoVQA training sets to the dataset pool for our second version, termed LayTextLLMvqa. Finally, we incorporate a comprehensive suite of KIE datasets—FUNSD, CORD, POIE, SROIE, ChartQA, and VisualMRC—as described by Wang et al. [], creating our most extensive version, LayTextLLMall. Note that all versions are based on the same pre-trained LayTextLLM weight.

### 4.3 Baselines

OCR-free baselines In the category of OCR-free MLLMs, we have chosen the following SOTA models as our strong baselines due to their superior performance in both document-oriented VQA and KIE tasks. These include UniDoc [], DocPedia [], Monkey [], InternVL [], InternLM-XComposer2 [], TextMonkey, and TextMonkey+ [].

OCR-based baselines For OCR-based baseline models, we implemented a basic approach using only OCR-derived text as input. This was done using two versions: Llama2-7B-base and Llama2-7B-chat. We also adapted the coordinate-as-tokens scheme from He et al. [] for these models, resulting in two new variants: Llama2-7B-basecoor and Llama2-7B-chatcoor. It’s important to note that we did not employ the ICL strategy with these models, as it would significantly exceed their maximum sequence length constraints. Additionally, we included results from a stronger baseline using the ChatGPT Davinci-003 (175B) model [], termed Davinci-003-175Bcoor. One other recent SOTA OCR-based approach, DocLLM [] is also considered in our analysis. Finally, LayoutLLM and LayoutLLMCoT [], which integrates visual cues, text and layout is also included.

### 4.4 Evaluation Metrics

To ensure a fair comparison with OCR-free methods, we adopted the accuracy metric, where a response from the model is considered correct if it fully captures the ground truth. This approach aligns with the evaluation criteria described by [, , ]. To further enhance the comparability with other OCR-based methods, we conducted additional evaluations using original metrics specific to certain datasets, such as F1 score [, ], ANLS [, , ] and CIDEr [, ].

**Table 1: Comparison with SOTA OCR-free MLLMs. ^∗ indicates the training set used.**
|  | Document-Oriented VQA | KIE |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | DocVQA | InfoVQA | Avg | FUNSD | SROIE | POIE | Avg |
| Metric | Accuracy % |  |  |  |  |  |  |
| OCR-free |  |  |  |  |  |  |  |
| UniDoc [] | 7.7 | 14.7 | 11.2 | 1.0 | 2.9 | 5.1 | 3.0 |
| DocPedia [] | 47.1^∗ | 15.2^∗ | 31.2 | 29.9 | 21.4 | 39.9 | 30.4 |
| Monkey [] | 50.1^∗ | 25.8^∗ | 38.0 | 24.1 | 41.9 | 19.9 | 28.6 |
| InternVL [] | 28.7^∗ | 23.6^∗ | 26.2 | 6.5 | 26.4 | 25.9 | 19.6 |
| InternLM-XComposer2 [] | 39.7 | 28.6 | 34.2 | 15.3 | 34.2 | 49.3 | 32.9 |
| TextMonkey [] | 64.3^∗ | 28.2^∗ | 46.3 | 32.3 | 47.0 | 27.9 | 35.7 |
| TextMonkey+ [] | 66.7^∗ | 28.6^∗ | 47.7 | 42.9 | 46.2 | 32.0 | 40.4 |
| text + polys |  |  |  |  |  |  |  |
| LayTextLLMzero (Ours) | 72.1 | 35.7 | 53.9 | 47.5 | 86.4 | 68.9 | 67.6 |
| LayTextLLMvqa (Ours) | 77.2^∗ | 42.1^∗ | 59.7 | 48.8 | 75.7 | 70.6 | 65.0 |

**Table 2: Comparison with other OCR-based methods. ^∗ indicates the training set used.**
|  | Document-Oriented VQA | KIE |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | DocVQA | VisualMRC | Avg | FUNSD | CORD | SROIE | Avg |
| Metric | ANLS % / CIDEr | F-score % |  |  |  |  |  |
| Text |  |  |  |  |  |  |  |
| Llama2-7B-base | 34.0 | 182.7 | 108.3 | 25.6 | 51.9 | 43.4 | 40.3 |
| Llama2-7B-chat | 20.5 | 6.3 | 13.4 | 23.4 | 51.8 | 58.6 | 44.6 |
| Text + Polys |  |  |  |  |  |  |  |
| Llama2-7B-basecoor [] | 8.4 | 3.8 | 6.1 | 6.0 | 46.4 | 34.7 | 29.0 |
| Llama2-7B-chatcoor [] | 12.3 | 28.0 | 20.1 | 14.4 | 38.1 | 50.6 | 34.3 |
| Davinci-003-175Bcoor [] | - | - | - | - | 92.6 | 95.8 | - |
| DocLLM [] | 69.5^∗ | 264.1^∗ | 166.8 | 51.8^∗ | 67.6^∗ | 91.9^∗ | 70.3 |
| LayTextLLMzero (Ours) | 65.5 | 200.2 | 132.9 | 47.2 | 77.2 | 83.7 | 69.4 |
| LayTextLLMvqa (Ours) | 75.6^∗ | 179.5 | 127.6 | 52.6 | 70.7 | 79.3 | 67.5 |
| LayTextLLMall (Ours) | 77.2^∗ | 277.8^∗ | 177.6 | 64.0^∗ | 96.5^∗ | 95.8^∗ | 85.4 |

**Table 3: Comparison with LayoutLLM. ^- indicates that the cleaned test set used in Luo et al. [].**
|  | Document-Oriented VQA | KIE |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | DocVQA | VisualMRC | Avg | FUNSD^- | CORD^- | SROIE^- | Avg |
| Metric | ANLS % |  |  |  |  |  |  |
| Visual + Text + Polys |  |  |  |  |  |  |  |
| LayoutLLM [] | 72.3 | - | - | 74.0 | - | - | - |
| LayoutLLMCoT [] | 74.2 | 55.7 | 64.9 | 79.9 | 63.1 | 72.1 | 71.7 |
| Text |  |  |  |  |  |  |  |
| Llama2-7B-base | 34.0 | 25.4 | 29.7 | 42.1 | 46.7 | 60.6 | 49.8 |
| Llama2-7B-chat | 20.5 | 9.9 | 15.2 | 15.1 | 20.0 | 35.6 | 23.5 |
| Text + Polys |  |  |  |  |  |  |  |
| Llama2-7B-basecoor [] | 8.4 | 6.7 | 7.5 | 4.3 | 33.0 | 47.2 | 28.1 |
| Llama2-7B-chatcoor [] | 12.3 | 12.2 | 12.2 | 11.9 | 6.4 | 39.4 | 19.2 |
| LayTextLLMzero (Ours) | 65.5 | 37.4 | 51.5 | 72.0 | 45.5 | 82.0 | 66.5 |
| LayTextLLMall (Ours) | 77.2^∗ | 41.7^∗ | 59.5 | 81.0^∗ | 82.5^∗ | 96.1^∗ | 86.5 |

### 4.5 Quantitative Results

##### Comparison with SOTA OCR-free Methods

The experimental results shown in Tab. 1 demonstrate the outstanding performance of the LayTextLLM series across various tasks. Note that the results for ChartQA are reported in Appendix E due to concerns about fairness in comparison, as the dataset does not include OCR-derived results and we used in-house OCR tools instead. Firstly, LayTextLLMzero significantly outperforms previous SOTA OCR-free methods, such as TextMonkey [], in zero-shot capabilities, even when these methods use the training set of the dataset. For example, in the DocVQA and InfoVQA datasets, LayTextLLMzero achieves accuracies of 72.1% and 35.7%, respectively, which are markedly higher than existing OCR-free methods such as TextMonkey and InternLM-XComposer2. When fine-tuned with corresponding datasets, LayTextLLM shows even greater performance improvements, particularly in document-oriented VQA datasets. Specifically, its accuracies on DocVQA and InfoVQA increase to 77.2% and 42.1%, respectively, demonstrating the model’s strong ability to leverage task-specific data. Additionally, LayTextLLMzero excels in KIE datasets, particularly on the SROIE and POIE datasets, achieving accuracies of 86.4% and 68.9%, respectively. These results significantly surpass those of previous SOTA OCR-free model (i.e., TextMonkey+) by margins of 40.5% and 34.1%, respectively. This significant performance gain is likely due to these datasets containing low-resolution images that are too blurred for current MLLMs to extract visual features, whereas LayTextLLM shows robustness in such challenging scenarios.

##### Comparison with SOTA OCR-based Methods

For comprehensive comparison, we have also conducted correspinding experiments to align with OCR-based methods [, ]. The experimental results presented in Tab. 2 showcase significant performance improvements achieved by LayTextLLM models compared to pure OCR-based SOTA methods such as DocLLM []. Specifically, when comparing with DocLLM, LayTextLLMzero demonstrates notably superior performance, with even its zero-shot capabilities being competitive with supervised SFT approaches. We believe that the subpar performance of DocLLM is likely due to its use of cross-attention and the masked span pre-training tasks [], which fail to leverage the autoregressive features of LLMs effectively. Similarly, when contrasting with coordinate-as-tokens employed in Llama2-7B, LayTextLLMzero again outperforms significantly. This disparity in performance can be attributed to the following three reasons: (1) The coordinate-as-tokens approach tends to introduce an excessive number of tokens, often exceeding the pre-defined maximum length of Llama2-7B (i.e., 4096). Consequently, this leads to a lack of crucial OCR information, resulting in hallucination and subpar performance. (2) When re-implementing the coordinate-as-tokens method with Llama2-7B, we did not introduce the ICL strategy, as it would contribute additional length to the input sequence. (3) The coordinate-as-tokens approach necessitates a considerably larger-sized LLM to comprehend the numerical tokens effectively.

In comparison to LayoutLLM [], our approach exhibits discrepant performance in different tasks, as shown in Tab. 3. In zero-shot scenarios, we outperform LayoutLLM in most KIE datasets, validating our capability to leverage OCR-based results effectively. However, we fall short on document-oriented VQA tasks since answering some questions that are strongly related to vision information may challenge our approach. Two main reasons may well explain this performance discrepancy: (1) The visual encoder in LayoutLLM provides additional visual information. (2) LayoutLLM incorporates the Chain-of-Thought (CoT) mechanism to model contextual information while it is not used in our approach. However, when fine-tuned with tailored data, LayTextLLM significantly outperforms LayoutLLM, showcasing its strong ability to utilize task-specific data. More qualitative example demonstrates can be found in Appendix A.

### 4.6 Analysis

**Table 4: Ablations on each component of LayTextLLM (Accuracy).**
|  |  |  | Document-Oriented VQA | KIE |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SLP | P-LoRA | LNTP+SSFT | DocVQA | InfoVQA | VisualMRC | Avg | FUNSD | CORD | SROIE | POIE | Avg |
|  |  |  | 71.5 | 31.9 | 31.1 | 44.8 | 50.5 | 90.2 | 91.6 | 54.1 | 71.6 |
| ✓ |  |  | 74.7 | 35.7 | 32.5 | 47.6 | 55.1 | 94.9 | 94.6 | 68.3 | 78.2 |
| ✓ | ✓ |  | 76.5 | 38.0 | 30.6 | 48.4 | 54.3 | 95.9 | 95.3 | 70.6 | 79.0 |
| ✓ | ✓ | ✓ | 78.8 | 42.7 | 34.4 | 52.0 | 63.0 | 95.9 | 95.2 | 62.1 | 79.1 |

Ablations To better assess the utility of each component in LayTextLLM, an ablation study was conducted, the results of which are presented in Tab. 4. Detailed information on the training setup for all variants is provided in Appendix B. The results clearly show that incorporating interleaved spatial layouts and texts significantly enhances the performance, evidenced by a 2.8% improvement in VQA and a 6.6% increase in KIE compared to the plain version, indicating that SLP is a critical component. Furthermore, enabling P-LoRA results in a modest performance increase of 0.8% in both VQA and KIE tasks. Finally, the enabling of LNTP+SSFT leads to a significant improvement in VQA and KIE tasks, with an increase of 3.6% and 0.1%.

Sequence Length Tab. 5 presents statistics on the average input sequence length across different datasets. Intriguingly, despite interleaving bounding box tokens, LayTextLLM consistently exhibits the shortest sequence length in three out of four datasets, even surpassing DocLLM, which is counterintuitive. We attribute this to the tokenizer mechanism. For example, using tokenizer.encode(), a single word from the OCR engine, like “International” is encoded into a single ID $[4623]$. Conversely, when the entire OCR output is processed as one sequence, such as “… CPC,International,Inc…”, the word “International” is split into two IDs $[17579,1288]$, corresponding to “Intern” and “ational” respectively. This type of case occurs frequently, more discussion in Appendix C. Moreover, we found that the spatial layout can be decoded back into coordinates, and inference requiring output bounding box coordinates can further alleviate the hallucination problem, more discussion in Appendix F.

**Table 5: Average sequence length of each data for different methods using Llama2 tokenizer.**
| Dataset | LayTextLLM | DocLLM [] | Coor-as-tokens [] |
| --- | --- | --- | --- |
| DocVQA | 664.3 | 827.5 | 4085.7 |
| CORD | 137.9 | 153.2 | 607.3 |
| FUNSD | 701.9 | 847.5 | 4183.4 |
| SROIE | 529.2 | 505.1 | 1357.7 |

## 5 Limitation

Although LayTextLLM has shown significant capabilities in text-rich VQA and KIE tasks, this alone does not suffice for all real-world applications. There are some instances, particularly in chart analysis, where reasoning must be based solely on visual cues (e.g. size, color)—a challenge that remains unmet. Questions such as “What is the difference between the highest and the lowest green bar?” illustrate this gap. The ChartQA results, detailed in Appendix E, also underscore these limitations. Addressing these challenges highlights the urgent need for future enhancements that integrate visual cue within the capabilities of LayTextLLM.

## 6 Conclusion

We propose LayTextLLM for various VRDU tasks, in which spatial layouts and textual data are seamlessly interleaved to make more accurate prediction by introducing a innovative Spatial Layout Projector. Two tailored training tasks — Layout-aware Next Token Prediction and Shuffled-OCR Supervised Fine-tuning — are designed to improve the comprehension of document layouts. Extensive experiments confirm the effectiveness of LayTextLLM.

## Appendix

## Appendix A Qualitative Examples

Qualitative examples of document-oriented VQA (upper row) and KIE (bottom row) are shown in Fig. 6. The results indicate that LayTextLLM is highly effective in utilizing spatial layout information to make more accurate predictions for these challenging examples. For example, in the upper right figure, many numeric texts in the receipt act as noise for the baseline method. In contrast, LayTextLLM integrates layout information to accurately predict the total price, as demonstrated by the other examples, underscoring the utility of LayTextLLM.

Figure: Figure 6: Qualitative comparison with the baseline method.
Refer to caption: x6.png

## Appendix B Implementation Detail

All training and inference procedures are conducted on eight NVIDIA A100 GPUs.

##### Training

LayTextLLM is initialized with Llama2-7B-Base model, the pre-training, SFT, and other model hyper-parameters can be seen in Tab. 6. Please note that all variants of LayTextLLM, including those utilized in ablation studies, are trained in accordance with the SFT settings. All baseline results are sourced from their respective original papers, with the exception of the Llama2-7B series and the Llama2-7B${}_{\text{coor}}$ series. These were re-implemented and can be referenced in [, ].

**Table 6: LayTextLLM trainng Hyper-parameters.**
|  | Backbone | Plora rank | Batch size | Max length | Precision | Train params | Fix params |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pretrain | Llama2-7B-base | 256 | 128 | 2048 | bf16 | 648 M | 6.7 B |  |
| SFT | Llama2-7B-base | 256 | 256 | 4096 | bf16 | 7.4 B | 0B |  |
|  | Learning rate | Weight decay | Scheduler | Adam betas | Adam epsilon | Warm up | Epoch |  |
| Pretrain | 1.0e-04 | 0.01 | cosine | [0.9, 0.999] | 1.0e-08 | 0.005 | 2 |  |
| SFT | 2.0e-05 | 0.01 | cosine | [0.9, 0.999] | 1.0e-08 | 0.005 | 2 |  |

##### Inference

For the document-oriented VQA test set, we use the original question-answer pairs as the prompt and ground truth, respectively. For Key Information Extraction (KIE) tasks, we reformat the key-value pairs into a question-answer format, as described in [, , ]. Additionally, for the FUNSD dataset, we focus our testing on the entity linking annotations as described in [].

To eliminate the impact of randomness on evaluation, no sampling methods are employed during testing for any of the models. Instead, beam search with a beam size of 1 is used for generation across all models. Additionally, the maximum number of new tokens is set to 512, while the maximum number of input tokens is set to 4096.

## Appendix C Discussion of Input Sequence Length

As mentioned in Section 4.6, it is intriguing that LayTextLLM has fewer input sequences than DocLLM, which is counterintuitive given that LayTextLLM interleaves bounding box tokens, typically resulting in longer sequence lengths. We attribute this to the Byte Pair Encoding (BPE) tokenizers [] prevalently used in modern LLMs such as Llama2.

BPE operates by building a vocabulary of commonly occurring subwords (or token pieces) derived from the training data. Initially, it tokenizes the text at the character level and then progressively merges the most frequent adjacent pairs of characters or sequences. The objective is to strike a balance between minimizing vocabulary size and maximizing encoding efficiency.

Thus, when tokenizing a single word like “International” on its own, the tokenizer might identify it as a common sequence in the training data and encode it as a single token. This is especially likely if “International” frequently appears as a standalone word in the training contexts. However, when the word “International” is part of a larger sequence of words such as including in a long sequence of OCR-derived texts like “…335
CPC,International,Inc…”, the context changes. The tokenizer might split “International” into sub-tokens like “Intern” and “ational” because, in various contexts within the training data, these subwords might appear more frequently in different combinations or are more useful for the model to understand variations in meaning or syntax.

When using LayTextLLM, we input word-level OCR results into the tokenizer, typically resulting in the former situation, where words are encoded as single tokens. Conversely, with DocLLM, the entire OCR output is processed as one large sequence, leading to the latter situation and a longer sequence length than in LayTextLLM. This difference underscores the utility of LayTextLLM in achieving both accuracy and inference efficiency due to its shorter sequence length.

## Appendix D Shuffle Ratio Exploration

Tab. 7 presents the results of exploring training and testing shuffling ratios on the FUNSD dataset using two different models: Llama2-7B-base and LayTextLLM. The table shows the performance of these models at various shuffling ratios (100%, 50%, 20%, and 0%).

LayTextLLM consistently outperforms Llama2-7B-base across all levels of shuffling, which further underscores the significance of interleaving spatial layouts with text. Particularly at the 100% shuffle level, Llama2-7B-base demonstrates limited accuracy at only 20.3, while LayTextLLM maintains a relatively higher performance. It is also interesting to note that Llama2-7B-base generally improves as the shuffling percentage decreases, whereas LayTextLLM performs best when 20% of the examples with OCR-derived text are shuffled. This observation suggests that LayTextLLM effectively utilizes spatial layouts and is less dependent on the sequence of input tokens. Therefore, a certain proportion of shuffled examples can serve as adversarial examples to enhance the model’s robustness, addressing situations such as errors in the text order from the OCR engine, which are caused by subtle differences in horizontal or vertical coordinates.

**Table 7: Shuffling ratio exploration in FUNSD dataset.**
|  | FUNSD |  |
| --- | --- | --- |
| Ratio | Llama2-7B-base | LayTextLLM |
| 100% | 20.3 | 44.7 |
| 50% | 49.1 | 62.1 |
| 20% | 50.2 | 65.4 |
| 0% | 52.3 | 65.1 |

## Appendix E Results of ChartQA

As shown in Fig. 7, the question-answer pairs in ChartQA [] tend to involve the visual cues for reasoning. However, with only text and layout information as input, the proposed LayTextLLM inevitably have difficulties in reasoning visual-related information. Thus, on the ChartQA dataset, LayTextLLM can hardly achieve better performance than previous methods that include visual inputs. Although the visual information is not used in LayTextLLM, it can still exhibit better zero-shot ability than UniDoc []. After incorporating the training set of ChartQA, the performance of LayTextLLM can be boosted to 35.4%. Considering the importance of visual cues in ChartQA-like tasks, we will try to involve the visual information into LayTextLLM in future work.

Figure: Figure 7: A failure case of LayTextLLM on ChartQA.
Refer to caption: x7.png

**Table 8: Comparison with SOTA OCR-free MLLMs on ChartQA. ^∗ indicates the training set used.**
|  | ChartQA |
| --- | --- |
| OCR-free |  |
| UniDoc [] | 10.9 |
| DocPedia [] | 46.9^∗ |
| Monkey [] | 54.0^∗ |
| InternVL [] | 45.6^∗ |
| InternLM-XComposer2 [] | 51.6^∗ |
| TextMonkey [] | 58.2^∗ |
| TextMonkey+ [] | 59.9^∗ |
| text + polys |  |
| LayTextLLMzero (Ours) | 22.8 |
| LayTextLLMvqa (Ours) | 23.4^∗ |
| LayTextLLMall (Ours) | 35.4^∗ |

## Appendix F Decoding Bounding Box Coordinates

We tested LayTextLLM on an in-house KIE dataset, requiring the model to output value texts along with their corresponding bounding boxes in a text format, such as "Oct 10[66,1,70,15]". We found that asking the model to output coordinates increased precision as shown in Tab. 9. This emphasizes the model’s need to search the input for corresponding information, which alleviates hallucination problems. Additionally, the model learns to combine and subtract coordinates. For example, if the output text is derived from two OCR input lines, the model will combine the corresponding OCR coordinates. Conversely, if the output is a substring of the input OCR text, the model will output the adjusted coordinates accordingly.

**Table 9: Outputting coordinates vs. not outputting coordinates**
| Model | Precision | Recall | F-score |
| --- | --- | --- | --- |
| Output w coor | 82.07 | 75.41 | 78.05 |
| output w/o coor | 83.48 | 74.07 | 77.69 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: (a) Are the reported numbers for LayoutLLM_CoT in Table 3 inaccurate? The current numbers seem to correspond to Vicuna 1.5 7B. Would it be more appropriate to report the results based on Llama2-7B for a fairer comparison?

(b) Why does LayTextLLM show significantly inferior performance on the ChartQA dataset, even compared to OCR-free methods?
soundness: 2
strengths: - LayTextLLM demonstrate that mapping the coordinates to a single bbox embedding can achieve delightful performance on document understanding while mitigating long sequence issues.
- Table 1 and Table show that LayTextLLM demonstrate performance improvement over several baseline methods on DocVQA, InfoVQA, and KIE tasks.
summary: This paper presents LayTexLLM, which interleaves layout and text in an LLM for document understanding. There are two key ideas:
- Spatial layout projector (SLP), which project the coordinate of each bbox (_i.e._, [x_min, y_min, x_max, y_max]) derived by an off-the-shelf OCR module into a $d$-dimensional embedding, so that the layout information can be input to the LLM and learned together with other text tokens via next token prediction (layout-aware).
- Shuffled-OCR supervised fine-tuning (SSFT), which randomly shuffles the order of OCR-derived text in a certain proportion of examples to mitigate the bias that OCR engines typically output text bboxes in a way of top-to-bottom and left-to-right. 
Experiments on DocVQA, InfoVQA, and KIE tasks show that LayTextLLM outperforms or achieves comparable performance to baselines, while experiencing a significant performance drop on the ChartQA dataset.
weaknesses: 1. The performance of the model heavily relies on the quality of the OCR engine, which poses a risk of significant performance drop, especially for handwritten documents or low-resource languages where OCR performance can be subpar. Moreover, incorporating an OCR module can negatively impact inference latency, making real-time or high-throughput applications challenging.

2. As indicated in Table 3, LayTextLLM_zero and even LayTextLLM_all demonstrate inferior performance on Document-Oriented VQA compared to LayoutLLM (Luo et al., 2024). Additionally, Table 8 reveals that LayTextLLM shows a substantial drop in performance on the ChartQA dataset. Specifically, OCR-free methods generally achieve scores between 50-59.9, whereas the best performance achieved by LayTextLLM_all is only 35.4.

3. The ablation study requires a clearer description of the variants involved. The first line in Table 4 lacks clarity—does it refer to a setting that uses naive coordinate-based pretraining (e.g., normalized [x1, y1, x2, y2]) with the proposed SSFT strategy? If not, it would be beneficial to include such a variant to demonstrate the effectiveness of the Spatial Layout Projector.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: See weakness.
soundness: 2
strengths: - The empriical performance shows a imporlvement on downstream document understanding tasks
summary: The paper use one token (generated by a small NNs called SLP) to replace the four coordinate representation of location for normal document understanding tasks. With such compression, the model can accept more tokens and longer context than a vanilla llm. The method show better results than some document understanding MLLMs.
weaknesses: Overall, the paper propose an emprical trick to compress four localization tokens into one token, however, the contribution is thin and the research problem is not valid to me (without futher considering the visual features). 


(updated in Nov 20)

- **Research Problem Validity**:
The core research problem studied in this paper appears incomplete without incorporating visual features. This concern is supported by two key pieces of evidence:
a) The evolution of document understanding models shows that pure OCR-text-only approaches (like LayoutLM[1]) have been quickly superseded by multimodal approaches (LayoutLMv2, LayoutLMv3[2,3]), demonstrating the crucial importance of incorporating visual information in document understanding tasks.
b) Recent findings from MMMU-PRO [4] demonstrate that even with PERFECT text information, the VQA task accuracy only improves by a marginal 0.5%. This empirically proves that text-only approaches are **insufficient*** for comprehensive document understanding, regardless of text quality.

- **Scope and Contribution**:
While I acknowledge the value of theoretical work, this paper reads more like an appendix to comprehensive document understanding rather than a standalone contribution. The presented approach seems to address only a narrow subset of document understanding challenges, without adequately justifying why visual features are excluded, especially given the strong empirical evidence supporting the necessity of visual information.

- **Broader Impact and Audience**:
For the broad ICLR audience, the paper offers limited insights and contributions. The work might be more suitable for venues specifically focused on document understanding, where the technical details of OCR processing would find a more relevant audience. Moreover, given the clear trend towards multimodal approaches in the field, this text-only approach may be of limited practical value to the research community.



[1] LayoutLM: Pre-training of Text and Layout for
Document Image Understanding

[2] LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding

[3] LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking 

[4] MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: refer to weakness
soundness: 3
strengths: 1. The paper proposes the adoption of a unified embedding method in LLM for the first time, interleaving the spatial layout with the text data.
2. By reducing the sequence length and making full use of the autoregressive characteristics, the performance of document understanding is significantly improved.
3. Comprehensive experimental results and ablation studies are provided, demonstrating the effectiveness of the proposed method.
summary: Traditional methods for document understanding usually rely on the text extracted by OCR and spatial layout information, but they have limitations when dealing with long text sequences. This paper proposes a method named LayTextLLM, which reduces the sequence length problem by mapping each bounding box to a single embedding and interleaving it with the text, while taking advantage of the autoregressive characteristics of LLM. Two training tasks are proposed: layout-aware next token prediction and supervised fine-tuning of shuffled OCR, to enhance the generality of the model. In Key Information Extraction (KIE) and Visual Question Answering (VQA) tasks, LayTextLLM significantly outperforms the existing state-of-the-art methods. In the zero-shot scenario, the performance of the KIE task is improved by 27.0% and that of the VQA task is improved by 19.8%.
weaknesses: Compared with the existing VLM papers, the paradigm is actually quite similar. Although it is not highly innovative, it offers a new perspective for handling OCR document understanding problems in this field. Another point is that perhaps the number of baselines for comparison seems a bit small. I wonder if there are more and newer works related to Text + Coordinates for comparison.
Additionally, there is a slight error in the bolded text in Table 1.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.The different training tasks such as Layout-aware Next Token Prediction mentioned，how many tokens of training data were used respectively?
2.Why does the chosen lora structure work as expected? Do other loras have the same effect, and how are ablation trials conducted?
3.Is there any analysis examining the impact on the llama models' ability to understand language after incorporating spatial information?
soundness: 3
strengths: 1.This work is the first work to employ a unified embedding approach that interleaves spatial layouts directly with textual data within a LLM.
2.They propose two tailored training tasks: (1) Layout-aware Next Token Prediction  and (2) Shuffled-OCR Supervised Fine-tuning task to better elicit the model generalizability in downstream tasks.
3.Experimental results sounds good.
summary: This paper introduce Interleaving Layout and Text in a Large Language Model (LayTextLLM) for document understanding, which projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. They also propose a tailored pre-training task—Layout-aware Next Token Prediction—a completely self-supervised task that enhances the alignment between layout and textual . LayTextLLM significantly improves performance on downstream document-related VQA and KIE tasks.
weaknesses: 1.The methods adopt a P-LoRA structure, which is introduced by existing work InternLM-XCompoer2, limiting the novelty of this work‘s.
2.The analysis of the impact of SLP training on the effectiveness of the model remains incomplete.

Meta-review: This paper introduces LayTextLLM, a method for incorporating document layout into LLMs. It uses: (1) a Spatial Layout Projector (SLP) to embed bounding box coordinates from OCR output, enabling layout-aware next token prediction; and (2) Shuffled-OCR supervised fine-tuning (SSFT) to reduce OCR-induced positional bias

The central contribution of this work is the SLP module, which effectively integrates layout and text. However, as noted in the related work section, Perot et al. (2023) use formats such as “HARRISBURG 78|09” to represent OCR text and corresponding layout, while He et al. (2023) use “[x_min, y_min, x_max, y_max]” to encode layout information. It is unclear whether fine-tuning with LoRA using these existing formats would yield comparable performance. If so, the primary advantage of the proposed method would be token savings achieved through coordinate embedding. Reviewer ZrGX raised a similar concern.

Decision: Reject