DocLLM: A Layout-Aware Generative Language Model for Multimodal Document Understanding (2024)

Abstract: Enterprise documents such as forms, receipts, reports, and other such records, often carry rich semantics at the intersection of textual and spatial modalities. The visual cues offered by their complex layouts play a crucial role in comprehending these documents effectively. In this paper, we present DocLLM, a lightweight extension to traditional large language models (LLMs) for reasoning over visual documents, taking into account both textual semantics and spatial layout. Our model differs from existing multimodal LLMs by avoiding expensive image encoders and focuses exclusively on bounding box information to incorporate the spatial layout structure. Specifically, the cross-alignment between text and spatial modalities is captured by decomposing the attention mechanism in classical transformers to a set of disentangled matrices. Furthermore, we devise a pre-training objective that learns to infill text segments. This approach allows us to address irregular layouts and heterogeneous content frequently encountered in visual documents. The pre-trained model is fine-tuned using a large-scale instruction dataset, covering four core document intelligence tasks. We demonstrate that our solution outperforms SotA LLMs on 14 out of 16 datasets across all tasks, and generalizes well to 4 out of 5 previously unseen datasets.

heading: 1 Introduction
text: Documents with rich layouts, including invoices, contracts, and forms, constitute a significant portion of enterprise corpora, and the automatic analysis of these documents offer considerable advantages (Kunduru, 2023). Although Document AI (DocAI) has made tremendous progress, there remains a significant performance gap in real-world applications due to the complex layouts, bespoke type-setting and template diversity exhibited by
*Equal Contribution.
these visually rich documents. In particular, accuracy, reliability, contextual understanding and generalization to previously unseen domains continues to be a challenge (Cui et al., 2021).
Conventional large language models (LLMs) such as GPT-3.5 (Brown et al., 2020), Llama (Touvron et al., 2023) or Falcon (Penedo et al., 2023) primarily accept text-only inputs and assume that the documents exhibit simple layouts and uniform formatting. They are not suitable for document intelligence tasks, which are inherently multi-modal, requiring the understanding of both text content and visual layout cues. Numerous vision-language frameworks (Li et al., 2022; Huang et al., 2022) that can process documents as images and capture the interactions between textual and visual modalities do exist. However, these frameworks necessitate the use of complex vision backbone architectures (Dosovitskiy et al., 2021) to encode image information, and often make use of spatial information as an auxiliary contextual signal (Xu et al., 2021; Lee et al., 2022).
In this paper, we present DocLLM, a lightweight extension to standard LLMs that excels in several visually rich form understanding tasks. Unlike traditional LLMs, it models both spatial layouts and text semantics, and therefore is intrinsically multimodal. The spatial layout information is incorporated through bounding box coordinates of the text tokens obtained typically using optical character recognition (OCR), and does not rely on a complex vision encoder component. Consequently, our solution preserves the causal decoder architecture, introduces only a marginal increase in the model size, and has reduced processing times. We demonstrate that merely including the spatial layout structure is sufficient for various document intelligence tasks such as form understanding, table alignment and visual question answering.
Existing efforts to incorporate spatial layout information typically involve either concatenating
8529
spatial and textual embeddings (Tang et al., 2023) or summing the two (Xu et al., 2020). In contrast, we treat the spatial information as a distinct modality and compute its inter-dependency with the text modality in a disentangled manner (Meng et al., 2021). Specifically, we extend the self-attention mechanism of transformers to include new attention scores that capture cross-modal relationships. There is often a correlation between the content, position and size of the fields in a form and hence representing their alignments at various abstraction levels across the transformer layers can enhance document understanding.
Visual documents often feature heterogeneous content, irregular layouts, and disjointed text segments. A classical next token prediction in selfsupervised pre-training can be restrictive for these documents since the preceding tokens may not always be relevant due to the diverse arrangements of text. To tackle this issue, we propose two modifications to the pre-training objective: (a) adopting cohesive blocks of text that account for broader contexts, and (b) implementing an infilling approach by conditioning the prediction on both preceding and succeeding tokens. Due to these modifications, the model is better equipped to address misaligned text, contextual completions, intricate layouts, and mixed data types. Although text spans and infilling tasks have been studied before (Du et al., 2021), our solution is tailored for visual documents with an emphasis on semantically coherent blocks.
We tune DocLLM on instruction data curated from multiple datasets for several document intelligence tasks including Key Information Extraction (KIE), Natural Language Inference (NLI), Visual Question-Answering (VQA) and document classification (CLS). The modifications introduced by DocLLM enhance the performance of Llama2-7B
model by 15-60% in four of five datasets unseen during training.
Our contributions include: (1) A lightweight extension to LLMs designed for understanding visual documents. (2) A disentangled spatial attention mechanism that captures cross-alignment between text and layout modalities. (3) An infilling pre-training objective tailored to address irregular layouts effectively. (4) A large instruction tuning dataset (with OCR data) specially curated towards visual document intelligence tasks. (5) Comprehensive experiments and insights into the model behavior. Fig. 1 summarizes the framework.

heading: 2 Related Work
text: General Purpose Models. By treating a document as text content, many text based LLMs (OpenAI, 2023; Touvron et al., 2023; Anil et al., 2023) can be directly utilized for document intelligence tasks. Despite the remarkable capabilities provided by these LLMs, their lack of understanding of visual elements and layouts can be severely limiting in the DocAI context. Although multi-modal LLMs (Li et al., 2023; Zhu et al., 2023; Liu et al., 2023a; Wu et al., 2023; Ye et al., 2023c; Zhang et al., 2023a) that explicitly include image information can account for visual signals, they often struggle to recognize specific structures and patterns prevalent in enterprise documents (Liu et al., 2023c). Instead of relying on generalized training, a model tailored for visually rich document understanding (VRDU) tasks can gain a nuanced comprehension of the language, formats and data structures unique to these types of documents.
Document Understanding Models. Models such as LayoutLM (Xu et al., 2020), LAMPreT (Wu et al., 2021), Pix2Struct (Lee et al., 2023), and
UDOP (Tang et al., 2023) specifically cater towards document processing tasks. They can account for different modalities including text, image and layout information, and are trained to exploit both the structure and content of documents, often using a large corpora. However, these models require task-specific fine-tuning, may lack a flexible interface and cannot understand open-domain instructions. Recent efforts like mPLUG-DocOwl (Ye et al., 2023a) and UReader (Ye et al., 2023b) build on LLMs and perform DocAI-focused instruction tuning. We differ from these by avoiding expensive visual encoders.
Model Architecture. Disentangled attention mechanisms, where different signals are represented by independent vectors, have been studied before (He et al., 2020). While we use a similar construct, our spatial position based encodings are more complex and applied in a multimodal context. Learning to infill autoregressive language models has been explored in Bavarian et al. (2022), Shen et al. (2023), and Du et al. (2021). Although we share their goal of adding fill-in-the-middle (FIM) capability, we differ in the mechanism by integrating FIM into the visual document contexts and avoiding extremely short segments.

heading: 3 DocLLM Framework

heading: 3.1 Architecture Overview
text: DocLLM is constructed upon the foundation of an auto-regressive transformer language model (Touvron et al., 2023; Penedo et al., 2023) following a causal decoder structure. It integrates lightweight visual information by utilizing the spatial positions and dimensions of text tokens obtained using OCR. Instead of simply augmenting the text with bounding box information via additive positional encoding (Xu et al., 2021), separate vectors are used to represent these two distinct modalities and the self-attention mechanism of the transformer architecture is extended to compute their interdependencies in a disentangled manner. Furthermore, the traditional left-to-right next token prediction during self-supervised training is replaced by a block infilling objective that better leverages contextual information. See Figure 2 for an overview.

heading: 3.2 Disentangled Spatial Attention
text: Let x = (x1, ..., xi, ..., xT ) be an input sequence of length T , where xi is a text token. In classical transformers, using a learned embedding matrix
based on the text vocabulary and a learned set of parameters for the token position in the sequence, the input tokens are first encoded into hidden vectors H ∈ RT×d. A self-attention head then computes the attention scores between tokens i and j as:
Qt = HWt,q, Kt = HWt,k, Ati,j = Q t iK t j ⊺ (1) where Wq ∈ Rd×d and Wk ∈ Rd×d are projection matrices, and the superscript t indicates the text modality. The attention scores A ∈ RT×T along with another projection matrix Wv are further used to compute the hidden vectors H′, which are in turn used as inputs for a subsequent layer:
Vt = HWt,v, H′ = softmax( At√ d )Vt. (2)
In DocLLM, the input is represented as x = {(xi, bi)}Ti=1, where bi = (left, top, right, bottom) is the bounding box corresponding to xi. To capture the new modality (i.e. spatial information), we encode the bounding boxes into hidden vectors represented by S ∈ RT×d. We then decompose the attention matrix computation into four different scores, namely text-to-text, text-to-spatial, spatial-to-text and spatial-to-spatial. Formally, the new attention mechanism is calculated as:
Qs = SWs,q, Ks = SWs,k,
Ai,j = Q t iK t j ⊺ + λt,sQ t iK s j ⊺
+ λs,tQ s iK t j ⊺ + λs,sQ s iK s j ⊺,
(3)
where Ws,q ∈ Rd×d and Ws,k ∈ Rd×d are newly introduced projection matrices corresponding to the spatial modality, and λs are hyperparameters that control the relative importance of each score. The input hidden vectors for the next layer H′ are computed exactly as before. However, in contrast to equation (2), the newly calculated hidden vectors rely not only on the text semantics but also on the layout information of the text tokens.
It is important to mention that the hidden vectors S are reused across different layers, while each layer retains the flexibility to employ different projection matrices. We also note that the number of extra parameters required to encode the bounding box information is significantly lower compared to the overhead introduced by image based models (Li et al., 2022). By simply adding S to H similar to Xu et al. (2020), we could have avoided using Ws matrices altogether and further reduced
the number of parameters. However, it would have irreversibly coupled the layout information with the text semantics. In contrast, our disentangled representation of these modalities in the attention scores enables selective focus when appropriate (He et al., 2020), thereby providing an optimal balance between model size and effectiveness.

heading: 3.3 Pretraining
text: DocLLM is first pre-trained in a self-supervised fashion on a large number of unlabeled documents. Visual documents are often sparse and irregular, featuring isolated and disconnected text fragments. It is preferable to consider coarse segments of related tokens during pre-training rather than focusing on individual tokens. Hence we use the broader context provided by multiple tokens, referred as blocks,1 for better comprehension. Most OCR engines can provide block level information, which makes it feasible to identify coherent text blocks such as a heading or an address.2
Learning to infill text, where the prediction is conditioned on both prefix and suffix tokens rather than only preceding tokens, can be beneficial for document understanding. The infilling objectives enable contextually relevant completions, provide
1In Figure 2 “Name”, “John Doe” , and “Doctor” are all examples of blocks
2In order to avoid any leakage of useful information, the block information is only used during pre-training, and the model is unaware of the number of tokens in a masked block.
robustness to OCR noise or misaligned tokens, and can better handle relationships between various document fields. Hence we modify the standard pre-training objective to predict blocks of text given preceding and following text blocks. Inspired by (Du et al., 2021), we follow an autoregressive block infilling objective, where text blocks are randomly masked, and the masked blocks are shuffled and reconstructed in a sequential left-to-right fashion.
Formally, let c = {c1, ..., cK} be a set of text blocks that partitions an input sequence x into nonoverlapping contiguous tokens such that c1 ∪ ... ∪ cK = x and ck ∩ ck′ = ∅. Let z = {zm}Mm=1 be M ≪ K different text blocks randomly sampled from c, where each block zm = (zm,1, ..., zm,Nm) contains a consecutive series of tokens. Further, let x̃ be a corrupted version of x where the contiguous tokens corresponding to a sampled text block are replaced with a special mask token [M]. To facilitate the identification of the block to be filled during text generation, each input block is augmented with a special start token [S] while the output block includes an end token [E]. For instance, a block with tokens (x4, x5) becomes [M] in x̃, ([S], x4, x5) when conditioned upon, and is expected to generate (x4, x5, [E]) as output autoregressively.3 Let θ denote all the parameters of the transformer model, including the projection matrices discussed above. The following cross-entropy
3See Figure 2 for an illustration of these configurations.
loss is then minimized for the infilling objective
LIF(θ) = − M∑
m=1
Nm∑
j=1
log pθ(zm,j |x̃, z<m, zm,<j).
(4)

heading: 3.4 Instruction Tuning
text: Following recent work in the field of VRDU (Tang et al., 2023; Ye et al., 2023a,b) and prior work in NLP (Wei et al., 2022; Chung et al., 2022), we instruction-tune DocLLM on a variety of instructions curated from multiple DocAI datasets using templates. We employ a total of 16 datasets with their corresponding OCRs, spanning four DocAI tasks.
The diversity of supervised fine tuning (SFT) instructions is critical in helping zero-shot generalization (Wei et al., 2022; Chung et al., 2022; Ouyang et al., 2022). Thus, we diversify templates per task when possible, with each template asking a different question, and in some cases, expecting different types of answers. We re-use the templates introduced in Ye et al. (2023a,b) when applicable.
We create the templates following what we believe end users would generally ask about documents (see Table 1). For KIE and CLS, we hypothesize that (1) the extraction instructions can teach DocLLM to correlate names of keys in the prompts with document fields so as to retrieve values, (2) the internal classification instructions can help the model understand what intrinsically characterizes each key or document type, and (3) the multiple choice question (MCQ) instructions can teach the model to leverage its comprehension of key names included as choices in the prompt (resp. document type names) to classify extracted values (resp. entire documents). The templates are as follows:4 Visual Question Answering. A single template. Prompt Example: What is the deadline for scientific abstract submission for ACOG - 51st annual clinical meeting? Natural Language Inference. A single template. Prompt Example: "The UN commission on Korea include 2 Australians.", Yes or No? Key Information Extraction. Three templates corresponding to extraction, internal classification, and MCQ instructions. Example prompt for extraction: What is the value for the "charity number"?
4Examples are derived from DocVQA (Mathew et al., 2021), TabFact (Chen et al., 2020), KLC (Stanislawek et al., 2021), RVL-CDIP (Harley et al., 2015).
Document Classification. Two templates corresponding to internal classification and MCQ instructions. Example prompt for MCQ: What type of document is this? Possible answers: [budget, form, file folder, questionnaire].
See Appendix A.2 for further details.

heading: 4 Experiments

heading: 4.1 Datasets
text: Pre-training. We gather data for pre-training from two primary sources: (1) IIT-CDIP Test Collection 1.0 (Lewis et al., 2006) and (2) DocBank (Li et al., 2020). IIT-CDIP Test Collection 1.0 encompasses a vast repository of over 5 million documents, comprising more than 16 million document pages. This dataset is derived from documents related to legal proceedings against the tobacco industry during the 1990s. DocBank consists of 500K documents, each featuring distinct layouts and a single page per document. We obtain a collection of 16.7 million pages comprising a total of 3.8 billion tokens. See Table 6 in the Appendix for detailed statistics.
Instruction Tuning. To instruction-tune the model for the VQA task, we collect DocVQA (Mathew et al., 2021), WikiTableQuestions (WTQ) (Pasupat and Liang, 2015), VisualMRC (Tanaka et al., 2021), and DUDE (Landeghem et al., 2023). For NLI, we only include TabFact (Chen et al., 2020) in our instruction-tuning data mix, due to lack of additional DocAI NLI datasets available. For KIE, we gather Kleister Charity (KLC) (Stanislawek et al., 2021), CORD (Park et al., 2019), FUNSD (Jaume et al., 2019), DeepForm (Svetlichnaya, 2020), PWC (Kardas et al., 2020), SROIE (Huang et al., 2019), and VRDU ad-buy (Wang et al., 2023) (with random train-test splitting). Finally, we use RVL-CDIP (Harley et al., 2015) to build our CLS instruction-tuning data. We also downsample RVL-CDIP in the train split to avoid hindering the other datasets due to size. See Table 7 in the Appendix for detailed statistics.
To the above datasets, we add BuDDIE (Zmigrod et al., 2024), a collection of ∼1,600 business entity filings curated from state registration websites within the US. BuDDIE is annotated for three tasks – VQA, KIE, and CLS – and we therefore include it in the respective instruction-tuning collections.5
5The instruction-tuning data is available upon request at https://www.jpmorgan.com/technology/ artificial-intelligence/initiatives/datasets
Table 1: Prompt templates used for instruction-tuning (spatial tokens not included).
Task Template type Prompt template Expected response
VQA Extraction {document} {question} answer annotation
NLI MCQ {document} "{statement}", Yes or No? answer annotation
KIE
Extraction {document} What is the value for the "{key}"? Associated value annotation
MCQ {document} What is "{value}" in the document? Possible choices: {keys}. (where keys is a subset of all the key names in the dataset in random order) Associated key annotation
Internal classification {document} What is "{value}" in the document? Associated key annotation
CLS MCQ
{document} What type of document is this? Possible choices: {classes}. (where classes is a subset of all the classes in the dataset in random order)
class annotation
Internal classification {document} What type of document is this? class annotation

heading: 4.2 Evaluation Setup
text: Model Configuration. We train two variants of DocLLM: DocLLM-1B, which is based on the Falcon-1B architecture (Penedo et al., 2023), and DocLLM-7B, which is based on the Llama2-7B architecture (Touvron et al., 2023).6 The maximum sequence length is set to 1,024 for both these models during the entire training process. See Appendix B for a detailed discussion on the model configuration and training hyper-parameters.
Settings. We investigate two experimental settings: Same Datasets, Different Splits (SDDS): Following previous work (Lee et al., 2023; Davis et al., 2022; Kim et al., 2022; Tang et al., 2023; Ye et al.,
6Since LLaMA2 does not come with pre-trained weights at 1B parameters, we use the Falcon-1B architecture for the smaller version of DocLLM.
2023a,b), we first evaluate DocLLM on the unseen test split (or dev split when labeled test split is not publicly available) of each of the 16 datasets composing the instruction tuning data. The motivation behind this very typical setting is to check how DocLLM performs when tasks and domains supposedly stay the same from train to test.
Same Tasks, Different Datasets (STDD): Following (Wei et al., 2022; Chung et al., 2022; Dai et al., 2023; Zhang et al., 2023a), we also evaluate DocLLM on held-out datasets. More precisely, we instruction-tune the pretrained checkpoint of DocLLM on prompts from 11 of the 16 datasets considered in SDDS, then evaluate DocLLM on the test split of the remaining five datasets. The rationale behind this evaluation setting is to assess the performance of DocLLM when tasks are unchanged but domains and layouts differ from train to test.
We believe examining this setting in the DocAI field is relevant because industry use cases usually encountered in practice revolve around VQA, KIE, and CLS, while document characteristics tend to change more often in production. We specifically isolate DocVQA, KLC, and BuDDIE for STDD evaluation in order to (1) exclude at least one dataset per task from SFT when possible, (2) leave enough datapoints per task in the training split of the instruction-tuning data, (3) avoid data leakage, and (4) benchmark models on popular yet
challenging datasets when possible. Due to the high cost of instruction-tuning, we were not able to run experiments with other held-out datasets.
Baselines. In SDDS and STDD, we benchmark DocLLM against comparably-sized SotA LLMs using ZS prompts that contain the text extracted from each document using an OCR engine (excluding the spatial information) (Touvron et al., 2023; Ouyang et al., 2022). In SDDS, we also report numbers from recent DocAI LLMs evaluated in a similar setting (Ye et al., 2023a,b). As motivated in Section 2, we do not consider DocAI models that require task-specific fine-tuning such as LayoutLMv3 (Huang et al., 2022) or Pix2Struct (Lee et al., 2023), and/or dataset-specific prompts such as UDOP (Tang et al., 2023). We instead focus on LLMs with out-of-the-box instruction following capability.7
Metrics. Following previous work (Borchmann et al., 2021; Lee et al., 2023; Ye et al., 2023b,a), we evaluate all VQA datasets using Average Normalized Levenshtein Similarity (ANLS) (Biten et al., 2019), with the exception of VisualMRC, for which we use CIDEr8 (Vedantam et al., 2015) and WTQ, for which we use accuracy. Performance on all
7Refer to Appendix C.4 for a comparison against SotA models regardless of architecture.
8This is done to remain consistent with the results reported by other baselines.
CLS and NLI datasets is measured using accuracy. We evaluate all KIE datasets with the F1 score.

heading: 4.3 Results
text: SDDS Setting. Table 2 shows that DocLLM-7B excels in 12 out of 16 datasets, inclusively compared to ZS results of GPT4 and Llama2, and SDDS results of mPLUG-DocOwl and UReader. Among equivalent models (excluding GPT4), our model outperforms in 14 out of 16 datasets. Specifically, DocLLM demonstrates superior performance in layout-intensive tasks such as KIE and CLS. In VQA and NLI, its performance surpasses that of most multimodal language models, although it underperforms compared to GPT4. GPT4 outperforms DocLLM in VQA, possibly due to the higher complexity of reasoning and abstraction involved in VQA datasets compared to tasks like KIE or CLS.9 DocLLM-1B demonstrates performance close to that of our larger model, suggesting that the smaller model can derive significant benefits from the architecture of DocLLM. STDD Setting. Table 3 shows that our model demonstrates superior performance compared to Llama2 across four out of five datasets, and achieves the best score overall for two of them (KIE task again). DocLLM also outperforms mPLUGDocOwl on DocVQA and both mPLUG-DocOwl and UReader on KLC, despite both baselines having been instruction-tuned on these datasets. However, it is important to note that classification accuracy is notably lower in our model. This discrepancy may stem from the fact that our model has been trained using only one CLS dataset, limiting its ability to generalize effectively to new datasets. Qualitative Comparisons. Figure 3 shows qualitative examples, comparing the outputs of DocLLM-7B and GPT4. Figure 3a corresponds to a KIE instruction, showing that DocLLM can provide correct answers when a question requires some knowledge of the semantic nuances of enterprise documents. DocLLM’s spatial reasoning abilities are demonstrated in Figure 3b, where the model correctly locates the heading ‘emergency protein allowances’ and identifies the text immediately underneath it. Figure 3c highlights a limitation, with the model failing at a counting task, at which GPT4 succeeds. See Appendix C.1 for more examples. Ablation Analysis. We conduct ablation studies based on Next Token Prediction (NTP) accuracy
9See Appendix C.2 for further details.
to validate the main contributions of DocLLM. We observe that incorporating the spatial modality in the attention mechanism performs better over the classical text-only modality, thereby validating the utility of disentangled spatial attention (See Table 4). Furthermore, block infilling with spatial modality outperforms causal learning, highlighting the value of fill-in-the-middle objectives (See Table 5). Appendix D contains more details.

heading: 5 Discussion
text: Impact. DocLLM enables language models to go beyond plain text settings and offers immediate utility in visually rich document understanding tasks. By accommodating complex layout structures, DocLLM allows documents with rich layouts to be included in the pre-training corpus without requiring extensive preprocessing. The explicit modeling of spatial relationships enables perceiving the documents as inherently structured knowledge.
Flexibility. The support for multi-page documents, implemented through page breaks and document boundaries, enhances the model’s ability to comprehend documents with diverse lengths. This overcomes the constraints of small multimodal models that can handle only a single page and multimodal LLMs mainly designed for images.
Limitations. The use of English-language datasets derived from limited enterprise domains (such as IIT-CDIP) may introduce inherent representational biases in VRDU models, including DocLLM. Also, DocLLM may be vulnerable to inaccurate bounding box information produced by an OCR engine.10 However, several modern off-the-shelf solutions can robustly extract text from documents (Hegghammer, 2022), mitigating this issue. DocLLM’s support for long-form documents is restricted by its context length. Increasing the model size and allowing unbounded context length during inference can address this limitation. DocLLM is trained and designed to capture the characteristics of enterprise documents. Consequently, its efficacy may be restricted when applied to documents outside this domain, such as presentation decks or marketing reports. Finally, DocLLM may not excel at complex reasoning tasks, especially those requiring a deep understanding of numerical concepts. See Appendix F for additional discussion.
10Appendix E studies the robustness of DocLLM to noisy input bounding boxes.

heading: 6 Conclusions
text: We introduced DocLLM, a lightweight extension to traditional LLMs, tailored for generative reasoning over documents with rich layouts. DocLLM eschews expensive image encoders and instead utilizes bounding box information to capture the spatial layout structure of documents. This is achieved through a disentangled attention mechanism that models cross-alignment between text and spatial modalities. Notably, our model addresses the challenges posed by irregular layouts and heterogeneous content using a learning to infill pre-training objective. Tuning the model on a carefully curated instruction dataset provides a flexible interface for interactions. Our evaluation across various document intelligence tasks demonstrates that DocLLM surpasses equivalent models both for in-domain and out-of-domain datasets. In the future, we plan to infuse vision into DocLLM in a lightweight manner.

heading: Acknowledgments
text: This paper was prepared for information purposes by the Artificial Intelligence Research group of JPMorgan Chase & Co and its affiliates (“JP Morgan”), and is not a product of the Research Department of JP Morgan. J.P. Morgan makes no representation and warranty whatsoever and disclaims all liability for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful. © 2024 JP Morgan Chase & Co. All rights reserved.

heading: A Dataset Details
text: A.1 Preprocessing
Of the datasets used in our study, IIT-CDIP and DocBank do not provide token-level OCR output. Therefore we process both datasets using the Tesseract-OCR engine.11 For the remaining datasets, we used the OCR output provided by each publisher.
A.2 Instruction Tuning Templates
For the extraction template, we add a “None” answer if a key does not exist in the given document, following Ye et al. (2023a,b). As described in Section 3.4 and Table 1, to increase diversity in the training data, we derive internal classification and MCQ instructions in addition to extraction instructions from the original KIE annotations. However, to stay consistent with benchmarks from previous work (Ye et al., 2023a,b), we only keep the prompts derived from the extraction template in the test split of each KIE dataset. To avoid the cold start problem induced by potentially unseen types of documents in testing or production usage, we only keep the MCQ prompts for the test split of each CLS dataset. Note that when a prompt accepts more than one answer, we create multiple copies of the prompt with one acceptable answer assigned to each. We only perform this “flattening” operation in the training split of the dataset.
A.3 Dataset Statistics
See Table 6 for pretraining dataset details and Table 7 for instruction tuning dataset details.

heading: B Training Details
text: DocLLM-1B is composed of 24 layers, each with 16 attention heads and a hidden size of 1,536. DocLLM-7B comprises 36 layers, 32 heads, and a hidden size of 4,096. Using pretrained weights as the backbone for the text modality, we extend the Falcon-1B and Llama2-7B models by adding the disentangled attention and block infilling objective as described in Section 3. We start directly from the pretrained weights of the backbone LLMs in order to continue their pretraining in a multimodal manner and avoid catastrophic forgetting of instruction following abilities (Luo et al., 2023; Zhai et al., 2024).
11https://github.com/tesseract-ocr/tesseract
Task #Train prompts #Test prompts
VQA 145,090 24,347 NLI 104,360 12,720 KIE 236,806 38,039 CLS 149,627 21,813 Total 635,883 96,919
For DocLLM-1B, we use a pre-training learning rate of 2 × 10−4 with 1,000 warmup steps, employing a cosine scheduler, and Adam optimizer (Kingma and Ba, 2015) with β1 = 0.9, β2 = 0.96 and a weight decay of 0.1. For instruction tuning we use a learning rate of 1 × 10−4 with 500 warmup steps and a cosine scheduler, and the same parameters for weight decay and Adam optimizer as the pre-training phase. The Adam epsilon is set to 1 × 10−5. We pretrain for one epoch, and instruction-tune for a total of 10 epochs.
For DocLLM-7B, pretraining involves a learning rate of 3× 10−4 with 1,000 warmup steps and cosine scheduler, weight decay of 0.1, and Adam optimizer with β1 = 0.9, β2 = 0.95. Instruction tuning uses a learning rate of 1× 10−4 with 500 warmup steps and a cosine scheduler, weight decay of 0.1, and Adam optimizer with β1 = 0.9, β2 = 0.95. Adam epsilon is set at 1× 10−6. We conduct one epoch of pretraining, followed by three epochs of instruction tuning, considering available computing resources.
The DocLLM-7B models are trained with 16-bit mixed precision on 8 24GB A10G GPUs using fully sharded data parallelism, implemented with the Accelerate library.12 The DocLLM-1B model, on the other hand, is trained on a single 24GB A10G GPU.
Table 8 provides an overview of the model configuration and training hyper-parameters that were used.
12https://huggingface.co/docs/accelerate

heading: C Detailed Performance Analysis
text: C.1 Qualitative Examples
Figure 4 shows additional qualitative examples from the DocLLM-7B output, where 4a highlights a VQA example from the DocVQA dataset (Mathew et al., 2021), 4b and 4c display two KIE examples from the DeepForm dataset (Svetlichnaya, 2020), and the bottom row shows CLS examples from the RVL-CDIP dataset (Harley et al., 2015).
As Figures 4a and 4e show, DocLLM can provide correct answers when the question requires some knowledge of the semantic nuances of enterprise documents. As an example, in Figure 4e, GPT4 mislabels a tax report issued by a local tax council as a scientific report, possibly due to the numeric contents of the table, whereas DocLLM is able to associate the content and the corresponding issuing authority with a budget report. Figure 4b demonstrates DocLLM’s spatial reasoning capability. The rightmost column of Figure 4 shows examples of failure by DocLLM. Each failure case demonstrates a limitation in the design and scope of the model. Figure 4c shows an example where DocLLM is unable to extract the gross amount. This error is due to the fact that the correct answer falls outside of the context window of the model, as it is located on the fourth page of a multi-page document. Lastly, Figure 4f shows an example for which the class predicted by DocLLM, i.e. “budget”, is semantically viable, but is nevertheless not the correct class. In future studies, we plan to address some of the above mentioned limitations, and increase the context length of the model.
C.2 DocVQA Deep-Dive
We conduct an in-depth analysis of the performance of DocLLM-7B on the various question categories of DocVQA. Table 9 lists the categories under which the DocVQA questions are listed. The “M” column identifies the modality that is expected to be uniquely helpful to the corresponding category. As an example, the “Figure/Diagram” category includes questions over charts and diagrams the answers to which rely on visual reasoning, e.g. “What is the variable taken along the x axis?” Whereas the “Form” category includes questions the answers to which require reasoning over layout, e.g. “What is the text at the top right corner of the page??”13
13Note that the text modality has been highlighted for the “Handwritten” category because the DocVQA dataset provides OCR output for all documents. In the absence of OCR output,
Needless to say, all modalities are often crucial to answering all question types. The modalities listed in the table are those expected to offer a uniquely important signal to the model.
As depicted in Table 9, DocLLM exhibits strong performance on “Form” and “Layout” questions, attaining scores of 82.2 and 72.4 respectively. These results underline the model’s proficiency in understanding and processing structured document formats and layouts. Conversely, the "Image/Photo", "Figure/Diagram", and "Yes/No" questions have lower scores of 47.9, 41.4, and 43.9 respectively. The absence of integrated vision features might account for DocLLM’s lower capacity in recognizing certain visual cues. Overall, DocLLM shows the strongest performance when reasoning over layout is key, and the weakest performance when visual reasoning is key.
C.3 GPT4V Performance Comparison
Given the recent roll out of the GPT4V API14 and the interest it has generated, we also benchmark DocLLM-7B against GPT4V on DocVQA and BuDDIE (Table 10). We select these datasets in order to include both SDDS and STDD results in the com-
the vision modality could play a more crucial role. 14https://openai.com/blog/ new-models-and-developer-products-announced-atdevday
parison. Moreover, as BuDDIE was not publicly released before this work, we can be certain that GPT4 and GPT4V were not trained on it. Due to cost and daily API usage limitations, we were not able to cover additional datasets.
We first observe that GPT4V does not uniformly outperform GPT4+OCR on the datasets considered. Both models show close ZS performance in BuDDIE CLS, but GPT4+OCR beats GPT4V in BuDDIE VQA while GPT4V tops GPT4+OCR on BuDDIE KIE and DocVQA. The additional vision component of GPT4V seems to help in general, especially for datasets such as DocVQA. However, as the characteristics of these model are undisclosed, analyzing their performance differences in depth is difficult. We do note that, despite its lack of visual and spatial features, GPT4+OCR fares well on VQA, KIE, and CLS tasks, and might be able to partially model the spatial relationships in documents based on the natural ordering of OCR tokens. Its robustness to OCR token position permutations is however not guaranteed.
Next, we observe that DocLLM-7B also outperforms GPT4V in addition to GPT4+OCR on BuDDIE SDDS. In the STDD evaluation setting, which is closer to out-of-distribution ZS inference, our model still exhibits competitive performance in VQA and KIE – although not consistently exceeding the scores of the likely larger GPT4 models. DocLLM’s lack of vision encoder appears to be mostly detrimental on DocVQA, where it particularly struggles on “Image/Photo” and “Figure/Diagram” questions, as seen in Section C.2.
C.4 SotA Performance Comparison In Table 11, we compare DocLLM-7B against the SotA on the datasets considered in this paper. Note that BuDDIE is not included here as it was not publicly released before this work. Similarly, DUDE and VRDU ad-buy are not considered in this section, since we used validation and bespoke splits respectively to evaluate models on them (see the caption on Table 2). FUNSD and PWC are also excluded from this study, as the prompts we built for these datasets leveraged annotations differently than previous work: our FUNSD KIE questions are based on the annotated key-value links, and our PWC KIE questions are formulated using the annotated set of Machine Learning tasks covered by the dataset.
Table 11 offers a few notable takeaways. First, despite the recent progress in multi-modal docu-
ment understanding, a foundation model that outranks others across a wide range of tasks and datasets does not currently exist. Most SotA models are single-task fine-tuned models that outperform others in one or a few datasets, as seen here with LayoutT5 (Tanaka et al., 2021), StructuralLM (Li et al., 2021), PASTA+DATER (Ye et al., 2023d), GPT-3.5+DP+PyAgent+MixSC (Liu et al., 2023b), and GraphDoc (Zhang et al., 2023b). The same observation applies to general NLP (Brown et al., 2020; Wang et al., 2022; Chowdhery et al., 2022; Naveed et al., 2023). While UDOP tops all models on three KIE datasets, it remains an expert model that requires dataset-specific prompts and per dataset fine-tuning (on top of its multitask supervised pretraining) in order to reach the performance reported. Similarly, Pix2Struct, a specialist model with a ViT encoder and a text decoder pretrained on 80M website screenshots, exhibits strong performance on DocVQA (76.6 for its 1.3Bparameter variant), but underperforms in chart and infographics understanding compared to generalist multimodal LLMs such as UReader. Its performance on KIE and CLS datasets is also understudied. On table-based datasets such as WTQ and TabFact, SotA models rely on large, text-only LLMs to reason over data using SQL or Pandas – thus reducing their ability to generalize to nontabular document data. The abstractive reasoning limitations of DocLLM-7B are more apparent on these table-based datasets, but our single model performs competitively in KIE and CLS (even on KLC and Deepform, despite DocLLM’s relatively short context-length).
Second, recent multimodal LLMs such as QwenVL-Max15 and GPT4V16 show impressive ZS performance in VQA. These generalist models report strong performance on DocVQA and other datasets like ChartQA (Masry et al., 2022) and InfographicVQA (Mathew et al., 2022) (which we do not consider in this paper) thanks to their additional vision encoder.17 However, the lack of transparency about their size, exact architecture, training procedure, and training data makes it hard to draw any conclusions. On DocVQA, DocLLM-7B outperforms Qwen-VL-10B (Bai et al., 2023). Moreover, as these recent multimodal LLMs were designed
15https://qwenlm.github.io/blog/qwen-vl/ 16https://openai.com/research/gpt-4 17In future studies, we hope to equip DocLLM with access to the vision modality too — albeit in a more efficient manner than is typically implemented.
to tackle a wide range of tasks (e.g., image captioning) and not just DocAI, their ZS performance on certain tasks considered here (document NLI, KIE, CLS) has not been investigated – making a thorough comparison with our model even more complex.
Finally, despite lower performance compared to the top-performing model in each category, DocLLM still shows superior performance to generalist LLMs of comparable size, as indicated in Table 2. The model also proves robust to out-ofdistribution data in ZS, as demonstrated in Table 3.

heading: D Ablation Studies
text: We conduct ablation studies to validate the three main contributions of DocLLM: (1) disentangled spatial features, (2) the block infilling pre-training objective, and (3) the masking strategy used for decoding. For all ablations, we use Next Token Prediction (NTP) out-of-sample accuracy to compare configurations at the pre-training stage. Due to resource restrictions, each experiment uses a subset of our pre-training corpus: we randomly sample 100,000 chunks and predict on 1,000 unseen documents. A chunk is a collection of documents wherein the total number of tokens across the collection is less than the maximum input context length. The hyperparameters are set consistently following Table 8 across all ablation experiments.
Disentangled Spatial Attention. To measure the effect of disentangled spatial attention on cross-
modal interactions, we train the models by setting the λ hyperparameter in Eq 4 to 0 or 1. Table 4 enumerates the attention combinations, and the results suggest that keeping only the spatial-to-spatial interaction (i.e. λs,s = 1) yields the highest NTP accuracy. The performance differences among other configurations, such as text-to-spatial and spatialto-text, are subtle. Notably, the vanilla text-only self-attention mechanism yields the lowest NTP accuracy, underlining the importance of incorporating spatial features for understanding documents with rich layouts. For all experiments in Section 4, we therefore set λs,s = 1, λs,t = 0, and λt,s = 0. We opt for simplicity by choosing a hard mode over a soft one while acknowledging the potential advantage of flexibility for the latter.
Autoregressive Block Infilling. To evaluate the effectiveness of the proposed autoregressive block infilling objective especially comparing with the conventional left-to-right causal learning, we benchmark three configurations in our ablation study: (1) causal learning, (2) causal learning with spatial modality, and (3) block infilling with spatial modality. As highlighted in Table 5, autoregressive block infilling exhibits the best performance. Additionally, the performance gain of adding the spatial modality to the causal learning proves the advantage of the spatial modality.
Prefix Decoder and Causal Decoder. For document-conditioned generation, an intuitive choice is to employ a prefix decoder with prefix masking that utilizes bidirectional attention mechanism for the entire document, as illustrated in Figure 5b. We investigate this assumption through experiments where we compare a prefix decoder against the conventional causal decoder. Specifically, we conduct experiments on these two decoders for different settings outlined in the Disentangled Spatial Attention ablation to study their resulting performance.
The results in Figure 6 show marginal differences between these two decoders across the five configurations, with the causal decoder having a slight edge over the prefix. The minor difference suggests that both masking methods are comparable in modeling documents. Thus the bidirectional attention enabled by the prefix decoder may not be crucial in this context, and we consequently elect to use a causal decoder for all experiments in section 4.

heading: E Robustness to inaccurate OCR Bounding Boxes
text: To assess DocLLM’s sensitivity to inaccurate token bounding boxes, we conduct experiments on DocVQA and inject variable amounts of noise to shift the borders of the original OCR data. Each border is shifted by ϵ ∼ N (0, l2σ2), where l is the length of the sides orthogonal to the border considered and σ is the hyperparameter we use to control the amount of noise injected. We clip tail values beyond ±2lσ and restrict σ to values between 0 and 0.25 to avoid accidentally swapping bounding box borders.
We observe in Table 12 that DocLLM-1B’s performance on DocVQA remains very stable when input OCR borders are randomly shifted, highlighting the model’s robustness to moderately inaccurate spatial coordinates.

heading: F Additional Discussion
text: The main concept for a cohesive block is to ensure meaningful infilling during the pretraining phase, preventing disconnected predictions. However, the choice of OCR engines to obtain such cohesive blocks remains an open area for exploration. Practical comparisons with various OCR engines and/or layout parsers are left as future work, as LayoutLMs underscore the importance of accurate OCR for improved VQA results. They leverage the Microsoft Azure API, demonstrating superior performance compared to TesseractOCR, as indicated in the DocVQA leaderboard.18 Consequently,
18https://rrc.cvc.uab.es/?ch=17&com=evaluation& task=1
researchers are also encouraged to utilize more accurate OCR engines for potential enhancements, if such resources are available.
We have presented a collection of SDDS results alongside zero-shot outcomes. To mitigate prompt influence in the zero-shot results, a rigorous methodology was implemented. This involved the engagement of three independent prompt engineers, each undergoing five rounds of refinement for zero-shot settings, followed by a series of postprocessing techniques to enhance result reliability. The best results are thus obtained from each of the three groups. We still acknowledge the potential for refinement and improvement.
We share some internal training experiences, acknowledging the absence of robust validation. First, we observe that a higher weight decay (e.g., 0.1 versus 0.01) generally improves performance in both pretraining and instruction tuning. During the instruction tuning phase, a higher initial learning rate, such as 1e-4 versus 5e-5, leads to enhanced performance. Overall, we’ve observed that the cosine scheduler tends to outperform linear or constant schedulers across various settings.

heading: Future Work
text: Practical comparisons with various OCR engines and/or layout parsers are left as future work, as LayoutLMs underscore the importance of accurate OCR for improved VQA results. They leverage the Microsoft Azure API, demonstrating superior performance compared to TesseractOCR, as indicated in the DocVQA leaderboard. 18 Consequently,
18https://rrc. cvc. uab. es/?ch=17&com=evaluation& task=1
researchers are also encouraged to utilize more accurate OCR engines for potential enhancements, if such resources are available. We have presented a collection of SDDS results alongside zero-shot outcomes. To mitigate prompt influence in the zero-shot results, a rigorous methodology was implemented. This involved the engagement of three independent prompt engineers, each undergoing five rounds of refinement for zero-shot settings, followed by a series of postprocessing techniques to enhance result reliability. The best results are thus obtained from each of the three groups. We still acknowledge the potential for refinement and improvement. We share some internal training experiences, acknowledging the absence of robust validation. First, we observe that a higher weight decay (e. g. , 0. 1 versus 0. 01) generally improves performance in both pretraining and instruction tuning. During the instruction tuning phase, a higher initial learning rate, such as 1e-4 versus 5e-5, leads to enhanced performance. Overall, we’ve observed that the cosine scheduler tends to outperform linear or constant schedulers across various settings