A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding

Authors: Jinghui Lu, Haiyang Yu, Yanjie Wang, Yongjie Ye, Jingqun Tang, Ziwei Yang, Binghong Wu, Qi Liu, Hao Feng, Han Wang, Hao Liu, Can Huang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: llm, docai, visually rich document understanding, kie

Abstract:
Recently, many studies have demonstrated that exclusively incorporating OCR-derived text and spatial layouts with large language models (LLMs) can be highly effective for document understanding tasks. However, existing methods that integrate spatial layouts with text have limitations, such as producing overly long text sequences or failing to fully leverage the autoregressive traits of LLMs. In this work, we introduce Interleaving Layout and Text in a Large Language Model (LayTextLLM)} for document understanding. In particular, LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. LayTextLLM not only streamlines the interaction of layout and textual data but also shows enhanced performance in Key Information Extraction (KIE) and Visual Question Answering (VQA). Comprehensive benchmark evaluations reveal significant improvements, with a 27.2% increase on KIE tasks and 12.0% on VQA tasks compared to previous state-of-the-art document understanding MLLMs, as well as a 15.1% improvement over other SOTA OCR-based LLMs on KIE tasks.

Decision: reject
Reviewer scores: 5, 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1730283482226

thread_type:
official_review

<thread_object>
creator: reviewer_p4ct

<content>
summary:
This paper introduce Interleaving Layout and Text in a Large Language Model (LayTextLLM) for document understanding, which projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. They also propose a tailored pre-training task—Layout-aware Next Token Prediction—a completely self-supervised task that enhances the alignment between layout and textual . LayTextLLM significantly improves performance on downstream document-related VQA and KIE tasks.

strengths:
1.This work is the first work to employ a unified embedding approach that interleaves spatial layouts directly with textual data within a LLM.
2.They propose two tailored training tasks: (1) Layout-aware Next Token Prediction  and (2) Shuffled-OCR Supervised Fine-tuning task to better elicit the model generalizability in downstream tasks.
3.Experimental results sounds good.

weaknesses:
1.The methods adopt a P-LoRA structure, which is introduced by existing work InternLM-XCompoer2, limiting the novelty of this work‘s.
2.The analysis of the impact of SLP training on the effectiveness of the model remains incomplete.

questions:
1.The different training tasks such as Layout-aware Next Token Prediction mentioned，how many tokens of training data were used respectively?
2.Why does the chosen lora structure work as expected? Do other loras have the same effect, and how are ablation trials conducted?
3.Is there any analysis examining the impact on the llama models' ability to understand language after incorporating spatial information?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730380385925

thread_type:
official_review

<thread_object>
creator: reviewer_zrgx

<content>
summary:
Traditional methods for document understanding usually rely on the text extracted by OCR and spatial layout information, but they have limitations when dealing with long text sequences. This paper proposes a method named LayTextLLM, which reduces the sequence length problem by mapping each bounding box to a single embedding and interleaving it with the text, while taking advantage of the autoregressive characteristics of LLM. Two training tasks are proposed: layout-aware next token prediction and supervised fine-tuning of shuffled OCR, to enhance the generality of the model. In Key Information Extraction (KIE) and Visual Question Answering (VQA) tasks, LayTextLLM significantly outperforms the existing state-of-the-art methods. In the zero-shot scenario, the performance of the KIE task is improved by 27.0% and that of the VQA task is improved by 19.8%.

strengths:
1. The paper proposes the adoption of a unified embedding method in LLM for the first time, interleaving the spatial layout with the text data.
2. By reducing the sequence length and making full use of the autoregressive characteristics, the performance of document understanding is significantly improved.
3. Comprehensive experimental results and ablation studies are provided, demonstrating the effectiveness of the proposed method.

weaknesses:
Compared with the existing VLM papers, the paradigm is actually quite similar. Although it is not highly innovative, it offers a new perspective for handling OCR document understanding problems in this field. Another point is that perhaps the number of baselines for comparison seems a bit small. I wonder if there are more and newer works related to Text + Coordinates for comparison.
Additionally, there is a slight error in the bolded text in Table 1.

questions:
refer to weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703127993

thread_type:
official_review

<thread_object>
creator: reviewer_xpl9

<content>
summary:
The paper use one token (generated by a small NNs called SLP) to replace the four coordinate representation of location for normal document understanding tasks. With such compression, the model can accept more tokens and longer context than a vanilla llm. The method show better results than some document understanding MLLMs.

strengths:
- The empriical performance shows a imporlvement on downstream document understanding tasks

weaknesses:
Overall, the paper propose an emprical trick to compress four localization tokens into one token, however, the contribution is thin and the research problem is not valid to me (without futher considering the visual features). 


(updated in Nov 20)

- **Research Problem Validity**:
The core research problem studied in this paper appears incomplete without incorporating visual features. This concern is supported by two key pieces of evidence:
a) The evolution of document understanding models shows that pure OCR-text-only approaches (like LayoutLM[1]) have been quickly superseded by multimodal approaches (LayoutLMv2, LayoutLMv3[2,3]), demonstrating the crucial importance of incorporating visual information in document understanding tasks.
b) Recent findings from MMMU-PRO [4] demonstrate that even with PERFECT text information, the VQA task accuracy only improves by a marginal 0.5%. This empirically proves that text-only approaches are **insufficient*** for comprehensive document understanding, regardless of text quality.

- **Scope and Contribution**:
While I acknowledge the value of theoretical work, this paper reads more like an appendix to comprehensive document understanding rather than a standalone contribution. The presented approach seems to address only a narrow subset of document understanding challenges, without adequately justifying why visual features are excluded, especially given the strong empirical evidence supporting the necessity of visual information.

- **Broader Impact and Audience**:
For the broad ICLR audience, the paper offers limited insights and contributions. The work might be more suitable for venues specifically focused on document understanding, where the technical details of OCR processing would find a more relevant audience. Moreover, given the clear trend towards multimodal approaches in the field, this text-only approach may be of limited practical value to the research community.



[1] LayoutLM: Pre-training of Text and Layout for
Document Image Understanding

[2] LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding

[3] LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking 

[4] MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709561554

thread_type:
official_review

<thread_object>
creator: reviewer_mynu

<content>
summary:
This paper presents LayTexLLM, which interleaves layout and text in an LLM for document understanding. There are two key ideas:
- Spatial layout projector (SLP), which project the coordinate of each bbox (_i.e._, [x_min, y_min, x_max, y_max]) derived by an off-the-shelf OCR module into a $d$-dimensional embedding, so that the layout information can be input to the LLM and learned together with other text tokens via next token prediction (layout-aware).
- Shuffled-OCR supervised fine-tuning (SSFT), which randomly shuffles the order of OCR-derived text in a certain proportion of examples to mitigate the bias that OCR engines typically output text bboxes in a way of top-to-bottom and left-to-right. 
Experiments on DocVQA, InfoVQA, and KIE tasks show that LayTextLLM outperforms or achieves comparable performance to baselines, while experiencing a significant performance drop on the ChartQA dataset.

strengths:
- LayTextLLM demonstrate that mapping the coordinates to a single bbox embedding can achieve delightful performance on document understanding while mitigating long sequence issues.
- Table 1 and Table show that LayTextLLM demonstrate performance improvement over several baseline methods on DocVQA, InfoVQA, and KIE tasks.

weaknesses:
1. The performance of the model heavily relies on the quality of the OCR engine, which poses a risk of significant performance drop, especially for handwritten documents or low-resource languages where OCR performance can be subpar. Moreover, incorporating an OCR module can negatively impact inference latency, making real-time or high-throughput applications challenging.

2. As indicated in Table 3, LayTextLLM_zero and even LayTextLLM_all demonstrate inferior performance on Document-Oriented VQA compared to LayoutLLM (Luo et al., 2024). Additionally, Table 8 reveals that LayTextLLM shows a substantial drop in performance on the ChartQA dataset. Specifically, OCR-free methods generally achieve scores between 50-59.9, whereas the best performance achieved by LayTextLLM_all is only 35.4.

3. The ablation study requires a clearer description of the variants involved. The first line in Table 4 lacks clarity—does it refer to a setting that uses naive coordinate-based pretraining (e.g., normalized [x1, y1, x2, y2]) with the proposed SSFT strategy? If not, it would be beneficial to include such a variant to demonstrate the effectiveness of the Spatial Layout Projector.

questions:
(a) Are the reported numbers for LayoutLLM_CoT in Table 3 inaccurate? The current numbers seem to correspond to Vicuna 1.5 7B. Would it be more appropriate to report the results based on Llama2-7B for a fairer comparison?

(b) Why does LayTextLLM show significantly inferior performance on the ChartQA dataset, even compared to OCR-free methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732367239850

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Invitation to Join in Discussion

comment:
Dear Reviewers,

We sincerely appreciate the time and effort you've dedicated to reviewing our work. We are grateful for the recognition of the novelty and performance improvements acknowledged by the reviewers. We have carefully reviewed all the feedback and addressed the concerns in our rebuttal. Based on this feedback, we will make the revisions to improve the manuscript.

As the discussion period is drawing to a close, we kindly request your acknowledgment of our reply. We are open to further discussions to clarify any remaining questions or concerns. We would greatly appreciate it if you could consider improving the evaluation after reviewing our responses.

Thank you very much for your consideration.

Sincerely, The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces LayTextLLM, a method for incorporating document layout into LLMs. It uses: (1) a Spatial Layout Projector (SLP) to embed bounding box coordinates from OCR output, enabling layout-aware next token prediction; and (2) Shuffled-OCR supervised fine-tuning (SSFT) to reduce OCR-induced positional bias

The central contribution of this work is the SLP module, which effectively integrates layout and text. However, as noted in the related work section, Perot et al. (2023) use formats such as “HARRISBURG 78|09” to represent OCR text and corresponding layout, while He et al. (2023) use “[x_min, y_min, x_max, y_max]” to encode layout information. It is unclear whether fine-tuning with LoRA using these existing formats would yield comparable performance. If so, the primary advantage of the proposed method would be token savings achieved through coordinate embedding. Reviewer ZrGX raised a similar concern.