Enhancing Document Understanding with Group Position Embedding: A Novel Approach to Incorporate Layout Information

Authors: Yuke Zhu, Yue Zhang, Dongdong Liu, Chi Xie, Zihua Xiong, Bo Zheng, Sheng Guo
Primary area: foundation or frontier models, including LLMs
Keywords: DocAI, LLM, Position Embedding

Abstract:
Recent advancements in document understanding have been dominated by leveraging large language models (LLMs) and multimodal large models. However, enabling LLMs to comprehend complex document layouts and structural information often necessitates intricate network modifications or costly pre-training, limiting their practical applicability. In this paper, we introduce Group Position Embedding (GPE), a novel and efficient technique to enhance the layout understanding capabilities of LLMs without architectural changes or additional pre-training. GPE achieves this by strategically grouping the attention heads and feeding each group with distinct positional embeddings, effectively encoding layout information relevant to document comprehension. This simple yet powerful method allows for effective integration of layout information within the existing LLM framework.  We evaluate GPE against several competitive baselines across five mainstream document tasks. We also introduce a challenging benchmark called BLADE, specifically designed to assess layout comprehension. 
 Extensive experiments on both established and BLADE benchmarks confirm the efficacy of GPE in significantly advancing the state-of-the-art in document understanding. Our code is available at https://github.com/antgroup/GroupPositionEmbedding.git

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 5

Reviews and discussion:
<thread>
creation date:
1730202029791

thread_type:
official_review

<thread_object>
creator: reviewer_ntfi

<content>
summary:
This paper proposes a simple yet effective method, Group Position Embedding (GPE), for encoding spatial positional information in LLM-based document understanding tasks. This technique enables LLM models to comprehend document layouts without altering their architectures. Additionally, the authors introduce a new benchmark, BLADE, for evaluating complex document processing.

strengths:
1.This paper introduces a simple layout embedding approach, termed Group Position Embedding (GPE), for enhancing LLM-based document understanding.
2.This paper proposes a new benchmark, BLADE, designed for the evaluation of complex document evaluation. 
3.Extensive ablation experiments have been conducted to validate the effectiveness of GPE.

weaknesses:
1. The rationale behind GPE is unclear. The introduction section fails to adequately explain the workings of the group position embedding, instead, it merely highlights the advantages without providing underlying reasons. It is also unclear what the principles and feasibility of the mapping function g_r() are, and the distinction between Gr(k) and Gr(i) on lines 196-197 is also not elucidated. Further explanation is needed on how the n-dimensional spatial position is mapped into different attention heads. The meaning of the hyperparameter scaling factor \lambda and its impact on the approach should also be clarified.
2. This paper is not the first to utilize head-specific layout position embeddings. Previously, LAGaBi [https://aclanthology.org/2023.findings-emnlp.521/] employed diverse Gaussian kernels to encode relative positional information for each attention head. 
3.The paper's writing quality is subpar. Inconsistent and non-standard academic expressions are used. For instance, in LLMs, l typically denotes the number of layers rather than dimensions, which should be represented by d or dim. The figures do not clearly demonstrate the mapping mechanism of group position embedding and convey some ambiguity. For instance, Figure 1 shows that only the key-state and value-state encode n-dimensional spatial position information, whereas the query-state receives only 1-dimensional position information, which is confusing.

questions:
Please refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730531906930

thread_type:
official_review

<thread_object>
creator: reviewer_wcmm

<content>
summary:
The paper makes two main contributions: (1) it uses group position embedding to ensure that different attention heads focus on different views of position, and (2) it introduces a new document AI dataset, BLADE, which highlights the model's ability to handle complex layout information. Experimental results demonstrate improved performance, and the ablation studies are comprehensive.

Updated on 23th Nov 2024: raise the score

strengths:
1. The proposed group position embedding is novel.
2. The proposed BLADE dataset is important to the field.
3. The experiments are extensive, especially the comparison of different approaches to fuse layout information is insightful.

weaknesses:
1. Although the experimental results in Tables 2 and 3 demonstrate the superiority of GPE, the results in Table 1 appear somewhat contradictory to those in Tables 2 and 3, requiring further explanation or experiment.
2. The paper needs a section discussing why GPE is effective in guiding attention heads to focus on different positional views. Including a visualization of attention scores or similar analysis would provide valuable insights, as the improved performance alone is insufficient to fully explain the mechanism.

3. The writing requires improvement, as several paragraphs are vague and difficult to understand. For example:
(1) Lines 248-249: What is the value of lambda? Does GPE discretize all coordinates into integers, allowing us to obtain position embeddings for specific position?
(2) Line 418: In the experiment regarding reading order, I cannot understand the difference between W/O and LOCAL. A concrete example would help illustrate the distinctions between each setting.
(3) Line 473: It would be helpful to introduce the design of the group function where it first appears, in Section 3.2, using a formal mathematical definition. This would clarify the paragraph, as I found it confusing without prior context on the group function.

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730536285333

thread_type:
official_review

<thread_object>
creator: reviewer_hpjd

<content>
summary:
This paper introduces a new positional encoding method to tackle the limitations of current LLMs/MLLMs' lack of layout information for addressing layout-aware document understanding tasks. A dataset named BLADE is designed to mainly focus on measuring the performance on complex-layout aware questions. Various experiments with different model configurations are conducted.

strengths:
1. The paper proposed a new positional encoding to address the limitation of existing LLMs/MLLMs neglecting layout information. 
2. A benchmark is proposed specifically focusing on evaluating the performance of various LLMs/MLLMs on document understanding tasks with complex layouts.

weaknesses:
1. Motivation: The limitations of current positional encoding methods adopted by other frameworks are not clearly defined. The research gaps are not clearly defined.
2. The methods are described well with the technical workflow without specific reasons and experiments as to why the positional encoding is working. For example, there is no explanation or citation as to why different position information giving various heads are reasonable for the research aim. 
3. The datasets are not clearly described making understanding breakdown categories difficult. 
4. The proposed methods are only evaluated on LLM which is expected to see whether it's workable on MLLMs and pretrained document understanding models. 
5. More reading order setups should be tried like XY-cut. 
6. There are some typos and some possible errors in the paper, like Table 2 SCOIE performance. Some bolded digits in the performance tables are not the highest.

questions:
1. Motivation and Related Work: what is the limitation of current positional encoding methods adopted by pretrained document understanding frameworks and what are the limitations of them? 
2. Dataset: is that necessary to have a layout-aware dataset. Only focusing on layout-complex questions may ignoring the performance on other generative tasks. 
3. Model and Evaluation: is there any reason you chose those LLMs/MLLMs? It would be better to give more insight analysis for this part by giving more ablation and case studies. 
4. Is it possible to show some qualitative analysis of benchmark datasets like CORD, SROIE?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704423796

thread_type:
official_review

<thread_object>
creator: reviewer_xtdc

<content>
summary:
This paper introduces Group Position Embedding (GPE), a method to enhance layout comprehension in Large Language Models (LLMs) without requiring architectural changes or extra pre-training. By grouping attention heads with distinct positional embeddings, GPE effectively encodes layout information for document tasks. Tested on five standard benchmarks and the challenging BLADE benchmark, GPE shows significant improvement in document understanding over existing methods.

strengths:
(1) interesting task

(2) new method

(3) extensive experiments

weaknesses:
(1) Missing important details for dataset construction: Any critiria to manual and filter challenging question-answer pairs for the data scenarios of Forms, Slides, and Websites? Why is Newspapers constructed with an initial manual selection whereas SynthDocs being synthetically generated? For SynthDocs whose answers are synthetically generated, any filtering measure to ensure the data quality? 

(2) Missing the intuitive motivation of the proposed method: Although this paper verified the effectiveness of the proposed method compared with other approaches on Layout-aware Position Embeddings through the empirical experiments, it lacks of more intuitive explanations on the advantages of the proposed method, which can provide more insights for the follow-up work.

(3) Missing the clarification of the experiment results: According to  the experiment results in Table 1, vanilla Qwen2-7B  outperforms GPE-Qwen2-7B on VisualMRC, which needs more clarifications.

questions:
Please refer to the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732691051791

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Awaiting Reviewer Feedback

comment:
Dear Area Chairs,

We have meticulously addressed each reviewer's comment with comprehensive responses during the rebuttal phase. At present, we have not yet received any responses from 2 reviewers (reviewer hPjd and XTDc). We are looking forward to receiving valuable feedback and insights from them, and appreciate your support throughout this process very much.

Best regards,

Submission 612 authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733037065086

thread_type:
public_comment

<thread_object>
creator: ~yi_zhang104

<content>
title:
Comments and Concerns Regarding the Paper

comment:
Dear Authors and Reviewers,  

We would like to bring to your attention the following points:

1. In Table 1, we noticed that the performance of DocLayLLM and DocLLM on the VIE row appears to be identical. **This should be an error.**

2. We observed that DocLLM employs the F1 metric for the VIE task, which does not align with the ANLS metric mentioned in Table 1. **Is such a comparison appropriate or fair?**

3. LayoutLLM did not utilize the DocVQA, VisualMRC, FUNSD, CORD, or SROIE datasets during training, whereas this paper includes these datasets in the training process. **Could the authors comment on whether this comparison remains fair?** Furthermore, DocLayLLM and LayTextLLM provide experimental results under different settings. **Could the authors confirm whether the comparison made in your paper is based on the zero-shot setting, the VQA setting, or the all-setting?** It would be helpful to understand whether the comparison is fair under these different configurations.

Thanks for this work, and we appreciate your attention to these points. We look forward to your responses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733059765214

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response for general concern

comment:
## Revised Paper and Future work
We have revised and re-uploaded the paper. The key changes are highlited in yellow.  The revised part mainly includes
- Comparison with LAGaBi in related work
- Grouping function definition
- A new Section(A.7) for the impact of scaling factor and experiments.
- Adding experiment of XY-cut for comparison
- More details of BLADE.
- A new Section(A.8) for intuitive analysis of GPE including the attention map visualization.
- Some typos

Future work of open source plan includes:
- BLADE and its evaluation scripts.
- Code of GPE as well as its training and evaluation
- Release model weights based on SOTA LLMs with GPE using large scale training data

## Motivation and Ituitive Analysis of GPE
**Motivation**
Our objective is to integrate layout information into LLMs. Given the limitations of existing methods, we have identified three key characteristics that our approach should embody:
1. **Unambiguous Layout Representation**: The representation of the layout should not result in any loss of information. 
2. **No Additional Vocabulary**: Introducing new vocabulary or modalities would complicate the training process.
3. **No Additional Tokens**: Adding extra tokens would disrupt the original text sequence, making the LLM hard to understand the original text.

The GPE is designed to meet all the properties. GPE satisfies the first property by ensuring that each dimension of layout information is processed individually by different heads.  In contrast, the ``Add Box Embedding'' used in LayoutLM just adds the embeddings of each layout dimension, which brings ambiguity, e.g. the representations of [x1, x2] and [x2,x1] are identical. GPE satisfies property 2 by reusing the original position embeddings of LLM. As it is not a new modality or new vocabulary, the model is easy to adapt. GPE does not change the input sequence thus satisfying property 3.

**Explanation of why GPE is effective**
The key technique that enhances the effectiveness of GPE is its **head-specific design**. This design ensures that each dimension of layout information is processed by each attention head. The benefits of this approach are akin to those of multi-head attention, as it encourages the model to learn positional relationships from various perspectives. By distributing the layout relations across multiple attention scores, rather than relying on a single score, the model can more effectively capture and utilize different aspects of the layout. Intuitively, some heads might focus on horizontal (left-to-right) relationships, while others concentrate on vertical (up-to-down) relationships.

**Visualization of Attention Map**
Please refer to the newly added Section A.8 in the revised paper.

## Details of BLADE
We have added more details to BLADE, which includes two parts: (1) an explanation of the differences in the construction details between Newspapers and SynthDocs; (2) additional information on the quality filtering process for SynthDocs.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors present a novel approach, Group Position Embedding (GPE), to enhance document understanding by enabling attention heads to focus on different positional views. The authors also propose a new benchmark, BLADE, for complex document processing, offers a valuable contribution to the field.

To further strengthen the paper, I recommend incorporating a more comprehensive discussion of related work, particularly in the area of layout encoding for language models. While the authors have considered recent advancements in LLMs and LMMs, exploring earlier work, such as ROPE: Reading Order Equivariant Positional Encoding for Graph-based Document Information Extraction (ACL 2021), can provide valuable insights and historical context.