Enhancing Document Understanding with Group Position Embedding: A Novel Approach to Incorporate Layout Information
Authors: Yuke Zhu
Yue Zhang
Dongdong Liu
Chi Xie
Zihua Xiong
Bo Zheng
Sheng Guo
Venue: ICLR 2025 Poster (2025)
Keywords: DocAI
LLM
Position Embedding
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper proposes a layout-aware position embedding that enable LLMs to comprehend complex documents.

Abstract: Recent advancements in document understanding have been dominated by leveraging large language models (LLMs) and multimodal large models. However, enabling LLMs to comprehend complex document layouts and structural information often necessitates intricate network modifications or costly pre-training, limiting their practical applicability. In this paper, we introduce Group Position Embedding (GPE), a novel and efficient technique to enhance the layout understanding capabilities of LLMs without architectural changes or additional pre-training. GPE achieves this by strategically grouping the attention heads and feeding each group with distinct positional embeddings, effectively encoding layout information relevant to document comprehension. This simple yet powerful method allows for effective integration of layout information within the existing LLM framework.  We evaluate GPE against several competitive baselines across five mainstream document tasks. We also introduce a challenging benchmark called BLADE, specifically designed to assess layout comprehension. 
 Extensive experiments on both established and BLADE benchmarks confirm the efficacy of GPE in significantly advancing the state-of-the-art in document understanding. Our code is available at https://github.com/antgroup/GroupPositionEmbedding.git

Published as a conference paper at ICLR 2025
ENHANCING DOCUMENT UNDERSTANDING WITH GROUP POSITION EMBEDDING: A NOVEL APPROACH TO INCORPORATE LAYOUT INFORMATION
Yuke Zhu, Yue Zhang,, Dongdong Liu', Chi Xie, Zihua Xiong, Bo Zheng', Sheng Guo MYbank, Ant Group
[telix.yb guosheng. beapbbenngggusseeeee. (rooshnngeemybaree on.
ABSTRACT
Recent advancements in document understanding have been dominated by Large Language Models (LLMs) and Large Multimodal Models. However, enabling LLMs to comprehend complex document layouts and structural information often necessitates intricate network modifications or costly pre-training, limiting their practical applicability. In this paper, we introduce Group Position Embedding (GPE), a novel and efficient technique to enhance the layout understanding ca- pabilities of LLMs without architectural changes or additional pre-training. GPE achieves this by strategically grouping the attention heads and feeding each group with distinct positional embeddings, effectively encoding layout information rele- vant document comprehension. This simple yet powerful method allows for ef- fective integration layout information within the existing LLM framework. We evaluate GPE against several competitive baselines across five mainstream docu- ment tasks. We also introduce a challenging benchmark called BLADE, specifi- cally designed assess the layout comprehension capabilities. Extensive experi- ments on both established benchmarks and BLADE confirm the efficacy of GPE in significantly advancing the oftte-of- in document understanding. Our code is available at httpg//twwwooooo/ooo/ooorronnrii
INTRODUCTION
Understanding complex document using AI has long been popular topic academia. Since the advent of Transformers (Vaswani et al. 2017), a series of pretrained models (Xu et al.) 2020; 2021 Huang et al. 2022) based on the Transformer architecture have been proposed. For an extended period, these methods have been the mainstream solutions for document understanding. Recently, large language models(LLMs)) like ChatGPT (OpenAl) 2022) have exhibited impressive capabilities across various tasks, particularly their Zero-Shot abilities, which have quickly established them new paradigm for future Al applications. In terms document tasks, LLM-based methods (Perot
et al. 2023 He et al. 2023 Zhang et al. 2023 Ye et al. 2023a Bai et al. 2023) have also emerged
as hot research direction.
The approaches to document processing based on LLMs have evolved into several paradigms. The first approach (Liu et al., 20236 Zhang et al.) 2023) Ye et al.] 2023a Bai et al., 2023] Shi et al. 2023) directly processes document images. It first encodes the image using a visual encoder and then inputs the encoded representation into the LLM. Such methods typically require substantial pre- training data enable the model to learn how understand documents based on visual information.
The second approach (Perot et al. 2023: Luo et al. 2024; Liao et al., 2024 Wang et al. 2023) utilizes OCR (Optical Character Recognition) models to first convert the document into text, then fed them to L.M. This method does not require the use of costly Vision-Language Language Pretraining, thereby reducing the challenge of direct document image understanding for L.M.. However, it struggles with preserving the original layout and structure information of the documents. To address this shortfall, recent efforts (Luo al. 2024 Wang et al. 2023
Liao et al. 2024) have employed large-scale
Major experimental contribution Major dataset contribution
Corresponding author.

---

Published as a conference paper at ICLR 2025
data pre-training to achieve this. In summary, when employing these methods to enable LLMs to understand complex documents, extensive pre-training is indispensable.
In this paper. we introduce novel and efficient method for modeling document layout information based on LLMs, characterized by its consice design that climinates the need for costly pre-training. We design new positional embedding scheme for layout information to help the LLM understand layout nuances. Notably, this approach does not rely on complex visual encoders or introduce any additional learnable parameters. Instead, we incorporate layout information solely by modifying the existing position embedding of LLM. Specifically, we view the layout information as a multi- dimensional vector and use multiple sets of position embeddings represent it. Inspired by the independence of attention heads in the multi-head attention mechanism, we partition the attention heads into groups as needed. Each group incorporates its corresponding positional information. We refer to this proposed positional embedding as Group Position Embedding (GPE). Its conceptual foundation and implementation are remarkably straightforward. In the context of document-speet specific tasks, we employ GPE to encode the coordinates of textual elements, effectively modeling layout information. The large language model, modified with GPE, does not incur any additional computa- tional overhead or alter the original model architecture; thus, it can achieve the goal of understanding complex documents with minimal fine-tuning.
Lu al. 2023) also attempt to incorporate layout information into large language mod- els. The simplest approach involves flattening the text along with its corresponding coordinate 2023) before inputting them into the model. However, this method not only increases the input length but also disrupts the original coherence of the text, and it does not neces- sarily guarantee that the model will learn the layout information embedded in the coordinate boxes. Alternatively, some methods treat coordinate box information as an additional modality (Wang et al.) 2023 Fujitake 2024; Liao et al.) 2024). integrating it with the original text before feeding it to the model. The introduction of this new modality necessitates extensive pre-training to achieve modality alignment. Other approaches (Luo et al. 2024) have introduced more complex layout information encoding network, which increase computational load in order to inject layout information. con- trast, our method preserves the original input structure of the text and employs an extremely simple approach to model layout information.
Some recent works (Luo et al. 2024 Wang et al. 2023 Liao al.
2024 Fujitake 2024
2024a Perot et al.
boxes (Perot et al.
For verification, we conducted experiments on several different base models and benchmarked our method on five mainstream documental tasks. Additionally, we introduced a more challenging Benchmark for Layout Analysis and Document Evaluation (BLADE). It is meticulously collected and designed to ensure that layout information is a critical factor for correctly answering questions. observed that method effectively enables various LLMs to
Through a series of experiments, learn layout information and demonstrates highly competitive results compared to similar methods. In summary, the contributions of our method include the following points:
1. A universal and extremely simple positional embedding called Group Position Embedding, which helps LLMs understand document layout information.
2. A new and more challenging document evaluation benchmark, BLADE, designed to assess LLMs' comprehension of complex layouts.
3. Comprehensive experiments and valuable insights into model behavior.
RELATED WORK
MLLMs for document understanding. Recently, large language models represented by Chat- Opennl 2022) have demonstrated astonishing performance in text tasks, particularly in Zero- Shot tasks. To enable L.LMs to understand more complex modal information, various Large Multi- modal Models (Liu et al.] 2023a Zhang et 2 Ye et al.] [023bb Bai et al., 2023) have shown impressive performance in adapting to different modalities. In document-related tasks, multimodal end-to-end approach (Liu et al.) 2023a Zhang et al.] 2023 Ye et al., 2023b Bai et al. 2023), which directly accepts raw image inputs and processes document image inputs in end-to-end manner. This approach requires the incorporation of an additional visual encoder and relies on costly pre-training processes to achieve
LLMs can be categorized into two types. One type employs
fine-grained alignment between the image and text modalities. The other type (Luo et al. 2024
2

---

Published as a conference paper at ICLR 2025
head
head
q
q
9
q
9
q
P
n-D
position
position
encoding
encoding
P
R-D P
k
k
k
k
k
Multi-Head Attention with previous PE
Multi-Head Atfention with our Group PE
Figure The concept of Group Position Embedding. Left: The standard multi-head attention, in which identical position embedding is applied to all heads. Right: Proposed group position embedding, which positional information differs across head groups.
Wang et al. 2023 Liao et 2024 Lu al., 2024a; Perot al. 2023 utilizes traditional OCR models to first convert document images into text and text box information before using the LLM to process this information. This latter method was widely used before the advent of LLMs and has proven effective at learning complex document layout information. Exploring how to integrate LLMs with document layout information is a research direction worth delving into. Our proposed Group Position Embedding is based on the second approach.
Layout-aware Position Embeddings. Positional embedding is a common method used in struc- tures like Transformers capture sequence positional information. Earlier sinusoidal positional encodings (Vaswani et al. (017)) were frequently used in tasks such as language modeling and machine translation. The original positional embedding
mainly designed represent the posi- tional information of one one-onneentt dee-ddmmesionnl sequences, but many tasks require two-dimensional or multi- dimensional positional information. In image-related tasks, it is necessary capture positional information in both width and height dimensions. A series of works exemplified by ViT (Dosovit- skiy 2020) achieve this by concatenating positional embeddings from dimensions represent 2D information. This concatenation method is also very common in detection tasks; for instance, in works like Deformable-DENe
(Zhu et al.] (2220). the positional embeddings of the bounding boxes are generated by encoding information for each dimension separately and then concatenating them together. In document tasks, LayoutL.M (Xu al. (020), as early pioneer, represents the spatial layout of documents by encoding the information from the corresponding bounding boxes for the text. It separately encodes each dimension and then sums the positional vectors of different dimen-
sions at the input stage. Subsequent improvements (Xu et al. 2020; 2021: Huang et al. 2022
et al. 2021b a Appalaraju et al.] 2021 Gu et al., 2021] Wang et al. 2022 Gu et al. 2022 Hong
al. 2022 Yu et al. |023] et al. 2022 Luo et al. 2023 Da
2023) based on LayoutLM
have continued to utilize similar coordinate encoding schemes. Compared with these approches, GPE is entirely new method that can be combined with any positional encoding and quickly applied to models.
3 METHOD
In this section, we will first review the commonly used position embeddings in large language mod- els. Later, we will illustrate the idea of Group Position Embedding(GPE) for LLMs and how to apply the grouping technique the commonly used position embeddings. Lastly, focusing the prob- lem of document understanding, we will demonstrate how LLM can facilitated to comprehend complex positional information.

---

Published as a conference paper at ICLR 2025
3.1 REVIEW OF POSITION EMBEDDINGS
We review two representative position embeddings. The first is Sinusoidal Position Embedding, which serves as exemplar of additive position embedding. The second is Rotary Position Embed- ding, representing the category of multiplicative position embedding.
Sinusoidal Position Embeddings. Sinusoidal Position Embeddings (Vaswani et al. 2017 are con- stant vectors that are added to token embeddings on input to the first layer of the transformer. They are pre-generated using sinusoidal function and selected based on the token index during model inference. For a given index m. the calculation process is as follows
(I) where (m denotes the Sinusoidal Embedding function. 2i or 2i + 1 denotes the index of the embedding channels and d is the dimension of the embedding. The calculated position embedding is then added to the input embedding x as P ( m) = x + PF
2 = cos(m/10000002)
Pm(m)2: - sin(m/
Rotary Position Embeddings (RoPE). Rotary position Embedding (Su et al. 2024 is the most widely used position embedding in current LLMs. It ingeniously combines relative position encod- ing with absolute position encoding and excels in length normalization. Given input and its corresponding position m, Rotary position Embedding (RoPE) is added as following steps. First, the embedding
viewed as set of complex numbers by pairing consecutive elements of the vector: x = (I1 + iæ2 33 + i.4 Id-1 + ima)) It can also be denoted as x = E (1y + (zz++)ee where ej is the j-th union direction vector. Then the RoPE for x is applied by
d/2
f(x, m) = Σ zye
imi
(2)
Based on RoPE, given query and key vector q, k, the attention score with position embedding is calculated as < f(q. fk, >= Ega a n),,
3.2 GROUP POSITION EMBEDDING
Group Position Embedding is proposed in order to feed multi-dimensional position information to transformers. As illustrated Figure in the conventional multi-head attention mechanism, all heads share common set of position embeddings. Following the intent of multi-head attention, GPE forces different heads to attend to different positional information. Given a multi-dimensional position P with dimension / and the number of attention heads h. Denote as the input embedding The computation of GPE unfolds as two steps. First, group attention heads to match the dimension of position P. We denote Gr(i) as the group mapping function, where i [1, 2, .... hi] and Gr(i) [1.2..... I]. Second, using the selected position embedding to apply the position information to each head. For Sinusoidal Position Embedding, we use P P) the position embedding function. It writes as
GroupPE(x,P.i) = MLP.(x) - PE(P Gr(s)
(3)
where i the head index and ML),(x) projects the input embedding to i-th query vector key vector. For Rotary Position Embedding, we use GroupRoP E(x, P) as the position embedding function. It writes as
GroupRoPE(x,P.) = f(MLP,(xx, ((r,)) Po)())
(4)
Grouping in a multi-head manner is a general technique and can be applied to any position em- beddings. It ensures that positional information from different dimensions holds an entirely equal status, collectively participating in the inference process of the large language model. The grouping technique is completely compatible with the original one-dimensional position encoding. For pure text input, we can just design Gr(i) (i) as an identity mapping, indicating that all heads being mapped to identical position index.
3.3 GROUP POSITION EMBEDDING FOR DOCUMENT UNDERSTANDING
In this section, we introduce the application of the GPE to the task understanding complex doc- uments. Conventionally, enable L.LMs to comprehend typical document, Optical Character
4

---

Published as a conference paper at ICLR 2025
A: The content is
Large Language Model with our Group PE
token
position encoding
idx bbox
P-D
n-D P
Q: What's 's the content the Table?
Input Doc Page
Figure 2: Group Position Embeddings for document understanding. Each token is equipped with N-D positional information, which includes its reading order and corresponding bounding box.
Recognition (OCR) techniques, among others, are first employed to extract text boxes and textual information from the document. This is followed by ordering this information according to certain rules before feeding it into the LLM for processing. However, more generic document tasks, obtaining one-dimensional sorting can be immensely challenging due to elements such tables and intricate layouts, necessitating the design of an exceedingly complex processing system. With GPE, don't need to restore the exact positional order for each token; instead, we simply assign ordering information using some straightforward strategies. For instance, we could adopt a global left-to-right sequence index tokens, directly assign them indices corresponding their positions within local bounding boxes. Both approaches have their pros and cons in practice, which will delve into further in subsequent discussions.
Specifically, given a token embedding x, we consider its positional information from two perspec- tives. Firstly, in terms of reading order, x corresponds spatial coordinates B of the bounding box where x resides. Without loss of generality. we utilize the coordinates of the top-left and bottom-right corners of the bounding box to denote its spatial position, i.e., B = [20, 30, r1, y/]. In practice, when handling spatial positions, we normalize the coordinates B, resulting in 3;
position id m. Secondly, we consider the
mari(y) mming) where À the scal- ing factor. It is set to 1000 in our implementation. Consequently, the positional information to be encoded for token embedding x can be represented as P [m. B]. For the grouping function, we map the majority of heads to the layout part, each dimension using equal heads, the remaining for the reading order. Assuming the total number of heads is 32, the Gr(i) is defined as
maz(z) maz(z))min(a)) M/:
zi-min(a)
pi-min(y)
Gr(i) = f (16-4)44111
i < < 4, 5 < < 32,
(5)
4)/7+1]
For the instruction part, only its one-dimensional positional information is considered, and this portion is encoded following the conventional manner in which LLMs process text sequences. An illustration of our method for imposing positional encoding on inputs of complex documents is provided in Figure using a document example.
Notably, there are various encoding schemes for the layout information of documents. For instance, we could choose encode all four corner points of the bounding box accommodate more complex scenarios. It will be discussed in experiments.
4 BLADE: A NEW BENCHMARK FOR COMPLEX LAYOUT ANALYSIS
We construct more challenging Benchmark for Layout Analysis and Document Evaluation, named BLADE, in response to the saturation trend seen in current common document evaluation bench- marks. BLADE focuses complex layout issues, with the majority of documents manually curated to ensure richness in layout information, thereby increasing the difficulty of assessment. Further- more, the questions formulated based on these documents have been meticulously designed to ensure
5

---

Published as a conference paper at ICLR 2025
Shitt
What
Camppiel vike Althar
www. ao00
(a)
(b)
Figure 3: (a) Samples from 6 scenarios of BLADE. (b) An example from Forms, in which several questions are listed. Answering these questions necessitates the structural info of each element, making the dataset rather challenging.
that correct answers can only be inferred with the aid of spatial layout information. Examples from illustrated in Figure Ba and Figure-33h Bb] It can be seen that they
some of these evaluation quite challenging for large language models that lack spatial layout understanding.
The data in BLADE be divided into two major categories. The first category comprises real- world data, including newspapers, web pages, slides, and various forms. The second category con- sists of synthesized data generated through rendering, involving synthetic documents and tables. For the real data, we primarily rely on manual annotation methods combined with the utilization of large language models to create question-ane -nnswwrr pairs. In contrast, for the synthetic data, question- answer pairs are constructed using template questions. For details of BLADE, refer to the appendix.
5
EXPERIMENT
5.1 DATASETS
For the training process of LLM with GPE, we collect several publicly available document un- derstanding datasets. They includes two synthetic datasets: Document Dense Description (DDD) and Layout-aware SFT used in (Luo et al.) (024) and four VisualQA Dotusets:DocQAAMMMM
jet al. 202T Info (Mathew et [l.22 CharrQA(Masry et al. 2022). VisualMRC (Tanaka ee al. )021) and three KIE SRtaSets:SROIE (Huang et al.) 2019). CORD(Park et al., 2019), and FUNSD (Jaume et al.) 2019). Besides, we also incorporate a proportion of instruction data for fine- tuning large models to combat the decline in the model's inherent capabilities after being trained on document data. For details of the training set, refer to the appendix.
5.2 IMPLEMENTATION DETAILS
For implementation, selected several representative base models apply GPE, including the Llama2-7B (Touvron et al. 2023), qwen2-7B (Yang et al. 2024), and ChatGL (GLM et al. 2024]. Although these models are all based the transformer architecture, they exhibit variations in certain details. For instance, qwen2 (Yang et al.] 2024) uses group query attention instead of standard multi-head attention. The CharGLM (GLM al. 2024) adopts prefix structure and utilize bidirectional attention mechanism for the historical tokens. Since these structural details generality. For the training recipe, we use learning rate of le-5 with 750 warmup steps and constant learning rate schedule, and Adam optimizer (Kingma) 2014) with betal=0.9, beta2=0.99 .99 and weight decay of 0.1. The maximum sequence length is set to 4096 for all base models during the entire training process. The training also involves the bf16 precision on 80GB A100 GPUs using the deepspeed training framework (Rasley et al. 2020) to complete one epoch.
are related GPE, we select them validate our approach, thereby demonstrating
5.3 RESULTS
In this section, we compare our method based on GPE with several document-orientee approaches across several commonly used benchmarks. We use M et al. 2021) as the evaluation

---

Published as a conference paper at ICLR 2025
Table Comparison with related methods on common document benchmarks. Note LLMs and ours methods are trained under identical setting. MLLMs' results are implemented using their provided weights. Results of LLMs with layout are copied from their papers. We use ANLS as the default evaluation metric for all benchmarks except VisualMRC, for which we use CIDEr.
Document VQA
VIE
Method
data amount
DocVQA
VisualMRC
FUNSD
CORD
SROIE
ChutGLM-6B
131.3
25.8
Plain Text
Llama2-7B-Chan
0.5M
66.6
335.4
62.7
75.6
88.8"
Qwen2-7B
71.8
367.4
78.1
83.4
97.7
MLLM
LLaVAR-7B
1.2M
11.6
13.6
2.4
LLaVA-1.5-1)
1.2M
13.3
Owen-VL-7BB
1.4B
47.1
30.0
58.6
TextMonkey4.
2.6M
66.7
42.9
46.2
LLM
DocLay LLM(wih image))
3.1M
72.8
310.6
80.7
79.4
84.4
LayTexttLLM-Llama2
GM
77.2
277.8
81.0
82.5
with Layout
Layoutl LL- M-MMi----n-----
6M
74.3
78.7
62.2
71.0"
D.cl.LMM-LJam2222
3.8M
264.1
67.4
ours
GPE-GLM-7R.
57.9
129.3
69.6
83.0
94.8"
GPE-Llama2-71
0.5M
77.2
318.9
77.6
84.7
97.9"
GPE-Qwen2-7B
78.1
344.0
82.6"
86.9"
1 fully zero-shot setting, only zero-shot on VIE, *test set from (Luo et al. 2024
metric for all benchmarks except VisualMRC, which we use CeDentVedantam et al. 2015) as the metric. The comparative methods can be broadly categorized into three types: the first comprising pure LLMs, for which we selected ChatGLM66 (LMM et al.] 2024). Llama2- 7B Touvron et al, 2023). and Qwen2-7B( Yang al.] (024);; the second type involves end-to-end Large Multimodal et al. (023a), Qwen-VL. (Bi et al., )023) and TextMonkey+ (Liu al.) 2024); and the third category consists of LLMs that model layout information, exemplified by LayoutLL et al. 2024). DocLLM(Wany et al., 2023). DocLayLLM(Liao et al. (224)) and LayTexLLLM (Lu al. (024bb. We implemented GPE on the above chosen pure LLMs, yielding GPE-GLM-66. GPE-Llama2- 2BB and GPE-Qwen2-7B. Note that for pure L.Mss, they are tuned under the identical setting GPE. Considering that these methods are based on different base models, trained with varying datasets and strategies, and some without publicly available model weights, the comparison is not entirely equitable. Nonetheless, valuable insights emerge from the comparative results presented in Table
Models, including LLa VAR-7B(Zhang et al. 2023).
Our findings reveal that, after integrating GPE, LLMs exhibit notable improvements across var- ious benchmarks compared to their original counterparts, affirming the efficacy of our proposed approach. Remarkably, GPE achieves highly competitive performance with significantly less train- ing data compared to other methods. Additionally, observe that end-to-end approaches fare less impressively on these benchmarks, as these tasks are inherently more challenging for such mod- els, requiring them also extract textual information from images. A noteworthy observation is the exceptional performance of the Qwen2-7B, particularly in VisualMRC, outperforming all other methods. This suggests that most of the benchmarks assess model's text comprehension capa- bility. Our analysis of these benchmarks(refer to the appendix) also indicates that scores on these benchmarks do not readily measure a model's ability to understand complex layout information. is under this context that we introduce new benchmark, BLADE, designed to address these lim- itations. Another observation is that GPE gets inferior performance compared with Qwen2-7B, means that GPE may slightly influence the LLM's text comprehension capability. This phenomenon further disccused the appendix.
5.4 COMPARISON
To ensure fair comparison between GPE and similar methods, we conduct comparative exper- iment in this section using the same base model, dataset, and training strategy. We compare GPE with three representative methods. 1. Text Box (Perot et al. 2023): This approach directly repre- sents bounding boxes as text, inserts the coordinate text into the original information text, and feeds the resulting new text sequence into the large language model input. 2. Add Box Embedding (Xu
7

---

Published as a conference paper at ICLR 2025
Table Comparison with other layout-aware methods on common document benchmarks. Methods in this table are trained under identical setting. Qwen2-7B is selected as the base model. We use ANLS as the evaluation metric for all benchmarks except VisualMRC, for which we use CIDEr.
Document VQA
VIE
Method
DocVQA
VisualMRC
FUNSD
CORD
SROIE
Raw Text
71.8
367.4
78.1
83,4
97.7
Text Box
71.5
357.5
78.4
81.5
98.0
Insert Box Embedding
301.4
71.2
63.8
90.3
Add Box Embedding GPE
71.5
350.1
78.4
79.5
87.9
78.1
344.0
82.6
86.9
97.8
Table 3: Comparison with other layout-aware methods on BLADE. Methods in this table are trained under identical setting. Qwen2-7B is selected as the base model.
Method
Tables
Forms
Slides
Websites
SynthDocs
Newspapers
Raw Text
50.3
33.0
42.2
46.1
53.3
43.8
Text Box
25.0
24.9
25.9
39,6
29.0
31.9
Insert Box Embedding
43.4
30.9
36.5
41.6
31.7
31.8
Add Box Embedding
51.4
32.8
43.2
50.3
57.6
45.1
GPE-Sinusoidal GPE-RoPE
67.2
45.6
48.3
66.5
55.2
48.7
77.3
57.2
64.5
79.3
62.2
52.2
et al. 2020): A commonly employed strategy in conventional methods, it encodes bounding boxes into vectors and adds them to the token embeddings of their corresponding text as input. 3. Insert Box Embedding (Lu et al. 2024a): Similarly encoding bounding boxes into vectors, it differs from LayoutLM by inserting the encoded position vectors into the original token embedding sequence. For GPE, we also implement the Sinusoidal version GPE-Sinusoidal for comparison. We select Qwen2-7B as the base model. The results of this comparison are shown in Table and Table-3 can observed that GPE demonstrates remarkably strong performance across these document tasks. Under our training setup, the Text box and Insert Box Embedding struggles to function effec- tively, primarily because they alter the input format of the LLM, hence, necessitating an expensive pre-training stage when applied practically. Besides, these two methods have to process longer input sequence compared to GPE and the method of Add Box Embedding. Add Box Embedding achieved second-best performance, but it is clearly weaker than the GPE method. We argue that this mainly because Add Box Embedding effectively changes the original input vector space, requiring the model adapt to new modality during training. In contrast, GPE directly reuses the original although
position embedding space, making it easier for the model learn. For GPE-Sinusoiin uses different type of position embedding with the base model, it still achieves second-best result. This implies that the grouping manner is a better way to incorporate layout information than the other methods.
5.5 ANALYSIS
For the following parts, Influence of Reading Order.We investigate the impact of reading order part in GPE on model performance, i. the design of in the sequence representation P [m, B). We compare sev- eral designs for reading order as follows. W/O: Solely utilize the positional information of text boxes without imposing any specific reading sequence. Left2Right: Follow the natural lefhhto-right, top-to-bottom order as humans typically read. XYCut: Following(Gu et al. (2022), obtain proper reading order for rich layout document. Random: Randomly shuffle the order of text boxes to arrange tokens in random sequence. Local: Treat each text box equally in this method, with all tokens within a box encoded sequentially starting from 0, ignoring their spatial relationship with other boxes. Considering that the attention structure also influences how reading order is handled. we select two base models for experiments: GLM and qwen2. The results are shown Table Based on the results, observe general concordance in trends between models employing the GLM and Llama2 architectures. Approaches that omit position encoding or utilize random posi- tion encoding exhibit inferior performance. The methods that excel are Local and Left2Right,
Llama2-7B as the base model if not particularly mentioned.
8

---

Published as a conference paper at ICLR 2025
Table 4: Ablation on the design of Reading Order representation.
Llama2
GLM
Method
SynthTables
Forms
SynthDoes
Newspapers
SynthTables
Forms
SynthDoes Newspapers
W/O
2.4
1.7
16.5
2.5
46.6
8.6
39.9
12.6
Random
63.9
31.8
63.0
26.7
66.0
15.9
47.0
32.3
Local
77.3
41.6
70.7
28.4
71.7
32.2
54.1
29.3
Legh2
76.4
38.7
72.7
35.7
69.8
21.2
56.2
40.3
XYCut
71.8
39.8
60.2
31,3
66.9
24.9
52,5
39.4
Table 5: Ablation on the representation methods of layout information on BLADE.
W/O Rotation
w/ Rotation
Setting
SynthTables
Slides
Websites
SynthTables
Slides
Websites
LeftPoint
76.0
36.9
55.1
73.6
31.5
51.6
Rectangular
76.4
38.7
60.5
74.3
38.8
71.6
Quadrilateral
77.1
42,9
56.4
75.1
40.7
57.4
with each demonstrating contrasting superiority across different datasets. Specifically, Left2Right excels on SynthDocs and Newspapers, while the Local method outperforms on SynthTables and Forms. We think that these results are influenced by the distinct characteristies of the respective do- mains. SynthDocs and Newspapers predominantly comprise natural language paragraphs, in which the sequential order of words holds significant importance. Although Left2Right may still occa- sionally disrupt the reading sequence, largely preserves sequential information, a feature crucial for understanding narrative coherence these text-heavy contexts. Conversely, SynthTables and Forms typified by tabular structures where the reading order among fields is less critical. Here, the Local approach proves advantageous as it enables the model to more effectively leverage spatial information, a key aspect in deciphering structured data arrangements common tables and forms.
Design of Layout Information. We focus on the impact of design of B P [m. B]. We compare three design approaches. LeftPoint: Utilize the coordinates of the top-left corner, where B = [v0. yo]. with (zo, yo) representing the position coordinates of the top-left corner. Rect- angular: Employ the coordinates of both the top-left and bottom-right comnrr,, expressed as B = [zo. yo. V1]. where Bi denote the position coordinates of either the top-left (i 0) or bottom-right (i 1) corner. Quadrilateral: Involve the coordinates of all four corner points, formatted as B [xo, 21, 21, T2, U2, Z3. Vs]. arranged in a clockwise order, with (I,, yi) being the coordinates of each corner point. In each configuration, normalization of coordinates con- sistently applied. We employed two conditions, with and without visual rotation, to investigate the impact of coordinate representation. Under the rotated condition, we applied random rotations to the entire evaluation set to emulate the effects of real-world shooting angles. The results depicted in Table Our findings reveal that the LeftPoint method yields the lowest performance across all eval- uation benchmarks, although suggests that a substantial portion of layout information within documents is indeed encapsulated by the top-left corner point. The Rectangular and Quadrilateral meethods are largely comparable. No- tably, we observed in our experiments that the Rectangular approach converges faster. Regardless of the rotational condition, Quadrilateral marginally outperforms Rectangular, though this advantage is not pronounced. These observations imply that, for the majority of document-relaan tasks, adopting the Rectangular method is sufficiently effective in achieving commendable outcomes.
significantly underperforming compared to other approaches. This
Design of Grouping F Function As previously mentioned. the dimension of P [m, B] and the number of heads in the model typically not align. Consequently, a grouping function is required to map multiple heads to their corresponding position information. In this part, we explore the effects of varying grouping methodologies. Several straightforward Grouping Functions are: PadZero First expand the position vector P by appending zeros till its dimension match the number of attention heads, then establish a one-to-one correspondence between them; More Reading Order Map the majority of heads to the reading order, the rest heads to the layout infomation; More Layout Map the majority of heads to layout information, the rest heads to reading order. The results are depicted in T In terms of results, the strategy of allocating more layout information to the group turns

---

Published as a conference paper at ICLR 2025
Table 6: Ablation on the selection of group function on BLADE.
Group Function
SynthTTabls
Forms
Slides
Websites
SynthDucs
Newspapers
Baseline
24.5
11.1
18.9
23,8
72.5
31.3
Pad7Zmm
2.8
10.7
1.4
2.0
11.8
1.6
More Reading Order
76.3
33.4
50.7
65.6
74.0
35.7
More Layout
76.4
38.7
56.2
70.5
72.7
35.7
2337
11:40
AHAC aateaiia 2000
-C Succteiii 2018
(a)
(b)
Qualitative results on BLADE from Forms and Newspapers. (a) The question requires list the whole column corresponding Candidate, only GPE outputs the correct answer. (b) To correct answer the question, model needs to focus on the correct paragraph. Only GPE achieves this.
Figure 4:
out to be more advantageous, performing better across nearly all evaluation sets. Coupled with the conclusions in Table-4 Reading Order information is also identified as indispensable.
Qualitative Results. We selected two samples from BLADE, and the output results from GPE and other methods are depicted Figure In the example from Forms, where complete response to the "Candidate" column is required. Other methods either provide additional irrelevant informa- tion or miss out on details, whereas only GPE accurately furnishes the answer. In the Newspapers instance, comprehension of the content within specified area is necessary to answer the question, and again, it is solely GPE that provides the correct response. These two examples illustrate that, with the aid of GPE, LLMs precisely locate the relevant sections, disregarding distractions from unrelated information, a feat unachievable without the assistance of layout information.
LIMITATION
While GPE shows promise, several areas remain unexplored. First, although GPE is applicable to any task requiring multidimensional positional encoding, this paper focuses only on document- centric tasks. Graph-structured data could also benefit from similar grouping strategies. Second, most methods, including ours, face decline in inherent capability, possibly due to structural changes or the quality of document-specifie training data. Our approach minimizes alterations to the base model, and we believe future iterations of GPE will further address these challenges.
7 CONCLUSION
We introduce Group Position Embedding (GPE), which enhances LLMs' ability to understand mul- tidimensional positional information, particularly in document-related tasks. GPE allows large mod- els to comprehend complex document layouts by injecting independent positional information into grouped attention heads without altering the model architecture or input format. This method outper- forms similar approaches. To evaluate large models' understanding of complex documents, we de- veloped BLADE, challenging benchmark that emphasizes layout information in both data sources and question design. Extensive validation on mainstream document tasks and BLADE demonstrates the effectiveness of our approach.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Srikar Appalaraju, Bhavan Jasani, and Bhargava Urala Kota. DocFormer: End-to-end transformer for document understanding. In /CCV, pp. 4171-4186, 2021.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint ar a0w.::230...2000000 2023.
BELLEGroup. Belle: Be everyone's large language model engine. htttps://github.co com/
LianjlaTechhBBII.. 2023.
Hiuyi Cheng, Peirong Zhang, Sihang Wu, Jiaxin Zhang. Qiyuan Zhu, Zecheng Xie, Jing Li, Kai Ding, and Lianwen Jin. M6doc: A large-scale multi-format, multi-type, multi-layout, multi- language, multi-annotation category dataset for modern document layout analysis. CVPR, pp. 15138-1555 2023.
Yiming Cui, Ting Liu, Wanxiang Che, Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang. and Guop- ing Hu, A span-extraction dataset for chinese machine reading comprehension. arXiv preprint arXiv:1810.07366. 2018.
Cheng Da, Chuwei Luo, Qi Zheng. and Cong Yao. Vision grid transformer for document layout analysis. In ICCV, pp. 19462-19472, 2023.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Masato Fujitake. Layoutllm: Large language model instruction tuning for visually rich document understanding. arXiv preprint arXiv:2403. 14252, 2024.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, al. Chatglm: family of large language models from glm- 130b to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
Jiuxiang Gu, Jason Kuen, Vlad Morariu, Handong Zhao, Nikolaos Barmpalios, Rajiv Jain, Ani Nenkova, and Tong Sun. Unified pretraining framework for document understanding. In NeurlPS, 2021.
Zhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, and Liqing Zhang. Xylayoutlm: Towards aware multimodal networks for visually-rich document understand- ing. In CVPR, pp. 4583-4592, 2022.
Jiabang He, Lei Wang, Yi Hu, Ning Liu, Hui Liu, Xing Xu, and Heng Tao Shen. Icl-d3ie: In-context learning with diverse demonstrations updating for document information extraction. ICCV, 2023.
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, et al. Dureader: a chinese machine reading comprehension dataset from real- world applications. arXiv preprint arXiv: MiXiv:111..00001 2017.
Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, and Sungrae Park. Bros: A pre-trained language model focusing text and layout for better key information extraction from documents. In AAAI, 2022.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. LayoutImv3: Pre-training for document ai with unified text and image masking. In ACM Multimedia, 2022.
Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawa- har. lcdar2019 competition on scanned receipt oer and information extraction. In 2019 Interna- tional Conference on Document Analysis and Recognition (ICDAR), pp. 1516-1520. IEEE, 2019.
Guillaume Jaume, Hazim Kemal Ekenel, and Chan-Philippe Thiran. Funsd: dataset for form understanding in noisy scanned documents, 2019.

---

Published as a conference paper at ICLR 2025
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Chenliang Li, Bin Bi, and Ming Yan. StructuralL Structural pre-training for form understanding. In ACL, 2021a.
Peizhao Li, Jiuxiang Gu, and Jason Kuen. SelfDoc: Self-supervised document representation learn- ing. In CVPR, pp. 3652-5660 2021b.
Xinyu Li, Fayuan Li, Lu Pan, Yuguang Chen, Weihua Peng, Quan Wang. Yajuan Lyu, and Yong
Zhu. Duee: scage dataset for chinese event extraction in real-world scenarios. In Natural Language Processing and Chinese Computing: 9th CCF International Conference, NLPCC 2020, Zhengzhou, China, October 14-18, 2020, Proceedings, Part II 9, pp. 534-545. Springer, 2020.
Yulin Li Yuxi Qian, Yuechen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, and Errui Ding. Structext: Structured text understanding with multi-modal transformers. In ACM Multimedia, pp. 1912-1920, 2021c
Wenhui Liao, Jiapeng Wang. Hongliang Li, Chengyu Wang, Jun Huang, and Lianwen Jin. Do- clayllm: An efficient and effective multi-modal extension of large language models for text-rich document understanding. arXiv preprint arXiv:2408. 15045, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023a.
Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Yang Liu, Biao Yang. Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models, 2023b.
Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473, 2024.
Jinghui Lu, Haiyang Yu, Yanjie Wang, Yongjie Ye, Jingqun Tang, Ziwei Yang, Binghong Wu, Qi Liu, Hao Feng. Han Wang, al. A bounding box is worth one token: Interleaving layout and text in large language model for document understanding. arXiv preprint arXiv:2407.01977 2024a.
Jinghui Lu, Haiyang Yu, Yanjie Wang, Yongjie Ye, Jingqun Tang, Ziwei Yang, Binghong Wu, Qi Liu, Hao Feng, Han Wang, al. A bounding box is worth token: Interleaving layout and text in a large language model for document understanding. arXiv preprint arXiv:2407.01978 2024b.
Chuwei Luo, Changxu Cheng, Qi Zheng, and Cong Yao. Geolayoutlm: Geometric pre-training for visual information extraction. In CVPR, pp. 7092-7101, 2023.
Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, and Cong Yao. Layoutllm: Layout
instruction tuning with large language models for document understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15630-15640, 2024.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque Chartqa: bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint Stti:2203.1122444 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa on document images. In WACV, pp. 2200-2209, 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvga. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Linyong Nan, Chiachun Hsich, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojeiech Kryšciński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, et al. Fetaqa: Free-form table question answering. Transactions of the Association for Computational Linguistics, 10:35 49, 2022.
12

---

Published as a conference paper at ICLR 2025
OpenAl. Introducing http//////////ooaaoo/o/////////a 20222
2022.
Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. {CORD}: A consolidated receipt dataset for post- (ocr) parsing. In Workshop on Docu- ment Intelligence at NeurlPS 2019, 2019. URL https: /opprrev/eee net forum?rummmmm
S31 05132659UF
Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang. Zhengjie Huang, Teng Hu, Wei- chong Yin, Yongfeng Chen, Yin Zhang, Shikun Feng, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-layout: Layout knowledge enhanced pre-training for visually-rich document under- standing. In EMNLP findings, 2022.
Vincent Perot, Kai Kang, Florian Luisier, Guolong Su, Xiaoyu Sun, Ramya Sree Boppana, Zilong Wang, Jiaqi Mu, Hao Zhang, and Nan Hua. Lmdx: Language model-based document information extraction and localization. arXiv preprint arXiv:2309. 10952, 2023.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for squad, arXiv preprint arXiv: 1806. 03822. 2018.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 3505-3506 2020.
Yongxin Shi, Dezhi Peng, Wenhui Liao, Zening Lin, Xinhong Chen, Chongyu Liu, Yuyi Zhang, and Lianwen Jin. Exploring capabilities of GPT-4V(ssion): A quantitative and in-depth evaluation. arXiv preprint arXiv:2310.1680 2023.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En-
hanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehension on document images. In AAAL, volume 35. pp. 13878-13888, 2021.
Tianchi. Ccks2021, 2021.
URL
https://tianchi http://iaaaa..allll.a...ocoo/// .ll.yunccoomdaatssst....
dataDerai1PPttaadd1111
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-basee image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Bingning Wang, Ting Yao. Qi Zhang, Jingfang Xu, and Xiaochuan Wang. Reco: A large scale chinese reading comprehension dataset on opinion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 9146-9153, 2020.
13

---

Published as a conference paper at ICLR 2025
Dongsheng Wang. Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, and Xiaomo Liu. Docllm: layout-aware generative language model for multimodal document understanding. arXiv preprint arXIv:2401.00008, 2023.
Jiapeng Wang, Lianwen Jin, and Kai Ding. Lilt: A simple yet effective language-independeni layout
transformer for structured document understanding. In ACL, 2022.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, et al. Mmlu-pro: more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01 0754,, 2024.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. Layoutimv2: Multi-modal pre-training for visually-rich document understanding. In ACL, 2021.
Yiheng Xu, Minghao Li, Lei Cui, and Shaohan Huang. LayoutLM: Pre-training of and layout for document image understanding. In KDD, 1192-1200, 2020.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
Jianxin Yang.
2023.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. mplug-docowl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.00 020499, 2023a.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang. Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality, 2023b.
Yuechen Yu, Yulin Li, Chengquan Zhang, Xiaoqiang Zhang, Zengyuan Guo, Xiameng Qin, Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang. Structextv2: Masked visual-textual prediction for document image pre-training. In ICLR, 2023.
Yanzhe Zhang. Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306. 17107, 2023.
Xizhou Zhu, Weijic Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04/553 2020.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.I DATASETS DESCRIPTION
The training set in this work is composed of the following several parts.
LayoutLLM-awaare SFT Data(Luo et al., 2024) The layout-aware SFT data for LayoutLLM is gen- erated by GPT-3.5 Turbo and converted from existing text-based Machine Reading Comprehension (MRC) datasets.
ChartQA Dataset Masry et al. 2022 This dataset encompasses 9,600 manually crafted questions along with 23, 100 questions generated from manually written chart summaries, aiming address complex problems involving visual and logical reasoning.
InfographicVQA Mathse et al. 2022) Designed for the Visual Question Answering (VQA) task on infographics, this dataset comprises around 10,000 images, with an average of about 10 questions per image, each question associated with approximately 4 to 5 answers. Infographics are graphical representations of information, data, or knowledge designed to make complex infor- mation more comprehensible, including various types such as charts, flowcharts, and maps.
Doc Pataset(Mather) et al. 2021 Specialized for document visual question answering tasks, this dataset focuses on textual and graphic elements within documents, assessing machine learning models' performance in understanding and answering questions related to document content. It includes roughly 11 1,000 images and 40,000 questions.
Dataset Tanaka al., 2021 Given question and document image, this dataset quires machines to read and comprehend the text within the image and answer in natural language. Compared to existing VQA datasets that involve image text, VisualMRC emphasizes nurturing natu- ral language understanding and generation abilities more. It consists of over 30,000 question-answw pairs abstracted from more than 10,000 document images sourced from diverse web domains.
MRC
FUNSD Dataset(Jaume et al. 2019) Comprising 199 fully annotated real scanned forms with 9,707 semantic entities and 31,485 words, this dataset organizes tables by semantic entity lists. Each entity is identified uniquely, tagged (as question, answer, title, or other), bounded by box, accompanied by a list of relations to other entities, and a word list. The dataset split into 149 training and 50 testing samples.
SROIE (autaee(Huang et al. 2019) This dataset contains 626 receipts for training and 347 for testing. Each receipt is organized list of bounding- boxed text lines. Four types of entities are labeled on each receipt: company, date, address, and total.
CORD Dataset et al., 2019) Created for post-OCR parsing, this is the first public dataset containing 11. 11,000 receipt images from Indonesian shops and restaurants. Collected through crowd- sourcing, each image was initially annotated using web-based tool and then reviewed for accuracy and adherence to annotation guidelines. To avoid unintentional disclosure of sensitive personal data, sensitive information like credit card numbers full names was blurred in the final receipt dataset.
In addition to the document-related public datasets mentioned above, we have also gathered extra data for fine-tuning LLM instructions. This additional data includes BELLE-CN (3.5 million (BLLL 2023), FireFly (1.1 million entries)(Yang 2023). OpenOread Yang 2023), and ShareGPT (90k entries from public). We start by combining these instructions and then randomly select portion of them to mix with the document data. Through experimentation, we have found that mixing ratio of document data text instruction data at 2:1 achieves well-balanced effect.
A.2 DETAILS OF BLADE
Overview. BLADE comprises six scenarios, SynthTables, Forms, Slides, Websites, SynthDocs, and Newspapers, totaling 3,007 documents and 7,924 question-answer pairs. The specific data volume for each scenario is illustrated in Table-Z] The construction of BLADE involves both manual creation and leveraging large language models. The images for Forms are sourced from the (Nan et al.) 2022). and those for Newspapers come from the (Cheng et al., 2023). The images for Slides and Websites are obtained through web crawling. The corpus for SynthTables originates from entity names and values extracted from Chinese entity extraction datasets, including (Li et al. 2020). Tianchi 2021
15

---

Published as a conference paper at ICLR 2025
The corpus for SynthDocs is derived from public reading comprehension datasets, including He et al. 2017 CCCui et al. 2018). (Rajpurkar et al. 2018), Wang et al. 2020).
Year
Series
Role
Genre
2013
Dirt Cheap
Ron Furick
Comedy
2012
Any Questions for Ben?
Brand Manager
Comedy
2011
It's Awkward (short)
Hero
Comedy
2006
Alex and Alexa (short)
Alex
Drama
Scheme Ask for all values corresponding header What corresponding content of 'Gnne'' the Table?
Scheme Ask for value header when value header A Q: When the 'Year' calumn is '2011', 1 what is the correspancling value the 'Rale"?
A: Hero
Scheme 3: Ask for the content of the nth row.
Q: "In the table, excluding the header row, what is the content of the 2nd row?"
A: 201 12yAAyy Questions for Ben?InBrand ManagerinComedy
Scheme Ask for the value the nth and mth column the mth column and nth
"What the 2nd 'Role'?" "What 2nd value corresponding 'Role'?"
Brand Manager
Figure 5: Types of questions from SynthTables.
Data construction. For the data scenarios of Forms, Slides, and Websites, the predominant method employed was manual annotation, where annotators would select question and answer pairs based on image information, followed by a manual screening process to filter out challenging question- answer pairs. In the case of Newspapers and SynthDocs, given their characteristic of extensive text blocks, a different approach was adopted, utilizing the power of large language models for generation. Specifically, this involved first identifying key segment within the text, then tasking the large model with constructing question-answer pairs based on that key segment using predefined templates. This way, question-answe pairs that met the required criteria synthesized.
The distinction between Newspapers and SynthDocs lies in the former's 's need for an initial manual selection of a text segment from the document, whereas SynthDoes, being synthetically generated, allows for an automated process of segment selection, which is then fed into the large model's pipeline for generating answer pairs. This is because given key segment text required in advance for LLM to generate QA pairs. Newspapers are typically documents with complex layouts, making difficult to obtain the correct reading order through simple OCR sorting. Therefore, we first have annotators to select one key segment, usually small paragraph, from which the reading order within this small paragraph can be obtained through simple OCR sorting. SynthDocs uses synthetic documents, for which the reading order of the text is already known, thus climinating the need for manual annotation. The generation pipeline of SynthDocs is shown in Figure-
For Santesables, also designed automated pipeline to generate table-question-ansserr pairs. Specifically, we begin by constructing a set of synthetic tables based on the collected entity data. Following this, SQL employed to generate series of question templates on these tables. The ques- tions can be categorized into 4 types, shown Figure Lastly, we execute the question templates to obtain answers. Through this process, get substantial number of table-question-anste pairs.
16

---

Published as a conference paper at ICLR 2025
Coupus From MRC
25ITE , λ RE Hit gea 5 12rg Eix M - EXB...... 1 MEEE M 2
ED0 E #00 m
T- ET........
However, the body strudure of leedies and their dose relatives very consslen: in this group.. indiviual caviies are far apart and run through the entire body. They are the main blood vessels. although they are side by side. up and down....
Image with comolex layout generation
QA Generation By LLM Self- Instrutttt
Prompt Example
ARCTISNNO, MERLTH OR GORTIR
F Corpus) HY NOE T 10 Prompt:
AELLR
The given text as follows (corous) Please propose two questions based ontis lexi, with arswers less then words
KERONGL
BAXK:
LLM Answer
Query:
GRRRR
Anaswer: ' KE E
X
Query: "What are the characterisics the body structure of leeches and their close relatives and how are they different fomo link animals?' Answer: 'Leeches and their elostvelstvees are not separated, which differert from other link animals."
Sorted OCR From Top Lofk To Bottom Right
<bbox <x01> <y01> <x02> <x02>
</bbox text
<bbox exils eytls <x12> <x12s
</bbox textl
ebbox <x21s ey2's <022> <x22>
</bbox text2
Final Data
Instruction: ( Ri FI FAI
19* 105t E a f m. To i FEt. (.. BDeY d20 T.
Abmm " F ii 80
FI:: K Ab
OuS K felt M. SH EM Fab hi
(o Fi ']
Bbox 11O,C, 00, 6, 242, 45), 10,0, 0, O]]
Instruction:
I'Referring :o the following paragraphs, answer the following questions. Howewr, leeches and their indivdual casities are for apart and run through the entire body. They are the main blood "f animals have neither jaws " body structure of their down, however, they are and there irreversbility ... Question: What the dharacterist the body structure of leeches and their dose relatives, and how are they diffrent from other link anima s? ']
relatives tubes. although they side by side, up and
Bboxt [1O,0, 0, 0j, [32,6, 6, 242, 45), [0,0, 0, OJ]
Figure 6: The data generation flowchart of SynthDocs.
Table 7: Statistics of BLADE.
params
SynthTables
Forms
Slides
Websites
SynthDocs
Newspapers
#docs
1110
162
111
174
1000
450
#qas
2287
296
518
732
2000
2091
These synthetic tables are subsequently rendered into images, with the detailed procedure outlined in Figure-Z
Quality Verification. For the SynthDocs data, given that the QA pairs are generated by a LLM, there may be instances of hallucination. To ensure the quality of these generated questions, utilize LLMs for screening process during this phase. The quality check primarily focuses on assessing the relevance between the generated questions and answers in relation to the provided text
17

---

Published as a conference paper at ICLR 2025
Collect entity names and entity values pairs
"
~ "
:
"L 7
Drop entity values randomly
LA
TN
Table Image with comple ex layout generation
SQL Generation
BEARS
2
"SQL" "SELECT 4 FROM
table name WHERE 'M
"SQL RES":
"M
EASEEEI
*
Final Data
Instruction: T パ 2 RJ, バリ |
"PR 3
Bbox: [[O, 0, 0, 0], [32, 6, 242, 45),
[0, 0, 0, O]
Figure 7: The data generation flowchart of Synth Tables.
content, as well as determining whether any hallucinations have occurred. The specific prompt used for quality screening is as follows:
18

---

Published as a conference paper at ICLR 2025
Prompt for quality verification
Given a text: (text), there is question and its corresponding answer, question: {question), answer: [answer]. Please make judgments on the following aspects:
1. Can the question be answered solely hased on the content of the material? Please only respond with "Yes" or "No".
2. Is the answer given based on the provided text? Please only respond with "Yes" or "No".
3. Is the answer correct? Please only respond with "Yes" or "No".
Return in the format: mm {{ 1: "Yes/No", 2: "Yes/No", 3: "Yes/No" }} mm
A.3 ANALYSIS OF COMMON DOCUMENT BENCHMARKS
In this section, we discuss the characteristics of several commonly used document evaluation bench- marks, specifically two Document Visual Question Answering (VQA) datasets, Voc and Vi- sualMRC, as well as three Visual Information Extraction (VIE) datasets, FUNSD, CORD, and SROIE. Based a model's capability to understand layout information. A robust text large language model, such as qwen2, can generally achieve high scores on these tasks. Our proposed method, GPE, offers limited improvement on top of such a model, and in the case of VisualMRC, including our approach, various methods we implemented to augment layout information actually led to performance degradation, which we attribute to potential impairment of the model's text comprehension abilities.
previous experiments, the majority of these evaluation tasks fail to assess
To fully comprehend the implications behind the numerical variations across these five benchmark datasets, we first examine from the sample level the demands they place on model capabilities. As illustrated Figure and and Figure 9 Doc encompasses a variety of photographed documents, where we find that while some questions may necessitate the integration of layout information, the majority can be answered without it, relying solely on the understanding of text paragraphs, as exemplified in samples F a). VisualMRR is collection of various multimedia documents, where accurately answering questions does not require layout information. FUNSD consists of form-like data, similar to VQA, yet still presents instances where questions and answers are so straightforward that layout understanding is unnecessary for correct responses. CORD and SROIE involve diverse receipt types, with questions predominantly focused on recognizing entities within receipts.
Evidently, intuitively analyzing these five benchmark sets reveals that except for minority of ques- tions in Doc and FUNSD that might benefit from layout comprehension, the vast majority of tasks, in principle, can be readily tackled with sufficiently powerful text understanding capabilities. Particularly, VisualMRC is fundamentally a reading comprehension task based on large segments of natural text, assessing model's ability extract and summarize information from paragraphs. This underscores the motivation behind introducing BLADE, to provide a more nuanced evaluation that goes beyond simple text comprehension and truly examines a model's capacity to understand the layout and structure inherent in document data.
A.4 TYPE OF POSITION EMBEDDING
In this section, we investigate the impact of the difference between the positional encoding used by GPE and the model's native positional encoding on the performance of downstream tasks. In previ- ous experiments, we implemented two types of encoding: GPE-RoPE and GPE-Sinusoidal. While GPE-Sinusoidal also brings improvements in document tasks, it does not match the performance boost achieved when GPE-RoPE employs the same encoding scheme as the model. Here, we ex- periment with altering the base scale of GPE-RoPE and observe its performance on BLADE The results, summarized in Table show that scale of le6 what the model originally utilizes.
We observe that, on benchmarks akin to reading comprehension question-answering tasks such as Newspapers and SynthDocs, settings close to the original scale yield better performance. For sev- eral other categories of tasks, scales approximately around the base scale, like le5 and le7, exhibit comparable performance. However, significantly deviating from this range, as seen with 1e4, leads to noticeable declines across all tasks. Our experiments have consistently revealed that the design of GPE has differential effects on two types of tasks: those involving extensive natural paragraphs, which emphasize understanding natural language, and those filled with tabular data, which empha-
19

---

Published as a conference paper at ICLR 2025
Mat Matthow Lesko
Above Average
the antiesatc and evurgeme
Leake's Law
4
specitio quene
Question: What code for?
Answer: Above Average
(Marthem
Question: What an important factor in assuring accuracy?
Answer: The design of volumetric apparatus
wan gnensly eepiret.
inportet taster
The design of volumetric apparatus the lenath of
Question: What "Lesko's Law" referring to?
(b)
Figure 8: Samples from DocVQA and VisualMRC. (a) DocVQA comprises a variety of pho- tographed documents.Some questions may necessitate the integration of layout information (the upper case), the majority can be answered without it (the bottom case). (b) VisualMRC is col- lection of various multimedia documents, Most of the designed question does need the layout information.
Question: What is the total price?
Question: What is in the filter production field?
20"
Answer: 31.0
Answer: MFG
0000
(b)
TRADDNNG
Question: What is the "company" the given document?
legoostbiiitt
Thtacco Bland
Zemsiter Boougas
47ic Ialarge
Filter Production
Making Racking
Janns
Answer:
TSH POWER
HARDWARE TRADING
(c)
Figure 9: Samples from FUND, CORD and SROIE. (a) FUNSD consists of form-like data. Some questions may need layout information, others not. (b) CORD involves diverse receipt types. Most questions are designed like entity recognition. Layout information is unimportant for accomplishing this (c) SROIE is similar to CORD.
20

---

Published as a conference paper at ICLR 2025
Table 8: Ablation on type of position embedding. Qwen2-7B is selected as the base model.
setting
SynthTTbless
Forms
Slides
Websites SynthDocs Newspapers
Baseline
50.3
33.0
42.2
46.1
43.8
UPE--Sinuodal
67.2
45.6
48.3
66.5
55.2
48.7
GPE-RoPE with base scale le4
77.4
54.4
59.8
79.1
58.9
48.1
GPE-RoPE with base scale le5
80.9
60.2
63.4
75.3
61.4
49.7
GPE-RoPE with base scale le6
77.3
57.2
64.5
79.3
62.2
52.2
GPE-RoPE: with base scale le7
77.1
58.7
63.0
76.6
61.6
49.0
size understanding layout relationships. This suite of experimental outcomes suggests that GPE can be optimized for these two types of tasks.
In this work, we adopt uniform encoding scheme across all heads by default. Exploring more suitable encoding designs separately for the reading sequence component and the coordinate box component, aiming for even more advantageous GPE configuration, is left as direction for future research.
A.5 EFFICIENT TUNING USING LORA
In previous experiments, employed full parameter training for our model. Under the setting of full parameter updates, GPE outperforms comparable methods with significantly less data, achiev- ing superior results within mere few tuning steps. Nonetheless, conducting full parameter updates on large models remains highly resource-intensive proaches in industry that are based on LoRA, which aim conserve GPU memory and expedite training by updating only portion of the model's parameters. The trade-off is that the final perfor- mance of the model may not match that of full parameter training; however, this discrepancy varies across different scenarios. In this section,
Currently, there exist lightweight training ap-
integrate the training methodologics of GPE with LoRA and examine the model's performance at different steps on BLADE, as illustrated in Figure- Our findings reveal that, compared full parameter training, the LoRA approach requires longer training duration to reach an equivalent level across all scenarios. Nevertheless, considering that LoRA training for wewllioo-pprameeee model can be executed on single 80GB A100 GPU, the combination of GPE with LoRA presents an efficiently applicable solution in most practical
10
scenarios.
Synth
Forms
Slidcs
SOVeIANLS
Synth
400
80
40
1003
4300
5002
6500
7030
4000
5350
1000
7003
6000
70-00
Websstte
Synth
ScORI/MALS
Sco0n 53
SCOPL/ALLS
60
60
Figure 10: The comparison between full parameter tuning and LoRA- LoRA-Tuning on BLADE.
21

---

DocVQA Published as a conference paper at ICLR 2025
A.6 INFLUENCE ON BASE MODEL CAPABILITY
We observe that the capabilitiiss of foundation language models are significantly influenced by the document training data they are subjected to. Through experimentation, we have found that this is primarily due to the quality of the document data. Using Qwen-7B as our base model, we mixed tex- tual instruction data with document-type data and adjusted the blending ratio to observe the model's et al., 2024). We discovered that the untrained original Qwen
performance on MMLU-Pro model achieved the best score 39.55 on MMLU-Pro, but once document-type data was incorpo- rated, the model's 's text capability document tasks increased, the decline became more pronounced, aligning closely with the trend ob- served in GPE. Additionally, under the same data mixing ratio, the performance degradation induced by GPE compared to the original Qwen was not substantial, suggesting that the changes introduced by GPE in position encoding have minor impact on the model's fundamental abilities. This ex- periment also revealed that the training data from various document tasks currently integrated can affect the model's inherent capabilities, aspect that seems have escaped widespread attention. We believe that this represents a highly promising direction for further research.
notably impaired. Furthermore, the proportion of mixed
Table 9: Influence of data mix ratio on base model's capabilities. The data ratio is organized as textual instruction data vs document data
Setting
MMLU-Pro
GPE with mixed data ratio 1:2
26.8
76.0
GPE with mixed data ratio 1:1
28.9
72.2
GPE with mixed data ratio 2:1
31.2
70.8
Qwen with mixed data ratio 1:2
27.1
Qwen with mixed data ratio 1:1
28.4
Qwen with mixed data ratio 2:1
31.6
Qwen-7B IB-nstrutt
39.6
A.7 IMPACT OF SCALING FACTOR
In this section, we explore the influence of the scaling factor lambda. We use 1000 in our default implementation. The results are shown in Table-10 It is observed that, with the exception of SynthTables, the variations in performance across the other datasets are relatively minor. However, for Table,, higher lambda leads significant improvement in performance. We find that this enhancement can be attributed to the dense arrangement of text boxes within SynthTables. these tables, up 10 text boxes often placed in single row, and their slight positional differences crucial for determining their column assignments.
By increasing the scaling factor, these subtle positional distinctions become pronounced, thereby aiding the model in more accurately distinguishing between different cells. This finding underscores the importance of fine-tuning the scaling factor, especially when dealing with datasets that have densely packed and closely positioned elements.
Table 10: The influence of scaling factor.
A
Tables
Forms
Slides
Websites SynthDocs Newspapers
100
53.3
55.7
59.1
64.3
62.1
51.5
500
72.1
56.6
66.3
75.7
61.3
52,9
1000(default)
77.3
57.2
64.5
79.3
62.2
52.2
5000
86.3
56.6
66.6
79.5
61.8
53.6
22

---

Published as a conference paper at ICLR 2025
A.8 INTUITIVE ANALYSIS OF GPE
The key technique that enhances the effectiveness of GPE is its ""hea--peciffic design**. This de- sign ensures that each dimension of layout information is processed by each attention head. The benefits of this approach are akin to those of multi-head attention, as it encourages the model to learn positional relationships from various perspectives. By distributing the layout relations across multiple attention scores, rather than relying on single score, the model can more effectively cap- ture and utilize different aspects of the layout. Intuitively, some heads might focus on horizontal (left-to-right) relationships, while others concentrate on vertical (up-to-down) relationships.
To intuitively illustrate the impact of GPE on different attention heads, we use heatmaps to visualize the influence of positional information across various attention heads. Specifically, we construct grid sequence of size 256x256, where each grid point corresponds to a token input into the LLM,, and its spatial position is represented by the grid coordinates [ar, y). We define the reading order of the grid points using left-to-right, top-to-bottom scanline sequence. This way, we establish the positional information P [m, for the grid points. We then encode P using GPE-RoPE with an embedding dimension of 128. use the grid point at the bottom right corner as the query and observe its attention scores with all other grid points based their positional information. The results are shown in Figure The visualization clearly demonstrates the contribution of different positional information to the attention scores of different heads.
[A]
[b]
(c)
Figure 11: Visualization attention heatmaps of each head. (a)Attention map with reading order. (b) Attention map with coordinate (c) Attention map with coordinate y.
23


Reviews:
[{"review_id": "LAdouIp40B", "reviewer": "Reviewer_XTDc", "summary": "This paper introduces Group Position Embedding (GPE), a method to enhance layout comprehension in Large Language Models (LLMs) without requiring architectural changes or extra pre-training. By grouping attention heads with distinct positional embeddings, GPE effectively encodes layout information for document tasks. Tested on five standard benchmarks and the challenging BLADE benchmark, GPE shows significant improvement in document understanding over existing methods.", "questions": "Please refer to the weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) interesting task\n\n(2) new method\n\n(3) extensive experiments", "weaknesses": "(1) Missing important details for dataset construction: Any critiria to manual and filter challenging question-answer pairs for the data scenarios of Forms, Slides, and Websites? Why is Newspapers constructed with an initial manual selection whereas SynthDocs being synthetically generated? For SynthDocs whose answers are synthetically generated, any filtering measure to ensure the data quality? \n\n(2) Missing the intuitive motivation of the proposed method: Although this paper verified the effectiveness of the proposed method compared with other approaches on Layout-aware Position Embeddings through the empirical experiments, it lacks of more intuitive explanations on the advantages of the proposed method, which can provide more insights for the follow-up work.\n\n(3) Missing the clarification of the experiment results: According to  the experiment results in Table 1, vanilla Qwen2-7B  outperforms GPE-Qwen2-7B on VisualMRC, which needs more clarifications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eXZZIsns8d", "reviewer": "Reviewer_hPjd", "summary": "This paper introduces a new positional encoding method to tackle the limitations of current LLMs/MLLMs' lack of layout information for addressing layout-aware document understanding tasks. A dataset named BLADE is designed to mainly focus on measuring the performance on complex-layout aware questions. Various experiments with different model configurations are conducted.", "questions": "1. Motivation and Related Work: what is the limitation of current positional encoding methods adopted by pretrained document understanding frameworks and what are the limitations of them? \n2. Dataset: is that necessary to have a layout-aware dataset. Only focusing on layout-complex questions may ignoring the performance on other generative tasks. \n3. Model and Evaluation: is there any reason you chose those LLMs/MLLMs? It would be better to give more insight analysis for this part by giving more ablation and case studies. \n4. Is it possible to show some qualitative analysis of benchmark datasets like CORD, SROIE?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposed a new positional encoding to address the limitation of existing LLMs/MLLMs neglecting layout information. \n2. A benchmark is proposed specifically focusing on evaluating the performance of various LLMs/MLLMs on document understanding tasks with complex layouts.", "weaknesses": "1. Motivation: The limitations of current positional encoding methods adopted by other frameworks are not clearly defined. The research gaps are not clearly defined.\n2. The methods are described well with the technical workflow without specific reasons and experiments as to why the positional encoding is working. For example, there is no explanation or citation as to why different position information giving various heads are reasonable for the research aim. \n3. The datasets are not clearly described making understanding breakdown categories difficult. \n4. The proposed methods are only evaluated on LLM which is expected to see whether it's workable on MLLMs and pretrained document understanding models. \n5. More reading order setups should be tried like XY-cut. \n6. There are some typos and some possible errors in the paper, like Table 2 SCOIE performance. Some bolded digits in the performance tables are not the highest.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w8BM3QMrpb", "reviewer": "Reviewer_WcMm", "summary": "The paper makes two main contributions: (1) it uses group position embedding to ensure that different attention heads focus on different views of position, and (2) it introduces a new document AI dataset, BLADE, which highlights the model's ability to handle complex layout information. Experimental results demonstrate improved performance, and the ablation studies are comprehensive.\n\nUpdated on 23th Nov 2024: raise the score", "questions": "See weakness", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed group position embedding is novel.\n2. The proposed BLADE dataset is important to the field.\n3. The experiments are extensive, especially the comparison of different approaches to fuse layout information is insightful.", "weaknesses": "1. Although the experimental results in Tables 2 and 3 demonstrate the superiority of GPE, the results in Table 1 appear somewhat contradictory to those in Tables 2 and 3, requiring further explanation or experiment.\n2. The paper needs a section discussing why GPE is effective in guiding attention heads to focus on different positional views. Including a visualization of attention scores or similar analysis would provide valuable insights, as the improved performance alone is insufficient to fully explain the mechanism.\n\n3. The writing requires improvement, as several paragraphs are vague and difficult to understand. For example:\n(1) Lines 248-249: What is the value of lambda? Does GPE discretize all coordinates into integers, allowing us to obtain position embeddings for specific position?\n(2) Line 418: In the experiment regarding reading order, I cannot understand the difference between W/O and LOCAL. A concrete example would help illustrate the distinctions between each setting.\n(3) Line 473: It would be helpful to introduce the design of the group function where it first appears, in Section 3.2, using a formal mathematical definition. This would clarify the paragraph, as I found it confusing without prior context on the group function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rruQGK4b4l", "reviewer": "Reviewer_Ntfi", "summary": "This paper proposes a simple yet effective method, Group Position Embedding (GPE), for encoding spatial positional information in LLM-based document understanding tasks. This technique enables LLM models to comprehend document layouts without altering their architectures. Additionally, the authors introduce a new benchmark, BLADE, for evaluating complex document processing.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1.This paper introduces a simple layout embedding approach, termed Group Position Embedding (GPE), for enhancing LLM-based document understanding.\n2.This paper proposes a new benchmark, BLADE, designed for the evaluation of complex document evaluation. \n3.Extensive ablation experiments have been conducted to validate the effectiveness of GPE.", "weaknesses": "1. The rationale behind GPE is unclear. The introduction section fails to adequately explain the workings of the group position embedding, instead, it merely highlights the advantages without providing underlying reasons. It is also unclear what the principles and feasibility of the mapping function g_r() are, and the distinction between Gr(k) and Gr(i) on lines 196-197 is also not elucidated. Further explanation is needed on how the n-dimensional spatial position is mapped into different attention heads. The meaning of the hyperparameter scaling factor \\lambda and its impact on the approach should also be clarified.\n2. This paper is not the first to utilize head-specific layout position embeddings. Previously, LAGaBi [https://aclanthology.org/2023.findings-emnlp.521/] employed diverse Gaussian kernels to encode relative positional information for each attention head. \n3.The paper's writing quality is subpar. Inconsistent and non-standard academic expressions are used. For instance, in LLMs, l typically denotes the number of layers rather than dimensions, which should be represented by d or dim. The figures do not clearly demonstrate the mapping mechanism of group position embedding and convey some ambiguity. For instance, Figure 1 shows that only the key-state and value-state encode n-dimensional spatial position information, whereas the query-state receives only 1-dimensional position information, which is confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)