Beyond Layout Embedding: Layout Attention with Gaussian Biases for Structured Document Understanding
Authors: Xi Zhu
Xue Han
Shuyuan Peng
Shuo Lei
Chao Deng
Junlan Feng
Venue: EMNLP 2023 Findings (2023)
Keywords: Structured Document Understanding
Layout Attention
Spatial Relationships
Polar Coordinates
TL;DR: We propose a novel layout attention with Gaussian biases in transformer for structured document understanding, that modifies self-attention by the bias reflecting spatial relationships via Gaussian kernels.

Abstract: Effectively encoding layout information is a central problem in structured document understanding. Most existing methods rely heavily on millions of trainable parameters to learn the layout features of each word from Cartesian coordinates. However, two unresolved questions remain: (1) Is the Cartesian coordinate system the optimal choice for layout modeling? (2) Are massive learnable parameters truly necessary for layout representation? In this paper, we address these questions by proposing Layout Attention with Gaussian Biases (LAGaBi): Firstly, we find that polar coordinates provide a superior choice over Cartesian coordinates as they offer a measurement of both distance and angle between word pairs, capturing relative positions more effectively. Furthermore, by feeding the distances and angles into 2-D Gaussian kernels, we model intuitive inductive layout biases, i.e., the words closer within a document should receive more attention, which will act as the attention biases to revise the textual attention distribution. LAGaBi is model-agnostic and language-independent, which can be applied to a range of transformer-based models, such as the text pre-training models from the BERT series and the LayoutLM series that incorporate visual features. Experimental results on three widely used benchmarks demonstrate that, despite reducing the number of layout parameters from millions to 48, LAGaBi achieves competitive or even superior performance.

Beyond Layout Embedding: Layout Attention with Gaussian Biases for Structured Document Understanding
Xi Zhu, Xue Han, Shuyuan Peng, Shuo Lei, Chao Deng and Junlan Feng"
JIUTIAN Team, China Mobile Research Institute, Beijing, China
(zhugian, hanxueai haaggammhhhmeeeeeeepeeehahaaaan pwwgshuyuan, lei leishuo, dengchao, fenggjunlanj@echhnmmbblel......
Abstract
performance in several downstream tasks like form comprehension and receipt understanding.
Effectively encoding layout information is central problem in structured document under- standing. Most existing methods rely heavily millions of trainable parameters to learn the layout features of each word from Cartesian coordinates. However, two unresolved ques- tions remain: (1) Is the Cartesian coordinate system the optimal choice for layout model- ing? (2) Are massive learnable parameters truly necessary for layout representation? In this paper, we address these questions by propos- ing Layout Attention with Gaussian Biases (LAGaBi): Firstly, we find that polar coordi- nates provide a superior choice over Cartesian coordinates they offer measurement of both distance and angle between word pairs, captur- ing relative positions more effectively. Further- more, by feeding the distances and angles into 2-D Gaussian kernels, we model intuitive induc- tive layout biases, i.e., the words closer within document should receive more attention, which act as the attention biases to revise tex- tual attention distribution. LAGaBi is model- agnostic and language-inde indaeee-indeeeddent, which can applied to range transformer-based mod- els, such as the text pre-training models from the BERT series and the LayoutLM series that incorporate visual features. Experimental sults on three widely used benchmarks demon- strate that, despite reducing the number of lay- parameters from millions to 48, LAGaBi achieves competitive or even superior perfor- mance. Our code is available on GitHub'.
Unlike conventional et al.. 2019; Vaswani et al., 2017; Kenton and Toutanova, 2019), SDU goes beyond comprehend- ing serialized text and requires the ability to in- terpret documents with diverse layouts (Xu et al., 2020; Huang et al., 2022: Powalski et al., 2021; Li et al., 2021a; Wang et al., 2022a). Documents with varying layouts often contain text fields po- sitioned in different ways. To take advantage of existing pre-trained language models, early meth- ods (Xu ct al., 2020; Li et al., 202 la,e Appalaraju et al., 2021; Chi et al., 2020) propose to directly add 2-D position embedding to the word embedding for each word as input for Transformer. The po- sition embedding encode the 2-D absolute coordi- nates (Zo, yo, I1, 71) of each word in the document through multi position encoding layers, where (ar-- yo) represents the upper left point and (x1, 3/) rep- resents the lower right point of the bounding box for each word. Some rescarches (Powalski et al., 2021; Hong et al., 2022; Lee et al., 2022; Xu et al., 2021a; Huang et al., 2022) further proposed that ab- solute positions are inefficient for representing the spatial relationships between words. They employ relative positions between words to encode spatial relationships. For example, Hong al.(2022) map the relative positions into embeddings, which are then multiplied with the semantic embedding of the word to calculate inter-word layout scores. This score is incorporated into the self-attention layers to combine semantic and layout features.
understanding (Liu
1 Introduction
Structured document understanding (SDU) has gained significant research attention in the field of intelligent document processing (Park et al., 2019; Jaume et al., 2019; Han et al., 2023). focuses on extracting layout structures and contents from scanned or digital documents, leading to enhanced
Despite the significant progress made, we argue that current methods, whether based on absolute or relative positioning, heavily rely on large number of trainable parameters for position embeddings from Cartesian coordinates. often comprising mil- lions of parameters. This raises two unexplored questions: (1) the Cartesian coordinate system the optimal choice for layout modeling? (2) Are
Juunlan Feng is corresponding author.

---

massive learnable layout parameters trily neces- sary? Concerning the former, various coordinate systems exist, including Cartesian, polar, and spher- ical coordinates, yet previous research has solely focused on Cartesian coordinates. Regarding the latter, it is intuitively expected that words closer within document should receive more attention. However, this simple inductive bias may not be effectively learned solely through gradient updates.
et al., 2020), empowering them effectively pro- cess structured documents and achieve significant performance gains of up to 27.01 points. Addi- tionally, LAGaBi can be incorporated into complex SDU models that leverage visual features, such as LayouILM (Xu et al., 2020), LayoutLMv2 (Xu et al.. 2021a), and LayoutLMv3 (Huang et al., 2022), leading further performance enhance- ments and establishing new state-of-the-art results.
In this paper, present unified investigation of the two aforementioned problems. Regarding the choice of coordinate systems, we find that polar coordinates offer a more efficient representation than Cartesian coordinates for expressing relative positions. By computing the differences in distance and angle between two words in polar space, po- lar coordinates outperform their Cartesian counter- parts by providing extra angle information. For lay- out learning, discover that layout modeling can be achieved by a specific distribution: words closer in space receive higher layout scores, eliminating the need for extra position embeddings. Combining these two choices, we propose LAGaBi (Layout Attention with Gaussian Biases). LAGaBi formu- lates pairwise spatial relationships between tokens using the distance and angle in the polar coordinate system. Moreover, the distance and angle are fed into 2-D Gaussian distribution output layout score. We choose the Gaussian distribution be- cause it guarantees that the layout score decreases when either the distance or angle variables increase, making it the most commonly used distribution for this purpose. The layout score is then incorporated into the original self eeff-attention as attention bias, re- sulting in a revised distribution that considers both text and layout features. We introduce trainable Gaussian kernels better align the semantic and layout scores different scales, adding just 4x at- tention heads extra parameters. For instance, based on RoBERTa (Liu et al., 2019) with 12 attention heads, there are only 48 additional parameters that need to be learned for encoding layout features.
Related Works
Significant progress has recently been made by using the Transformer- based pre-trained model (PTM) to learn the cross-modality interaction be- tween textual and layout information, which has been demonstrated be critical for structured doc- ument understanding (Xu et al., 2020; Huang al., 2022; Powalski et al., 2021; Li et al., 2021a; Wang et al., 2022a). LayoutLM (Xu et al., 2020) modified the input of BERT (Kenton and Toutanova, 2019) by adding position embedding layers to encode word-level 2-D coordinates, while Structual (Li et al., 2021a) proposed to encode segment-level positions. LiLT (Wang et al., 2022a) encoded text and layout using two different transformer layers separately and adopted bi-directional attention fuse them. Besides, there are a series of works that use multi-modal transformers to model text, lay- out, and image simultaneously. such as SelfDoc (Li et al., 2021b), DocFormer (Appalaraju et al., 2021), StrucTexxT (Li et al., 2021c), ERNIE-Layou (Peng et al., 2022), mmLayout (Wang et al., 2022b).
Most of the above approaches encode the 2-D absolute positions, ignoring the critical relative spatial relationships between words that are es- sential to textual semantic understanding. Hong et a2.22) proposed to encode the spatial relation- ships as relative position embeddings, which are then multiplied with the semantic embedding of the token calculate inter-word layout scores. This score incorporated into the self-attention to combine semantic and layout features. GeoLay- outLM (Luo et al., 2023) introduces geometric re- lations and brand-new geometric pre- pre-training tasks in different levels for learning the geometric lay- out representation, whose geometrie relations are largely dependent on some pre-defined manual rules. TITL (Powalski et al., 2021) encodes the relative positions between words in simpler man- ner, and it adopts linear layers to convert the 2-D discrete distance (implemented through bucketing)
Extensive experiments demonstrate that LAGaBi achieves remarkable performance on diverse SDU benchmarks, including FUNSD (Jaume et al., 2019), CORD (Park et al., 2019), and XFUND (Xu al., 2021b), across both monolingual and multi- lingual scenarios. LAGaBi emerges module that seamlessly integrates with transformer- based language models, such as BERT (Kenton and Toutanova, 2019), RoBERTa, and InfoXLM (Chi
versatile

---

Layout Attention with Gausslan Blases
COARRT TTE PEPORT Foeu
YBAR
CIOSEETTE TI RESPORT FOHM
COARSTTE
REPORT
Fonm
YEAR
CIGARETTE
OIBARETTE
8.00
REPORT
5:00
0.29
REPORT
6:22
0.50
0.21
0.01
0.00
FORM
FORM
FORM
YEAR
02
0.81
8.01
0.22
0.08
6.6
0.04
602
8.54
0.00
YEAR
2.00
0.00
3.00
0.30
0.0
MONO
MONO
MONO
Original Distribution
Gausslan Blases
Revised Attention
(query-key attention scores based textual representation)
kerrels
CIGARETTE
REPORT
CIGARETTE
REPORT
FORM
FSC
$640 MAL
FILTER LEUTT)
FILTER TO1
PLAAO DNV
1921 11:1
ls 297
LeT MUEEIC. ENTE.
YEAR
YEAR
Structured Documents
① Polar Coorcinates Centered at REPORT
Polor Coordinates Centered YEAR
Spatial Relationshipe with Polar Coordinates
Figure I: An illustration of our Layout Attention with Gaussian Biases (LAGaBi). For each query word, we first compute the spatial relationships of each keyword in polar system centered it. Then Gaussian kernels are employed to transform these spatial relationships into attention biases, which will revise the original query-key attention distribution. Such biases are shared across different Transformer layers, When using LAGaBi, we can encode the document layout structure without adding position embedding layers at the bottom of the network.
between words into attention biases motivated by T5 (Raffel et al., 2020). Such relative attention bi-
W [wi] N and their associated layout positions {a]h-- where is the number of words while C. - [a]] vl. xx! v! represents the left-top and right-bottom coordinates of the bounding box that contains the in word. An ideal model for doc- ument understanding should take both text words and their layout positions into consideration.
ases have also been employed by LayoutLMv2 (Xu et al., 2021a) and LayoutLMv3 (Huang et al.. 2022), yielding notable results. Different from them, we model the relative positions from new perspective, i.e. introducing polar coordinates with distance and angle that more efficient in repre- senting relative spatial relationships. Furthermore, we employ the extremely lightweight Gaussian ker- nels to encode polar coordinates into attention bi- ases, offering a more streamlined approach with fewer trainable parameters and aligning better with human intuition compared to linear layers.
The key structured document comprehension, building upon the achievements of text understand- ing techniques such as Transformer ( (Vaswani et al., 2017), lies in effectively representing document layouts while remaining compatible with language representation. Therefore, there are two crucial problems that need to be solved: 1. How model document layouts efficiently? 2. How to integrate the layouts into Transformer to guide textual se- mantic understanding the document content?
3 Methodology
Structured documents typically contain both tex- tual words and layouts, where textual words de- note the main content of the document, while lay- outs record the organizational form of text. Op- tical character recognition (OCR) as a classical technique for image document parsing can recog- nize the text as well as its locations. Formally, given a document D, OCR identifies the text words
In this paper, we introduce novel layout mod- eling method for structured document understand- ing, namely Layout Attention with Gaussian Biases (LAGaBi). As shown in Figure 1, it mainly con- sists of two parts: modeling relative positions with polar coordinates and layout attention with Gaus- sian biases. The former is responsible for modeling the 2-D relative positions between words by po-

---

lar coordinates which are capable of representing inter-word spatial relationships; The latter focuses on transforming polar coordinates into attention biases, which will modify the original semantic query-key attentions between words into a more suitable distribution that accurately captures the underlying layout structure of documents.
spatial relationship between the words "FORM" and "REPORT" can be represented as polar co- ordinate (0.064, 0), indicating a distance of 0.064 and an angle of 0 degrees, while that between the words "YEAR" and "REPORT" is (0.297, 1 1.432).
3.2 Layout Attention with Gaussian Biases
How to use the essential relative spatial relation- ships to guide the model to perceive layout infor- mation is problem worth exploring. Inspired by ALiBi (Press et al., 2021) and T5 bias (Raffel et al., 2020) that encode the 1-D relative position information as attention biases upon the query-key scores instead of positional embedding, we propose to revise attention scores/distribution with 2-D at- tention biases that integrate spatial relationships. Specifically, the attention score in single-head self-attention can be modified as:
3.1 Spatial Relationships with Polar Coordinates
In a structured document, the relative spatial re- lationships between words are shown to be more important than their absolute coordinates, which can assist humans in better understanding pair-wise semantic dependencies. For example, tokens in the same line generally have stronger semantic associ- ations with each other, while tokens that are farther away and on different lines are more difficult to form strong associations. Although some previous works (Powalski et al., 2021; Xu et al., 2021a; a; Li et al., 2021a) have proposed to model the relative distances, which acted as learnable attention biases. they still rely on learnable positional embeddings and only utilize the relative horizontal and vertical distances in Cartesian coordinate system.
exp(qkk] / Vdd a (g(u.)) - 1))
aij Σド= explqxkk /vd. + (g(u:)) - 1))
(3)
where qi is the ieh query vetor, qi is the jen key vetor, and dę is the dimension of the attention head. g(ug)) denotes the attention biases, which is de- rived from the 2-D Gaussian kernel with learnable parameters based on polar coordinates u indicating spatial relationships. The Gaussian kernel ensures that words farther within the document are assigned smaller layout score. Therefore, by incorporating reversed term (g(u.)) - 1), we can significantly penalize the attention scores of query-key pairs that are farther, while making only slight revisions the scores of closer pairs. or is a hyper-parane parameerr that makes a trade-off between semantic associa- tion and spatial dependency. It denotes how much the spatial relationship between the key and the query contributes to their semantic association. In particular, convenient formula of g(u) is:
Different from previous works, we propose to capture inter-word spatial relationships through polar coordinates, in which both orientations and distances be preserved. For each query token, we can build polar coordinate system centered at its position, and calculate the polar coordinates (spatial relationships) of its keys. More concretely, the position of the query token is regarded as the reference point (pole), and the horizontal direction in the current Cartesian coordinate system is set the reference direction (polar axis) following the most common reading habit, i.e., left-to-right and top-to-down. Formally, given query token with its 2-D coordinates e; as the pole, the polar coordinates Usj = (Pij. (1y) of the juh key in this document page can be calculated as below:
--- 1 -u- - (u 4)) (4) g(u) = exp(-=
Pig = (x, x;)²+ + (w) - 41
(1) where Σ and are learnable 2 x 2 and 2 x co- variance matrix and mean vector of a Gaussian kemne, respectively. We further restrict the covari- ances to have diagonal form, resulting in 2 x 2 parameters per kernel for each attention head. Note that the Gaussian kernels are different across differ- ent attention heads, but are shared across different self-attention layers. Thus, there is a total of x x Nheads learnall parameters for our attention biases, where Nheads denotes the attention head
0g = lan '(11 - 11)((1; ai:))
(2)
where Piy [0,1] and 6, 1- m denote the distance and angle (orientation) from the ies token the juh token respectively, and (I,,) w:) are the normalized coordinates of the top-left point of the inh bounding box. For instance, in Figure 1, when taking "REPORT" as the reference point, the

---

number in each self-attention layer. For example. taking the RoBERTa base as the backbone, there are 48 parameters that need to be learned. Notably, we only include layout information in the keys and queries but not in the values, ensuring that the text semantics are not corrupted.
of baselines: 1) monolingual models (BERT (Ken- ton and Toutanova, 2019) and RoBERTa (Liu al., 2019)). 2) multilingual model (InfoXLM (Chi al., 2020)), and 3) document understanding models (LayoutLM (Xu et al., 2020), Layoutl.Mv2 (Xu et al., 2021a), and La (Huang et al., 2022)). Gaussian kernels will be included in the self-attention blocks of each model, which describe the relative spatial relationship between tokens. Pre-training. We only conduct pre- pre-training tasks for the two monolingual models: BERT+LAGaBi and RoBERT&+LAGaBi. We initialize the weight of them with the corresponding baselines, except the Gaussian kernels. The parameters of the Gaus- sian kernels, namely covariance matrixes and mean vectors, are randomly initialized. Both models are simply supervised by masked language mod- eling (MLM) loss during pre-training. Adam op- timizer (Kingma and Ba, 2014) is adopted with a learning rate of 5e 5, weight decay of le - 2 and (BL, [22) = (0.9, 0.999). The batch size is set to 128 and all the two models are trained for 200,000 steps on 8 NVIDIA v100 32GB GPUs.
Properties of our LAGaBi: (1) It is easy to imple- ment and can be adapted to any transformer-basee model without changing its structure. (2) The po- sition embedding layer is discarded, and there are very few parameters to learn, which can be done during the fine-tuning stage, so it is quite efficient. (3) It decouples layout and text understanding, al- lowing the potential of language models to be fully exploited in structured document understanding.
4 Experiments
4.1 Datasets
Pre-training Dataset. Following LayoutL. (Xu et al., 2020), we also pre-train model using the IIT-CDIP Test Collection 1.0 (Lewis et al., 2006), which is large-scale dataset with over 11 million scanned document images. Only million of them are used for fast pre-training. We pre-process each document page using Tesseract?, an open-source OCR engine, to retrieve the textual contents as well as their layouts. We normalize the coordinates of each token to integers in the range of 0 to 1000 and add empty bounding box [0, 0, 0, 0] to the special tokens [CLS], [SEP], and [PAD]. Fine-tuning Datasets. We evaluate our method on both monolingual (English) and multilingual docu- ment information extraction datasets listed below. FUNSD (Jaume et al., 2019) is a form dataset that uses forms to extract and organize textual infor- mation. It contains 199 documents, 149 of which are for training and 50 of which are for testing. CORD (Park al., 2019) is dataset for receipt key information extraction that includes 800, 100, and 100 receipts for training, validating, and test- ing, respectively. XFUND (Xu et al., 2021b) is a multilingual version of FUNSD with 8 languages, each language containing 199 instances (149 for training and 50 for testing) as FUNSD.
Fine-tuning. In this paper, we mainly focus on the document understanding task of semantic entity labeling, which aims assigning each semantic en- tity BIO label. We add token-level classification layer upon the base models (including monolingual, multilingual, and document understanding models) to predict the BIO labels for this task. Word-level Fl score is adopted as the evaluation metric. The fine-tuning process takes 2000 steps using a batch size of 16 and the Adam optimizer with learning rate of 5e-5 for FUNSD and 7 for CORD and XFUND. Fine-tuning configurations for document understanding models follow their official releases. Hlyer--paaaeeee
set to for experiments. which has been tuned on the CORD's val set.
4.3 Experimental Results
4.3.1 Performance on monolingual datasets
We first evaluate our method on the monolingual form and receipt understanding datasets. From the results shown in Table 1, we can observe that:
(1) The lightweight LAGaBi enables simple in- tegration with language models, allowing them to effectively process structured documents. For example, without any pre-training on document data, LoBERTa+LAGaBi has achieved 84.84% and 95.97% Fl scores on FUNSD and CORD, sur- passing baseline model RoBERTa by 18.38% and
4.2 Implemention Details
Our approach is model-agnostic and language- independent, which be applied to range of transformer-basee models. In this paper, we have evaluated our method based on three kinds

---

Model
#arraeetess
Pre-training
Modality
FUNSD
CORD
BERT (Kenton and Toutanova, 2019)
110M
T
60.26
89.68
RoBERTa (Liu et al., 2019)
125M
66.48
93.54
BROS (Hong al., 2022)
110M
11M
T+L
81.21
FormNet (Lee et al., 2022)
217M
11M
T+L
84,69
LiLT (Wang et al., 2022a)
131M
11M
T+L.
88.41
96.07
BERT+LAGaBi w/o pre-train)
110M+48
T+L
74.14 (+13.88)
93.44 (+3.76)
BERT+LAGaBii(w/ pre-train)
110M+48
IM
T+L
87.27 (+27.01)
95.82 (+6.14)
RoBERTa+LLAGaBive pre-train)
125M+48
T+L
84.84 (+18.36)
95.97 (+2.43)
RoBERTa+LAGaBi(w pre-ttrr pre-train)
125M+48
IM
T+L
89.15 (422.67)
96.56 (+3.02)
La (Xu al., 2020)
160M
11M
T+L+1
79.27
94.31"
Layoutl Mv2 (Xu et al., 20
200M
11M
T+1++1
82.70
94.95
StrucTexT (Li et al., 2021c)
107M
11M
T+L+I
83.09
DocFormer (Appalaruju et al., 2021)
183M
11M
T+L+I
83.34
96.33
LvyoudLMv3 (Huang et al., 2022)
133M
11M
T+L+I
90.29
96.56
pre-train)
160M+48
T4LAI
87.77 (48.10)
94.93(+0.62)
LayoutLMv2+LAGaaiiv pre-train)
200M+48
T+L+I
88.16 (45.49)
97.05(42.10)
Layoutl Mr3+ pre-train)
133M+48
T+L+1
91.00 (+0.71)
97.05 (+0.49)
Table Performance on FUNSD and CORD for monolingual structured document understanding. "T/L/T" denotes the ""ext/layout/mmaee" modality. (+x) denotes the gain F1 score compared to base model, while show the result from our re-implementation. All the Fl scores in percentage (%) are reported.
2.43% respectively. RoBERTa+LAGaB also out- performs several representative document under- standing models such as LayoutLM (Xu et al., 2020) and LayoutLMv2 (Xu et al., 2021a). The results show that LAGaBi is a powerful method for capturing essential layout features, allowing language models to be easily extended to adapt to structured document understanding tasks.
is particularly efficient since it only requires fine- tuning based on the published pre-trained weights rather than pre-training from scratch. Furthermore, by combining LAGaBi with the top-performing LayoutLMy3 model, we achieve new state-of-the art results on both FUNSD and CORD datasets.
4.3.2 Performance on multilingual dataset
(2) Pre-training brings profits. After pre-training 1 million unlabeled document data, our method exhibits extra improvements. RoBERTa+LAGab) with pre-train surpasses other approaches ex- cept LayoutLMv3 (Huang et al., 2022). While there is still a minor difference on FUNSD be- tween our model and LayouLLMv3, our method is significantly easier to implement, introducing only 48 extra learnable parameters to the vanilla Transformers structure, making more computa- tionally efficient and flexible.
Following the multi-lingual LayoutXLM (Xu al., 2021b) and LiLT (Wang et al., 2022a), we also evaluate our method based on InfoXLM (Chi et al., 2020) on three sub-tasks: language-specifie fine- tuning, multi-task fine-tuning, and zero-shot trans- fer learning. We first perform fine-tuning based on the Gaussian kernels with random initialization. For fair comparison, following LiLT, we also adopt the pre-trained Gaussian kernels for further fine-tuning, and we employ the Gaussian kernels RoBERRTa+LAGaBi which have been pre-trained on IM monolingual document data in Sec 4.3.1. Results on XFUND are shown in Table 2.
(3) The LAGaBi could also be seamlessly cou- pled with other layout embedding and multi- modal-based document understanding models, improving their performance even further. Per- formance improvements on LayouLLM (Xu et al., 2020), LayoutLMv2 (Xu et al.. 2021a) and Lay- outLmv3 (Huang al., 2022)) are obvious, with Fl score gains of 8. 100% 5.49%, 0.71% FUNSD and 0.62%, 2.10%. 0.49% on CORD. The process
LAGaBi is also valid in multilingual scenarios, allowing multilingual language models to un- derstand structured documents and outperform existing west-performing methods. LAGaBi largely increase the performance of the multilin- gual language model InfoXLM on three tasks, regardless of whether the Gaussian kemelss are

---

Pre-train Data
FUNSD
XFUND
Task
Model
Avg.
size & language
EN
ZH
JA
ES
FR
IT
DE
PT
Language specific
XLM-RoBERTa (2020)
66.70
87.74
77.61
61.05
67.43
66.87
68.14
68.18
70.47
InfoXLM (2020)
68.52
88.68
78.65
62.30
70.15
67.51
70.63
70.08
72.07
L.ayoutXIMM (2021b)
30M-Mutli
79.40
89.24
79.21
75.50
79.02
80.82
82.22
79.03
80.56
LiLT (2022a)
11M LoMMoono
84.15
89.38
79.64
79.11
79.53
83.76
82.31
82.20
82.51
InfoXLM+LAGaBI
83.35
89.38
84.01
78.06
82.80
84.40
84,95
83.25
83.78
InfoXLM+LAGaBi
IM-Mono
84.17
89.65
84.73
77.51
83.86
85,19
83.89
83.47
84.06
XLM-RoBERTa (2020)
66.70
41.44
30.23
30.55
37.10
27.67
32.86
39.36
38.24
Zero-shot
InfoXLM (2020)
68.52
44.08
36.03
31.02
40.21
28.80
35.87
45.02
41.19
LayoutXLM (2021b)
30M-Muti
79.40
60.19
47.15
45.65
57.57
48,46
52.52
53.90
55.61
LiLT (2022a)
11M- Mono
84.15
61.52
51.84
51.01
59.23
53.71
60.13
63.25
60.61
InfoXLM+LAGaBi
83.35
47.91
46.13
48.98
54.84
47,40
53.23
58.82
55.08
InfoXI.MLAAGaRi
IM-Mon
84.17
39.88
35.77
44.82
54.45
46.72
51.64
56.37
51.73
XLM-RoBERTa (2020)
66.33
88.30
77.86
62.23
70.35
68.14
71.46
67.26
71.49
Multi-task
InfoXLM (2020)
65.38
87.41
78.55
59.79
70.57
68.26
70.55
67.96
71.06
LayoutXLM (2021b)
30M-Muti
79.24
89.73
79.64
77.98
81.73
82.10
83.22
82.41
82.01
LiLT (2022a)
IM-Mono
85.74
90.47
80.88
83.40
85.77
87,92
87.69
84.93
85.85
InFoXLM+LAGaBI
86.67
90,90
86.49
81.74
86.44
87.74
88.18
86.72
86.89
InfoXLM-LAGaBi
IM-Mono
86.35
92.00
86.86
81.50
87.00
87.96
87.58
87.24
87.22
Table The performance on FUNSD and XFUND with different settings, including language-specifie fine-tuning (fine-tuning on X. testing on X), zero-shot transfer (fine-tuning on FUNSD, testing X), and multi-task fine-tuning (fine-tuming on all 8 languages, testing on X). "IM-Mono" denotes million monolingual (English) documents used for pre-training while "30M-Multi" the multilingual version. All the F2 scores in percentage (%) are reported.
pre-trained or not. InfoXLM+LAGaBi using the pre-trained Gaussian kernels outperforms the top- performing method LiLT on both language-speccfic and multi-task fine-tuning tasks, with average Fl scores of 84.06% and 87.2 87.22%, demonstrating the efficacy of LAGaBi multilingual scenarios. On the zero-shot transfer learning task, LAGaBi fell slightly behind its counterparts. This may be due to the inherent gaps between different languages, such as differences in reading order and semantic den- sity. For example, English usually uses spaces to separate words and has uneven word lengths, while Chinese appears tighter sequence with smaller semantic units (i.e., characters). Such layout knowl- edge learned from a specific language shows lim- ited contributions to other languages. This phe- nomenon also proves the effectiveness of LAGaBi in modeling layouts, i.e. it actually has acquired the layout knowledge for a specific language after fine-tuning on the corresponding data.
ducted extensive ablation experiments based on sev- eral RoBERTa variants equipped with different lay- out encoding mechanisms (e.g. layout embedding layers, linear bias layers, and fixed/lcarnable Gaus- sian Kernels), and spatial relationships (e.g. dis- tance, angle, and 2D-xy distance). All the variants are evaluated without further pre-training, and only fine-tuned for 2000 steps on FUNSD and CORD. Results on FUNSD's ' test set and CORD's valida- tion set are shown in Table 3. From the result, we can observe that LAGaBi with learnable Gaussian kernels and polar coordinates (#8) can significantly outperform the baseline (#1) and the model with linear layout embedding layers (#2), indicating that models encoding layouts as attention biases are superior to layout embedding- methods.
Impact of the Gaussian kernels. To study the effects of various methods for converting polar co- ordinates to attention biases, we compared linear layers (#3), fixed Gaussian kernels (#4), and learn- able Gaussian kernels (#8). The results demon- strate that linear layers are far less effective than Gaussian kernels, which is likely due to the fact that the Gaussian kernels are more in line with human
4.4 Ablation Studies
To investigate the impacts of our learnable Gaus- sian kernels and polar coordinates, we have con-

---

Map RoB
Nop ReBERTS LACaBI
STRUCTURE URE
232
E.0
D10
6.
STRUCTURE
11.34
28
30
82
08
CONVENTS
66
212
D0
6.
314
50
COMMENTS
0.5
(1.22
20
10
42
90
04
Ergs
110
100
100
121
Engl
2.1
20
010
0.30
11.05
193
6:24
00
60
500
003
0.11
0.11
0.30
DATE
STA CTT
Drgg
Figure 2: Visualization of the attention maps. The word-word attention level attention in the last layer of transformers. We also annotate the position and order of each word in the original document page, and the attention scores are rounded to two decimal places for better visualization.
obtained by aggregating token-
Task
Ablation Strategy
FUNSD
CORD
4.5 Analysis
RoBFRTa(bascline)
66.48
92.29
Impact a. Hyper-parameter a makes a trade- off between semantic and layout contributions when computing pair-wise attention scores in our LABaBi, which is important. We conduct several experiments with different a settings to study the impact of based on RoBERTa without any further pre-training. F1 scores CORD's validation set are listed in Table 4, showing the model achieves its best performance when 4.
+ embedding layers
69.59
92.67
Impact of 3 + linear bias layers
76.32
93.00
Gaussian
+ fixed kernels
83.81
93.00
+ Euclidean distance
73.05
93.72
Impact of Polar-Coor.
+ Angle
84.48
94.70
7 + 2D-xy distance
79.85
94.21
+ LAGaBi
84.84
94.77
Table 3: Ablation studies on the effectiveness of the learnable Gaussian kernels and the polar coordinates.
1
2
3
4
5
CORD 92.29
92.82
93.08
93.46
94.77
94.39
intuition than linear layers. Learnable Gaussian kernels (#8) also achieve better performance than fixed Gaussian kernels whose mean is 0 and vari- ance is (#4), since the learnable Gaussian kernels enjoy better flexibility to adapt to different formats. Impact of the polar coordinates. Polar coordi- nates, which consist of two elements: distance and angle, is typical technique for describing spatial relationships. We analyzed the effects of distance (#5) and angle (#6), as well as the classical 2-D relative horizontal and vertical distances proposed by TITL (Powalski et al., 2021) (#7). The results suggest that the model with angle information (#6) is more effective than the model with distance in- formation (#5 and #7). We hypothesize that this is because angles are less impacted by size scal- ing than distances, but they are more sensitive to location changes. Furthermore, LAGaBi achieves much better performance than the model that em- ploys horizontal and vertical distances, which fur- ther reveals the superiority of polar coordinates.
Table 4: Impact of different a settings. All the scores are from CORD's validation set.
Visualization analysis. We visualize the word- level attention maps of the baseline RoBERTa and our RoBERTa+LAGaBBi Due to space limitations, in this paper, we only show the attention maps of the first 8 words in the input sequence. The case in Figure 2 is from the test set of FUNSD. As shown in Figure 2, the RoBERTa incorrectly associates the character "R." with all the words, while most other words are treated as unrelated. According to the attention map in the RoBERTa+LAGaBi greater attention scores arise between "R." and "F.", "J." and "D.", ", all of which are placed in the signature area, whereas attention scores between remote irrel- evant words such "DATE" and "STRU" are zeros. This demonstrates that LAGaBi indeed learns more accurate semantic associations by incorporating layout information. More examples and detailed analysis can be seen Appendix.A.

---

5 Conclusion
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary. Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451.
In this paper, we propose a model-agnostic and language-independdnt method that leverages Lay- out Attention with Gaussian Biases to encode the relative spatial positions for structured document understanding (SDU). Specifically,, we first model the inter-word spatial relationships using polar co- ordinates. Then the query-key attention scores are revised by the Gaussian biases that are related to their spatial relationships. Our method can be ap- plied to a series of Transformer-based models with extremely few parameters, improving their perfor- mance for SDU tasks. Experiments based on six transformer-based SDU models and three mono- jingual/multilingual benchmarks fully demonstrate the effectiveness of our proposal. This research provides new ideas for structured document under- standing tasks, which are expected to promote the efficient development of document intelligence.
Xue Han, Yi-Tong Wang, Jun-Lan Feng, Chao Deng, Zhan- Chen, Yu-An Huang. Hui Su, Lun Hu, and Peng- Wei Hu. 2023. survey of transformer- based multimodal pre-trained modals. Neurocomput ing, 515:89- 106.
Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Dachyun Nam, and Sungrac Park. 2022. Bros: A pre-trained language model focusing on text and layout for better key information extraction from documents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10767- 10775.
Yupan Huang, Tengehao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutlmv3: Pre-training for doe- ument with unified text and image masking. Proceedings of the 30th ACM International Confer- ence Multimedia, pages 4083-4091.
Limitations
Despite the superior performance exhibited by LAGaBi, it does have some limitations. Firstly, in our experiments with the LayoutLM series that integrate multi-modal features, LAGaBi was only fine-tuned for validation without pre- pre-training. We believe that leveraging multi-modal pre-training could further improve LAGaBi's performance based on LayoutL., and this will be explored in future investigations. Secondly, although have empirically demonstrated the effectiveness of po- lar coordinates and Gaussian distribution in layout learning, our motivation is driven by a simple intu- ition rather than rigorous mathematical proof.
Guillaume Jaume, Hazim Kemal Ekenel, and Jean- Philippe Thiran. 2019. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pages 1-6. IEEE
Jacob Devlin Ming-Wer Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of NAACL-HILT, pages 4171 1166.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6900 1412.6980
Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vin- cent Perot, Guolong Su, Nan Hua, Joshua Ainslie. Renshen Wang, Yasuhisa Fujii, and Tomas Pfister. 2022. Formnet: Structural encoding beyond sequen- tial modeling in form document information extrac- tion. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 3735-3754.
Acknowledgements
We sincerely thank all the anonymous reviewers for their valuable comments and suggestions.
References
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. 2021. Docformer: End-to-end transformer for document understanding. In Proceedings of the IEEE/CVF international con- ference computer vision, pages 993-1003.
David Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, David Grossman, and Jefferson Heard. 2006. Building a test collection for complex document in- formation processing. In Proceedings of the 29th annual international ACM SIGIR conference on Re- search and development in information retrieval, pages 665-666.
Zewen Chi. Li Dong, Furu Wei, Nan Yang, Sak- sham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2020. In- foxlm: An Ekormation-theoree cross-lingual language model pre-training. arXiy preprint arXivv2007.0755 078344.
Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 202 Structurallm: Structural pre-training for form understanding. arXiv preprint 1r05v:2105.11221
framework for

---

Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad Morariu. Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. 202 Selfdoc: Self-supervised doe- ument representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5652-5660.
Jiapeng Wang, Lianwen Jin, and Kai Ding. 2022a. Lilt: A simple yet effective language- Iaggaag-inndeeedeen layout transformer for structured document understanding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7747-7757.
Yulin Li, Yuxi Qian, Yuechen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, and Errui Ding. 2021c. Structext: Struc- tured text understanding with multi-modal transform- In Proceedings of the 29th ACM International Conference on Multimedia, pages 1912-1920.
Wenjin Wang, Zhengjie Huang. Bin Luo, Qianglong Chen, Qiming Peng, Yinxu Pan, Weichong Yin, Shikun Feng. Yu Sun, Dianhai Yu, et al. 20226. mm- layout: Multi-grained multimodal transformer for document understanding. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2577-4886.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint 190.:1997.11690
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang. Yijuan Lu, Dinei Florencio. Cha Zhang, Wanxiang Che, et al. 2021a. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 2579-2599
Chuwei Luo, Changxu Cheng, Qi Zheng, and Cong Yao. 2023. Geolayoutlm: Geometric pre-training for visual information extraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7092-7101.
Yiheng Xu, Minghao Li, Lei Cui. Shaohan Huang, Furu Wei, and Ming Zhou. 2020. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD Interna- tional Conference on Knowledge Discovery & Data Mining, pages 1192-1200.
Seunghyun Park, Seung Shin, Bado Lec, Junyeop Lec, Jacheung Surh, Minjoon Seo, and Hwalsuk Lee. 2019. Cord: a consolidated receipt dataset for post-ocr parsing. In Workshop on Document Intelligence at NeurlPS 2019.
Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang, Yuhui Cao, We- ichong Yin. Yongfeng Chen, Yin Zhang, et al. 2022. Frmie-layout: Layout knowledge enhanced pre- training for visually-rich document understanding. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 3744-3755
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang. Yi- juan Lu, Dinei Florencio, Cha Zhang, and Furu Wei. 2021b. Layoutxlm: Multimodal pre-training for multilingual visually-rich document understanding. arXiv preprint 2rXiv:2104.08833
A Appendix
Figure in this section shows the attention dis- tribution of four additional examples on FUNSD. From all the examples, can be observed that the RoBERTa model typically treats each word in isolation, while RoBERTa+LAGaBi can learn the more accurate correlations between different words based on their layout relationships.
Rafat Powalski, Lukasz Borchmann, Dawid Jurkiewiez, Tomasz Dwojak, Michal Pietruszka, and Gabriela Palka. 2021. Going full-tilt boogie on document un- derstanding with text-image-layee -lyout transformer. In Document Analysis and Recognition-ICUL -CDAR 2021: 16th International Conference, Lausanne, Switzer- land, September 5-10, 2021, Proceedings, Part II 16, pages 732-747. Springer.
We also visualize the word-level attention maps of LayoutLMy3 and LvyouL.MV++LAGaBi Lay- outLMv3 is currently the top- perfforffnin method for structured document understanding, which uti- lizes the 2-D relative positions through linear atten- tion biases. From the attention maps shown in Fig- ure 4, we can observe that LayoutLM33+LAGaa refers more to layout information when modeling the iote semantic correlations, while Lay- LutLMv3 is relatively independent. For example, in the third sample of Figure LAGaBi learns more dense associations among "Tiers", "II.", and " "&= in the neighborhood than LayoutLMv3, which is in line with human intuition.
Ofir Press, Noah A Smith, and Mike Lewis. 2021. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2 12409.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text text trans- former. The Journal of Machine Learning Research, 21(1):5485- 5551.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

---

Attantion
LAG
FROM:
FROM:
SUSRCT
SUNJECT
8:02
X-
601
0.22
601
041
X-
0.00
0.00
8.8
0.0
0.0
0.
01
00
00
PRE-- 0.
0.
0.00
0.
0.01
0.04
0.0
0.4
0.
0.0
00
001
SELL 0.6 00
0.00
00
0.00
00
DISTRIBUTION
6.0
DISTRIDTIONN
00
0/0
FREM!
MIL
SAREET
Atterion Map RoBlER'S
Aftention Hap RoSERB-LLGGN
2:
68
2.19
AU7
63
TD: FU31
CC
88
34C
0.00
8.00
30
00
02
CCL
0:30
022
0.55
033
39
00
Crodaststel
68
60
2.
64
Crodetion
20
0.52
2.0
90075099
2.0
00
30
Yark
Aftentia Map of FORERRS
Attectio - Mep FoREFITH DEARI
T02
70:
3.62
0103
6.01
6.00
0.01
PROM
6.4
00L
LA0
2.
00
8.44
096
3.62
069
8.44
6.00
3.06
6.46
DATE:
5.11
0.2
DKTE
090
3.62
060
081
6.00
0.11
001
0.L
0.05
8.11
6.01
3.1
10
DNTE
DITE:
Attention FOBERTS
33
3.12
0.02
E.0L
3.1
DOL
0.
DRFECTIVE
33
ORJECTIVE
8.00
TUICET
22
MALL
TRRCET
66
6.M4
COMMRINCTION
80
33
33
800
30
0.00
CO
DOPEN MICATOM
64
6.3
302
0.0
005
00
PLATTORM
33
PLATOO MM
0J
64
0.3
SHOWTCS
107777
Horte:
Puttrore
Figure 3: Visualization more examples based RoBERTa.

---

Altentian Map of LoyoutL9ec
Attention Map LeyousENN LACAHB
DAPE
80
38
8:11
2.2
CO
DATE
62
10
DO
STRUCTURE
9.0
040
6.2
50
03
60
00
STRUCTUPE
000
0.31
3.5
000
00
0.9
COMMENTS-
3.0
08
C22
60
02
COMMENTTS
8222
0.82
3.2
03
8.0
0.
Eegle
MMG
R-
1.0
01
00
0.0
20
120
4,1
9.
60
64
00L
1.00
3.8
004
5.00
121
0.99
Am
176
66
Am
STRUCTORE
commeNT
Darg
SYRUCTURE
COMmENITY
LayeutLMMu
SUNJCT
302
E30
SUKICI
0.82
0.00
X
03
0.
0.0
99
60
30
90
00
00
00
62
3.8
00
00
PRD
K.0
9:11
1941
60
PRD-
(1:21
00/11
1.00
0007
1.00
D0
SHESSEN
10
HoM
SELL
Attention Mop of LayousENN3
Attention Nap L
Trom:
0.03
0.5L
0.01
0.2
00
EO
0.
03
Pomt
904
0.34
D:0
0.0
5.0
03
GO
D0
72:
60
0.31
0.
60
08
03
0/24
0:09
6.0
50
03
60
80075400
50075408
Times
0.8
6.0
00
0.
00
00
6011
03
TWm
00
3.0
0.
0.L
007
009
5/
60
0.2
60
22L
640
542
60
3.0
0.9
60
566
00
00
001
03
60
0/22
0.00
COMPATIIN IPTEONATTIN CONIELLIETRE
Mao Layou
Laysurt/tvs
0:91
FROM
0.81
OATE:
0.3
20
DATE
0.0
228
0.00
56
65
NANURICT LIEER
00
02
0.
003
0.0
03
300
VAKUFACTUFER
00
60
0.4
0.0
86
53
04
BRAND:
0.0
93
BRAND
0.02
C22
3.81
80
02
DEL
J0N.
Dr
640
3.81
TE:
DATE
BMASO
MAkSECTTME
Figure 4: Visualization of examples base on Lay3


Reviews:
[{"review_id": "wa3fuhFvu5", "reviewer": "Reviewer_MynQ", "summary": "", "questions": "A: What are the impacts of Gaussian kernel and polar coordinates on the CORD?\n\nB: Compared with methods based on cartesian coordinates, what are the advantages of polar coordinates?\n\nPlease refer to the comments in the Reasons To Reject for details.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for the reviewer's positive evaluation such as interesting ideas, effective results, etc. There are also very valuable questions and advice we try to answer as follows.\n>Ablation experiments were only conducted on one dataset, It would be better to conduct ablation experiments on more datasets. For example, what are the impacts of Gaussian kernel and polar coordinates on the CORD?\n\nThanks for such a constructive suggestion. We have conducted ablation experiments on the CORD dataset following the settings depicted in Sec. 4.4, and the experimental results on the validation set are shown in the table below. \n| Task | # | Ablation Strategy |  F1 score on CORD |\n| :-----| :---- | :---- | :----: |\n|   | 1 | RoBERTa(baseline) | 92.29 |\n|   | 2 |  + layout embedding layers | 92.67 |\n|  Impact of Gaussian | 3 |  + linear bias layers | 93.00 |\n|   | 4 |  + fixed kernel  | 93.95 |\n|  Impact of Polar-Coor. | 5 |  + Euclidean distance | 93.72 |\n|   | 6 |  + Angle | 94.70 |\n|   | 7 |  + 2D-xy distance |  94.21 |\n|  **ours** | **8** |  **+ LAGaBi** | **94.77** |\n\nThe ablation results on the CORD dataset align with the findings on the FUNSD dataset (as illustrated in Table 3 in Sec. 4.4). This suggests that both the Gaussian kernel and polar coordinates play a role in enhancing the baseline model's ability to comprehend structured documents. Thanks for your advice, and we will include these ablation results in Table 3 in our final version.\n\n>Compared with methods based on cartesian coordinates, what are the advantages of polar coordinates? I hope that the authors can provide a more in-depth analysis.\n\nThis is a very thought-provoking question, and we try to explain it as follows：\n\nPolar coordinates offer a representation of spatial relationships in terms of distance and angle, with the angle component being particularly valuable in capturing the relative direction between words within structured documents. This representation provides an intuitive understanding of how words are positioned with respect to each other. Considering common writing and reading habits (such as left-to-right and top-to-down), words within the same line tend to exhibit stronger semantic connections, while words on different lines are typically less semantically related. When taking a specific word as an anchor, words that are located farther away from the reference reading direction (e.g., words on different lines) generally exhibit larger relative angles compared to words along the reading direction. Thus, polar coordinates, which include angle information, prove to be more sensitive to layout changes compared to Cartesian coordinates that solely rely on distances.\nFurthermore, based on the findings from the ablation studies (as demonstrated in Table 3 in Section 4.4), it is evident that Polar coordinates surpass Cartesian coordinates in effectively modeling structured documents.\n>However, I think comparing the attention maps of LayoutLMv3 and LayoutLMv3+LAGaBi would be more meaningful for understanding the effect of LAGaBi.\n\nComparing the attention maps based on LayoutLMv3 is indeed a promising idea. We will incorporate this analysis in the appendix of the final version.\n\n>The authors miss some related work (see Missing References)\n\nThanks for the detailed advice! We have also noticed these excellent works you mentioned, and we will include discussions of these related works in our revised version. \n\n>About reproducibility\n\nOur method only depends on the simple learnable Gaussian kernels and does not involve introducing complex model structures, training objectives, or additional datasets. All experiments were conducted on publicly available datasets, making them easily reproducible. Once the paper is accepted, we will promptly release our source code and models on Github or HuggingFace repository, to foster the development of the entire community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of replacing the Cartesian coordinates with the polar coordinates is interesting.\n2. The paper is well written and easy to understand.\n3. The main experiment results show the proposed method is effective.", "reasons_to_reject": "1. Ablation experiments were only conducted on one dataset. It would be better to conduct ablation experiments on more datasets. For example, what are the impacts of Gaussian kernel and polar coordinates on the CORD?\n2. Compared with methods based on cartesian coordinates, what are the advantages of polar coordinates? I hope that the authors can provide a more in-depth analysis. For example, the authors compare the attention maps of RoBERTa and RpBERTa+LAGaBi. However, I think comparing the attention maps of LayoutLMv3 and LayoutLMv3+LAGaBi would be more meaningful for understanding the effect of LAGaBi.\n3. The authors miss some related work (see Missing References).", "questions_to_authors": "A: What are the impacts of Gaussian kernel and polar coordinates on the CORD?\n\nB: Compared with methods based on cartesian coordinates, what are the advantages of polar coordinates?\n\nPlease refer to the comments in the Reasons To Reject for details.", "paper_topic_and_main_contributions": "The authors propose the Layout Attention with Gaussian Biases. They leverage the polar coordinates to capture the layout information in the document images. Furthermore, they model an intuitive inductive layout bias, the words closer within a document should receive more attention, by feeding the distances and angles into a 2-D Gaussian kernel. They conduct experiments on three document image information extraction datasets to evaluate the performance of their method.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] mmLayout: Multi-grained MultiModal Transformer for Document Understanding. ACM MM 2022.\n\n[2] ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding. Findings of EMNLP 2022.\n\n[3] GeoLayoutLM: Geometric Pre-training for Visual Information Extraction. CVPR 2023."}}, {"review_id": "h4Lc3Q4cfw", "reviewer": "Reviewer_zPEC", "summary": "", "questions": "Question A：Why are there no experiments conducted on the DocVQA dataset?\nQuestion B：What are the differences between the layout bias in LAGaBi and the Spatial-Aware Self-Attention Mechanism in LayoutLMv2?\nQuestion C: Why are the scores for distant word pairs all 0 in the attention matrix, which is inconsistent with the actual situation?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for the reviewer's positive evaluation such as the lightweight method, significant improvement, etc, There are also very valuable questions and advice we try to answer as follows.\n>The innovation is somewhat limited. Using polar coordinates and the Gaussian function to calculate the relative position relationship matrix for bounding boxes is essentially analogous to computing 2D relative positions in document pre-training models like LayoutLMv2.\n\nThis is a very insightful opinion, and we try to make more detailed explanations as follows :\n\nIndeed, both LAGaBi and the Spatial-Aware Self-Attention Mechanism (SASAM) in LayoutLMv2 utilize relative positions, which has been a hot topic in NLP recently. However, it is important to note that there are *fundamental differences between them in terms of **motivations**, **methodologies**, and **impacts***.\n\n**1.Different motivations.**\n\nOur main motivation is to decouple position encoding from word embeddings and eliminate the need for overlay layout embedding layers. This allows the task of structured document understanding to be independent of specific model architectures, languages, and scale variations. On the other hand, SASAM aims to incorporate relative positional information on top of position embeddings, taking inspiration from the relative position encoding strategy in T5.\n\n**2.Different methodologies.**\n\n*(a)Different methods for representing relative positions.* LAGaBi using relative polar coordinates to model the relative positions between different words from a new perspective, modeling distances as well as directions. It aligns well with human reading habits and proves to be more sensitive to layout changes compared to solely considering distances. Conversely, SASAM employs a heuristic strategy for modeling relative positions by using discrete distances in the vertical and horizontal directions (implemented through bucketing), which is reminiscent of the 1-D relative positions commonly used in NLP tasks.\n\n*(b)Different strategies for mapping relative positions.* LAGaBi adopts simple Gaussian kernels to transform the relative positions (polar coordinates) into attention biases, with just 4× attention heads extra parameters to be learned. Moreover, this approach aligns better with human intuition, as it assigns larger correlation weights (attention biases) to word pairs with smaller distances or angles within a document page. SASAM, on the other hand, uses linear layers to convert relative positions into attention biases, requiring more parameters and a larger amount of training data to learn a proper mapping function that aligns well with human reading habits.\n\n**3.Different impacts.**\n\nLAGaBi, based on RoBERTa without any visual features, achieves impressive F1 scores of 89.15% and 96.56% on the FUNSD and CORD datasets, respectively, surpassing the multi-modal LayoutLMv2 with SASAM by 6.45% and 1.61%. Furthermore, LAGaBi can be effectively combined with SASAM to further enhance the model's performance. For instance, by incorporating LAGaBi, the F1 scores of LayoutLMv2 on the FUNSD and CORD datasets can be improved by 5.49% and 2.10%, respectively.\n\nOverall, the novelty lies in the important scenario where the technique is introduced, the effectiveness it enables, as well as potential changes and insights it inspires. We will add these discussions in our revised version to make this point more clear.\n\n>The evaluation datasets used in this paper are FUNSD and CORD, both of which are sequence labeling tasks. However, no comparison was made with metrics on the document-based question-answering dataset, like DocVQA.\n\nThanks for such a constructive suggestion.\nWe primarily focus on the text-centric tasks in structured documents, specifically information extraction tasks, and we have achieved significant results on both monolingual and multilingual scenarios. Additionally, we have also conducted experiments on tasks such as document question answering (DocVQA) and document image classification (RVL-CDIP) based on Roberta, with the following results:\n\n| Method | DocVQA (ANLS) | RVL-CDIP(accuray) | \n| :----- | :----: | :----: | \n| Roberta| 66.42 | 90.06 |\n| **Roberta+LAGaBi** (w/o pre-training) | **70.31** | **90.59**|\n\nThe results show that our method also brings improvements on DocVQA and RVL-CDIP over baseline (Roberta) without pre-training,  further supporting the claim that LAGaBi can make language models able to understand structured documents more accurately. \n\n>Why is there no pre-training experiment on the multi-modal transformer?\n\nThank you for appreciating and having high expectations for our work.\n\nThe core contribution of this paper is the proposal of the new paradigm in layout modeling, LAGaBi, which decouples the process of modeling layouts and texts, enabling transformer-based language models to understand structured data in a straightforward manner. This simple and pluggable layout modeling approach allows the task of structured document understanding to be independent of specific model architectures, languages, and scale variations. \n\nThrough our research, we have showcased significant improvements on 3 public monolingual/multilingual document datasets for 3 transformer-based language models (i.e. BERT, Roberta, and InfoXLM) with/without pretraining. Additionally, we have evaluated our method based on 3 powerful multi-modal transformer-based models (i.e. LayoutLMv1~v3) by fine-tuning, impressive performance on 2 datasets also demonstrating LAGaBi's ability. These results can strongly support the our claims and contributions.\n\nDue to limitations in computational resources (only 8 V100 GPUs), we were unable to conduct pre-training on the multi-modal Transformer models. We have mentioned this issue in the Limitations section. We are looking forward to acquiring enough resources for pre-training multi-modal models in the future, and we also encourage more researchers to validate and extend our method, contributing to the advancement of the entire community.\n\n> Why are the scores for distant word pairs all 0 in the attention matrix, which is inconsistent with the actual situation?\n\nThis is due to the rounding operation for better visualization, where the attention scores are rounded to two decimal places. In this case, attention scores for distant word pairs less than 0.01 are truncated to 0. As we can see in Figure 2, word pairs with an attention score displayed as 0 indeed have weak or no strong semantic correlations between them. Therefore, the rounding operation does not affect the understanding of the model's visualization results. To avoid any potential confusion, we will make a note of this in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The layout information is represented in polar coordinates and transformed into layout bias using Gaussian function, which directly influence the attention score without requiring additional layout embedding. This approach significantly reduces the number of parameters compared to the layout embedding method.\n- The proposed approach has demonstrated significant improvements on FUNSD and CORD datasets for both transformer and multi-modal transformer models.", "reasons_to_reject": "- The innovation is somewhat limited. Using polar coordinates and Gaussian function to calculate the relative position relationship matrix for bounding boxes is essentially analogous to computing 2D relative positions in document pre-training models like LayoutLMv2. \n- The evaluation datasets used in this paper are FUNSD and CORD, both of which are sequence labeling tasks. However, no comparison was made with metrics on the document-based question answering dataset, like DocVQA. \n- The experiment is incomplete. Why is there no pre-training experiment on the multi-modal transformer?", "questions_to_authors": "Question A：Why are there no experiments conducted on the DocVQA dataset?\nQuestion B：What are the differences between the layout bias in LAGaBi and the Spatial-Aware Self-Attention Mechanism in LayoutLMv2?\nQuestion C: Why are the scores for distant word pairs all 0 in the attention matrix, which is inconsistent with the actual situation?", "paper_topic_and_main_contributions": "This paper focuses on the task of structured document understanding and proposes a method that utilizes layout information. Unlike previous document pre-training methods that involve overlaying layout embeddings onto input embeddings, this approach incorporates the relative positional relationships between layout elements directly into the attention matrix using Gaussian function. The approach demonstrates positive outcomes in both transformers and multi-modal transformers. Specifically, the method transforms the representation of layout information from Cartesian coordinates to polar coordinates to compute the relative positional relationships between bounding boxes. Subsequently, it models the layout bias using Gaussian function and directly adds the obtained results to the attention matrix.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding(https://arxiv.org/pdf/2210.06155.pdf)\n2. StructuralLM: Structural Pre-training for Form Understanding(https://arxiv.org/pdf/2105.11210.pdf)"}}, {"review_id": "YZqAW5V56R", "reviewer": "Reviewer_NEpE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for the reviewer's positive evaluation such as the flexible method, promising insights, effective results, etc. There are also very valuable questions we try to answer as follows.\n\n>Marginal performance improvement for some backbone models (e.g., LayoutLMv3 in Table1, and the performance in several languages in Table2.).\n\nOur method achieves state-of-the-art results on all tasks, including FUNSD, CORD, and XFUND. In response to the reviewer's concerns, we provide the following explanations:\n\nLayoutLMv3 is currently the top-performing model for document understanding, which has achieved performance close to the upper bound of FUNSD and CORD. By incorporating our LAGaBi, the F1 scores on Funsd and CORD are further improved from 90.29% and 96.56% to 91.00% and 97.05% in Table 1, respectively, achieving the state-of-the-art with negligible extra computational cost.\n\nIn the multilingual scenario, InfoXLM serves as the backbone. InfoXLM+LAGaBi brings significant improvements on all three tasks: language-specific fine-tuning, zero-shot transfer, and multi-task fine-tuning, with average F1 gains of **11.99%**, **10.54%**, and **16.16%**, which is a highly remarkable achievement. We have excerpted the results of InfoXLM and InfoXLM+LAGaBi (w/ pre-training) from Table 2:\n| Task | Model |EN |  ZH |  JA |  ES |  FR | IT | DE | PT | Avg. | \n| :-----| :----- | :----: | :----: |  :----- | :----: | :----: |  :----- | :----: | :----: | :----: | \n| language-specific fine-tuning | InfoXLM | 68.52 |  88.68 | 78.65 |  62.30 | 70.15 | 67.51 | 70.63 | 70.08 | 72.07 | \n|| InfoXLM+LAGaBi |  84.17|  89.65 | 84.73 | 77.51 | 83.86 | 85.19 | 83.89 | 83.47 | 84.06| \n|| **gains**| **+15.65** | **+0.97** | **+6.08** |**+15.21** |**+13.71** |**+17.68** |**+13.26**|**+13.39** |**+11.99**\n||\n| zero-shot transfer | InfoXLM |68.52 |  44.08 |  36.03 |  31.02 |  40.21  | 28.80 |  35.87 |  45.02 |  41.19 | \n|| InfoXLM+LAGaBi |   84.17  | 39.88 |  35.77  | 44.82  | 54.45  | 46.72 |  51.64 |  56.37  | 51.73| \n|| **gains**| **+15.65** | -4.2 | -0.26 |**+13.8** |**+14.24** |**+17.92** | **+15.77** |**+11.35** | **+10.54**|\n||\n| multi-task fine-tuning | InfoXLM | 65.38 |  87.41 |  78.55  | 59.79 |  70.57  | 68.26  | 70.55 |  67.96  | 71.06 | \n|| InfoXLM+LAGaBi |  86.35 |  92.00 |  86.86 |  81.50 |  87.00 |  87.96 |  87.58  | 87.24 |  87.22| \n|| **gains**| **+20.97** | **+4.59** | **+8.31** |**+21.71**|**+16.43** |**+19.70** | **+17.03** |**+19.28** | **+16.16**|\n\nAccording above results, LAGaBi has shown significant effectiveness across all tasks and languages, except for Chinese and Japanese in the zero-shot transfer scenario. Zero-shot transfer in this paper refers to fine-tuning on an English dataset (FUNSD) and testing on the target language. English and Chinese/Japanese exhibit distinct differences in reading order and semantic dependencies. For instance, English typically uses spaces to separate words and has uneven word lengths, while Chinese have tighter sequencing with smaller semantic units (i.e., characters). Such variations in layout and semantic structures between English and Chinese/Japanese limit the effectiveness of learned layout knowledge from English when applied to Chinese/Japanese. This phenomenon also validates the efficacy of LAGaBi in modeling layouts, as it efficiently acquires language-specific layout knowledge after fine-tuning on corresponding datasets. \nWe have discussed this point in Sec.4.3.2 (from lines 479-492) of our summited paper.\n\n>Some critical comparison are missing. (e.g., LayoutLM v1-v3 W/ pre-training).\n\nThank you for appreciating and having high expectations for our work.\n\nThe core contribution of this paper is the proposal of the new paradigm in layout modeling, LAGaBi, which decouples the process of modeling layouts and texts, enabling transformer-based language models to understand structured data in a straightforward manner. This simple and pluggable layout modeling approach allows the task of structured document understanding to be independent of specific model architectures, languages, and scale variations. \n\nThrough our research, we have showcased significant improvements on 3 public monolingual/multilingual document datasets for 3 transformer-based language models (i.e. BERT, Roberta, and InfoXLM) with/without pretraining. Additionally, we have evaluated our method based on 3 powerful multi-modal transformer-based models (i.e. LayoutLMv1~v3) by fine-tuning, impressive performance on 2 datasets also demonstrating LAGaBi's ability. These results can strongly support the our claims and contributions.\n\nDue to limitations in computational resources (only 8 V100 GPUs), we were unable to conduct pre-training on the multi-modal Transformer models. We have mentioned this issue in the Limitations section. We are looking forward to acquiring enough resources for pre-training multi-modal models in the future, and we also encourage more researchers to validate and extend our method, contributing to the advancement of the entire community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A new method is proposed to address the structured document understanding (SDU) task, along with an investigation into two unresolved questions in this domain.\n\nThe proposed method can be flexibly combined with existing PLMs.\n\nThe proposed method brings performance gains for PLMs on SDU tasks.", "reasons_to_reject": "Marginal performance improvement for some backbone models (e.g., LayoutLMv3 in Table1, and the performance in several languages in Table2.).\n\nSome critical comparison are missing. (e.g., LayoutLM v1-v3 W/ pre-training).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on addressing the problem of efficient document layout representation learning for structured document understanding. It conducts an investigation into the efficiency of current methods, such as the Cartesian coordinate system and potential redundant training parameters. Based on the observations and analyses, the authors propose a new approach called \"Layout Attention with Gaussian Biases\" for the task of structured document understanding. The proposed method aims to enhance efficiency and effectiveness in capturing document layouts, and provides a promising insights for improving the representation learning process in this context.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce a novel attention module that leverages polar coordinates instead of cartesian coordinates to capture the layout of the images in the documents. While the reviewers raised some concerns about the experimental setup, most of them seem to have been addressed in the rebuttal.
Decision: Accept-Findings