Global Structure Knowledge-Guided Relation Extraction Method for Visually-Rich Document
Authors: Xiangnan Chen
Qian Xiao
Juncheng Li
Duo Dong
Jun Lin
Xiaozhong Liu
Siliang Tang
Venue: EMNLP 2023 Findings (2023)
Keywords: Visually-Rich Document
Visual relation extraction

Abstract: Visual Relation Extraction (VRE) is a powerful means of discovering relationships between entities within visually-rich documents. Existing methods often focus on manipulating entity features to find pairwise relations, yet neglect the more fundamental structural information that links disparate entity pairs together. The absence of global structure information may make the model struggle to learn long-range relations and easily predict conflicted results. To alleviate such limitations, we propose a GlObal Structure knowledge-guided relation Extraction (GOSE) framework. GOSE initiates by generating preliminary relation predictions on entity pairs extracted from a scanned image of the document. Subsequently, global structural knowledge is captured from the preceding iterative predictions, which are then incorporated into the representations of the entities. This ``generate-capture-incorporate'' cycle is repeated multiple times, allowing entity representations and global structure knowledge to be mutually reinforced. Extensive experiments validate that GOSE not only outperforms existing methods in the standard fine-tuning setting but also reveals superior cross-lingual learning capabilities; indeed, even yields stronger data-efficient performance in the low-resource setting.

Global Structure Knowledge-Guided Relation Extraction Method for Visually-Rich Document
Xiangnan Chen', Qian Xiao², Juncheng Li", Duo Dong', Jun Lin?, Xiaozhong Liu², Siliang Tang'
Zhejiang University 2 Alibaba Group 3 Worcester Polytechnic Institute (xnchen2020, junchengli, 22121222, siliang] edu. cn (xiaoqian. xq, linjun. waw.aaalababbaaaaae inc com, xliul 4@wpi. edu
Abstract
Tree Positive
False Positive
False Negative
Visual Relation Extraction (VRE) is power- ful means of discovering relationships between entities within visually-rich documents, Ex- isting methods often focus manipulating entity features to find pairwise relations, yet neglect the more fundamental structural infor- mation that links disparate entity pairs together. The absence of global structure information may make the model struggle learn long- range relations and easily predict conflicted re- sults. To alleviate such limitations, we propose G1Obal Structure knowledge-guided relation Extraction (GOSE) framework. GOSE initi- ates by generating preliminary relation predic- tions entity pairs extracted from scanned image of the document. Subsequently, global structural knowledge captured from the pre- ceding iterative predictions, which are then in- corporated into the representations of the enti- ties. This "generate-capture iipppurr-iccorporatt" cycle is repeated multiple times, allowing entity rep- resentations and global structure knowledge to be mutually reinforced. Extensive experiments validate that GOSE not only outperforms exist- ing methods in the standard fine-tuning setting but also reveals superior cross- cross-lingual learn- ing capabilities; indeed, even yields stronger data-efficient performance in the low-resource setting. The code for GOSE will be available at htttp:://gi thub. com/ httpccccnnnnnnnnnmm/
Wrapping
NO, OF
Tabels
Nhite
Closures
Bluse
Tear Taps
Nhite
Cartons
Markings
Sample No. on
each Carton
(a) Conflicted Links
(b) Missing Links
Figure 1: Incorrect relation predictions by the LiLT model on FUNSD.
a critical part of VrDU, has recently attracted exten- sive attention from both the academic and industrial communities (Jaume et al., 2019; Xu et al., 2021; Hwang et al., 2021; Wang et al., 2022a). The VRE task aims to identify relations between semantic entities in VrDs, severing as the essential basis of mapping VrDs structured information which is closer the human comprehension process of the VrDs (Zhang et al., 202 Recently, inspired by the success of pre-training in visually-rich doc- ument understanding (Li et al., 202 Xu et al., 2020; Wang et al., 2022a), many fine-tuning works predict relation for each entity pair independently, according to local semantic entity representations derived from the pre- mre-trained model.
Although existing VRE methods have achieved promising improvement, they ignore global struc- ture information, i.e. dependencies between en- tity pairs. Without considering global struc- ture knowledge, the model may easily predict conflicted links and struggle to capture long- range relations. Taking the state-of-the -het--rrtt model LiLT (Wang et al., 2022a) as an example, as shown in Figure ((a), although each relational entity pair predicted by the LiLT may make sense semanti- cally, there are conflicted relational entity pairs such as (Labels, Blue) and (Closures, White), as well as (Tear Tape, White) and (Cartons, White), whose link crosses each other from global view.
1 Introduction
Visually-rich document understanding (VrDU) aims to automatically analyze and extract key in- formation from scanned/digital-bomm documents, such as forms and financial receipts (Jaume et al., 2019; Cui et al., 2021). Since visually-rich doc- uments (VrDs) usually contain diverse structured information, Visual Relation Extraction (VRE), as
Work done during an internship at DAMO Research. Alibaba Group.
"orresponndig author.

---

This phenomenon indicates that methods only us- ing local features do not have the sufficient dis- criminatory ability for conflicted predictions. Fur- thermore, as shown in Figure 1(b), even though LiLT accurately identifies the relational entity pairs (No.OF STORES, 19) and (No.OF STORES, 18), LiLT still hardly learns long-range relations such (No.OP OF STORES, 16) and (No. STORES, 17). Our intuition is that global structure knowledge can help the model learn long-range relations. The model can predict the relational entity pair (No.OF STORES, 17) by analogy with the global structure consistency between (No.OF STORES, 17) and (No.OF STORES, 18).
Figure I(a), we can recognize crossover between entity pairs in 2D space by computing the spatial layout of linking lines. Furthermore, in order to increase the robustness of GOSE and handle the noisy mining process, we introduce an iterative learning strategy to combine the process of entity representations learning and global structure min- ing. The integration of global structure knowledge can help refine entity embeddings, while better entity embeddings can help mine more accurate global structure knowledge. In summary, the contributions of our work are as follows:
We propose a global structure knowledge- guided visual relation extraction method, named GOSE. It can use the spatial layout clue to mine global structure knowledge effec- tively and leverage the iterative learning strat- egy to denoise global structure knowledge.
In this paper, we present the first study on lever- aging global structure information for visual rela- tion extraction. We focus on how to effectively mine and incorporate global structure knowledge into existing fine-tuning methods. It has the fol- lowing two challenges: (1) Huge Mining Space. Considering N entities in a VrD, the computational complexity of capturing dependencies between en- tity pairs is quadratic to entity pair size (N2 x N²).
GOSE can be easily applied to existing pre- trained VrDU models. Experimental sults on the standard fine-tuning task over 8 datasets show that our method improves the average Fl performance of the previous SOTA models by a large margin: LiLT(+14.20%) and LayouIXLM(+12.885)
So it is difficult to mine useful global structure information in the lack of guidance.
(2) Noisy mining process. Since the process of mining de- pendencies between entity pairs relies initial entity representations and lacks direct supervision, global structure information learned by the model is likely to contain noise. Mined noisy global struc- ture knowledge by a model can in turn impair the performance of the model, especially when the model has low prediction accuracy in the early training stage.
We further perform comprehensive exper- iments covering diverse settings of VRE tasks, such as cross-lingual lcarning, and low- resource setting. Experimental results illus- trate advantages of our model, such as cross- lingual transfer and data-efficient learning.
2 Preliminaries
To this end, we propose general global struc- ture knowledge-guider visual relation extraction method named GOSE, which can efficiently and accurately capture dependencies between entity pairs. GOSE plug-and-plas which can be flex- ibly equipped to existing pre-trained VIDU mod- els. Specifically, we first propose a global struc- ture knowledge mining (GSKM) module, which can mine global structure knowledge effectively and efficiently. The GSKM module introduces a novel spatial prefix-guided self-attention mecha- nism, which takes the spatial layout of entity pairs the attention prefix to progressively guide min- ing global structure knowledge in local-global way. Our intuition is the spatial layout of entity pairs in VrDs may be a valuable clue to uncov- ering global structure knowledge. As shown in
In this section, we first formalize the visually-rich document relation extraction task and then briefly introduce how the task was approached in the past.
2.1 Problem Formulation
The input to the VRE task is a scanned image of a document. Each visually rich document contains a set of semantic entities, and each en- tity is composed of group of words and coor- dinates of the bounding box. We use lower- letter to represent semantic entity, where [[w+, w² ..., w*), [x+ y² x² y2]]. The quence [w], w² ..., ..., w²] means the word group. x! /2² and y /y² are left/right x-coordinates and top/down y-coordinates respectively. The corre- sponding boldface lower-case letter indicates its

---

Incorporate
Iterative Learning
RFG module
Iocorgurali
Rg Thae
Bodimax
Classifier
Captere
Tape
Enbiyy
Pooling
Reh-111
BidMig Muite
FFN
Capoure
Altantia
Amtoniin
4a (Dc ejy? (Ot By
VIDU Encoder
A
Finet
Allanio
glbbll loken
Cooeee
Profie quide
That Tigo
Loual attcaii Layer
fnleraciion Liger
GSKM module
module
R/6
Skd
(b)
Figure 2: An illustration of the proposed GOSE framework. (a) The overview of GOSE. The framework consists of a Base module to generate initial keylvalue features based on entity representations from a pre-trained VrDU model, relation feature generation (RFG) module to generate relation feature for each entity pair, and a global structure knowledge mining (GSKM) module to mine global structure information in local-globe way. Besides, the RFG and GSKM modules are performed K times in an iterative way to generate final predictions. construction. The spatial geometric features of entity pairs computed as the spatial prefix to guide attention.
(b) Spatial prefix
embedding. Let & and R represent the set of enti- ties and relations respectively, where & = [ R = {(ei, e)) C E x E, ez)) mean the key- value entity pair and the directed link from e; to ej- So each visually rich document can be denoted as D = {E, R}. The goal of VRE task is to determine whether a relation (link) exists between any two semantic entities. Notably, the semantic entity may exist relations with multiple entities or does not have relations with any other entities.
module, and a Global Structure Knowledge Min- ing (GSKM) module. In Section 3.1, we present how the Base module can be combined with pre- trained VrDU model to generate initial key/value features. In Section 3.2, we introduce the RFG module generate relation features. Then, in Sec- tion 3.3, we elaborate on how the GSKM module mines global structure knowledge in a local-global way. We further provide a theoretical analysis to better understand the efficient and effective features of the GSKM module. Finally, we show how to ap- ply our iterative learning strategy to mine accurate global structure knowledge in Section 3.4.
2.2 Fine-tuning for Visual Relation Extraction
Inspired by the success of pre-training in visually- rich document understanding, most existing meth- ods (Jaume al., 2019; Xu et al., 2021; Wang et al., 2022a) fine-tune pre-trained VIDU model for VRE task. These methods take entity representations from a pre-trained VIDU model as input and then train a binary classifier to predict relations for all possible semantic entity pairs. Specifically, these methods project entity representations to key/value features respectively by two FFN layers. The fea- tures of key and value are concatenated and fed into a binary classifier compute loss.
3.1 Base Module
Given a visually rich document, the Base mod- ule first generates preliminary key/value features, which is same as most fine-tuning methods (Xu et al., 2021; Gu et al., 2022; Wang et al., 2022a). Specifically, we use a pre-trained VrDU model obtain semantic entity representations. Then entity representation e is fed into two separated Feed- Forward Networks (FFN) to generate the initial key feature and value feature (denoted as II(() and (0) respectively), as written with Eq. (1):
3 Methodology
H
(0) Wheye + bkey:
(1)
In this section, we describe the proposed frame- work named GOSE detail. As shown in Fig- ure 2, our method consists of three main modules, a Base module, a Relation Feature Generation (RFG)
(0) W Wratue e + boalue:
where W key valure E Rdd, xd,, are trainable weights and b value E RRd are trainable biases.

---

3.2 Relation Feature Gneration (RFG) Module
the spatial information of linking lines between bounding boxes in 2D space). Therefore, inspired by the success of prompt learning (He et al.. 2022; Li et al., 2023a) we propose a spatial local self-attention (SPLS) layer to mine global structure knowledge in a parameter-efficient man- ner.
Taking keyvalue features as input, the RFG module generates relation features for all entity pairs. We denote key/value features at the 1-th iteration as ( and ( respectively, where H//e IR Nxd, At the t-th iteration, the RFG module concatenates H and _ as input and uses a bi-affine classifier to compute the relation logits of each entity pairs with (2):
Spatial Prefix Construction. We calculate spatial geometric features (denoted as S E XxNxx) of linking lines between each entity pair as the spatial prefix. As shown in Figure 2(b), for an entity pair (ei,e)), we calculate the direction and Euclidean distance of the line linking from the ver- tice (æi, 81) of the bounding box of e to the same vertice (xy, Uj) of ej as follows:
((0) 1 Wh H + (
(2)
where ((0) E RNXXX2 denotes relation logits at the t-th iteration. Then we employ FFN block to generate unified relation feature map (denoted R(O XxxNxd)), which contains relation fea- tures of all entity pairs. the relation feature map is calculated as follows:
9ij. [Wa (. j); Wi),,,
d(i.j) V(z: x2)² + (yi -y;)² (4)
O(i.j) arctan yj yi
ROO
(( + by,
(3)
Ij Iz
where wold E R da fore the spatial geometric features S..j of entity pair (Ci, Cÿ) calculated as follows:
are trainable weights. There-
where W, E 2 is trainable weight and by E Rd is trainable biases.
3.3 Global Structure Knowledge Mining (GSKM) module
St, = -oWl Diys s.yl.
(5)
where g", ge, ger indicate top-left, center, and bottom-right points respectively. Then we treat S as the spatial prefix and compute attention on the hybrid keys and values.
The GSKM module mines the global structure knowledge on the relation feature map in local- global way. As illustrated in Figure 2, this mod- ule consists of a spatial prefix-guided local self- attention (SPLS) layer and a global interaction layer. The SPLS layer first partitions the relation feature map into multiple non-overlapping win- dows and then performs spatial prefix-guided local self-attention independently in each window. The global interaction layer brings long-range depen- dencies to the local computation cost.
Spatial Prefix-guided Attention. We first parti- tion R(() into non-overlapping windows and then perform spatial prefix- prefix-guided -uudedd local attention within each local window. The variant formula of self- attention with the spatial prefix as follows !:
SW5 R (W....
Rocal softmax (R. (R. W,(SW:: Rd wal")
with negligible
S. = softmax (2 [Se; 1
(
3.3.1 Spatial Prefix-guided Local Self-attention (SPLS) Layer
= (1 ((R Attn t K (1)
slandbrr
To address the challenge of mining global structure knowledge in a large computational space, we con- sider the spatial layout information of entity pairs in VrDs to guide self-attention computation. Most of existing VIDU methods independently encode the spatial information of each entity in the pre- training phase (i.e., the coordinate information of the bounding boxes in 2D space). However, it is necessary to insert spatial layout information of entity pairs explicitly in the fine-tuning phase (i.e.,
X(R Aut((( S Sw.
spatial-peatia grided sttention
(6)
Expp(O STY, Σ. exp( +, exp K
Without loss of generalization, we ignore scaling fac- for Vd of the softmax operation for the convenience of expla- nation.

---

where T refers to the local attention output at the t-th iteration, A ()))) denotes the scalar for the sum of normalized attention weights on the key and value vectors from spatial prefix.
complexity N of self-attention ( x5))
(S 5)² = =2 2 S where S denotes window size. Meanwhile, the computation complexity of the global interaction layer (N² M) is negligible, as the number of global tokens M is much smaller than the window size S in our method.
3.3.2 Global Interaction Layer (GIL.)
After the SPLS layer effectively aggregates local correlations with window priors, we introduce global interaction layer bring long-range de- pendencies to the local self-attention. As illus- trated in Figure 2, we use learnable global tokens T E IR Mxdh to compute the global interaction at the t-th iteration as follows:
3,4 Iterative Learning
To alleviate the noisy mining process, we further propose an iterative learning strategy enable global structure information and entity embeddings mutually reinforce each other. Specifically, we in- corporate global structure knowledge into entity representations through a gating mechanism:
TO = Attn( (t)
K(t V(O))
R
(8)
R(00) global = Attn( Q
V())
- sigmoid W. (H (1 k/ui H by)
(0)
(10)
H(+1) k/u k/v + A7
H()
where R(t)
global refers to the global interaction at the t-th iteration. T will be updated in the same way as Hk/o throughout the iterative learning process. Subsequently, we compute R(t 1) and employ the mean pooling operation to obtain the context-aware key and value features as:
k/v
Finally, these new key and value features are fed back the classifier for the next iteration. After repeating this iterative process times, we get updated key and value features H(K) to compute final logits ((K), Finally, we calculate Binary Cross Entropy (BCE) loss based on ((K) as follows 2:
R(111) R(()
Rooal + RROO global
A = mean-pooling ( R(++)),
N
N e0
L
(11)
where (( contain global structure information. k/w Analysis of GSKM. Here we give some analysis to help better understand GSKM, especially effective and efficient features.
where Biy E [0,1] is binary ground truth of the entity pair (e,, {(., .) is the cross-entropy loss.
Effectiveness. GSKM can effectively learn global structure knowledge guided by spatial layout in- formation in VrDs. As shown in Eq. 6, the first (())) (t)) is the standard atten- tion in the content side, whereas the second term represents the 2D spatial layout guidelines. In this sense, method implements 2D spatial layout guide the attention computation in a way simi- lar linear interpolation. Specifically, the GSKM module down-weights the original content atten- tion probabilities by a scalar factor (i.e., A) and redistributes the remaining attention probability to attend to spatial- -reefix guided attention, which likes the linear interpolation.
Experiments
In this section, we perform detailed experiments to demonstrate the effectiveness of our proposed method GOSE among different settings. Besides the standard setting of typical language-specific fine-tuning (section 4.2), we further consider more challenging settings to demonstrate the generaliz- ability of GOSE such as cross-lingual zero-shot transfer learning (section 4.3) and few-shot learn- ing (section 4.4). Before discussing the results, we provide the details of the experimental setup below.
term Attn Q (t)
R
4.1 Experimental Setup
4.1.1 Datasets
Efficiency. GSKM can reduce the computation complexity 4 to 22 x 52, where S denotes win- dow size. In the SPLS layer, with relation fea- ture map R(t) NxNdx, as input, we parti- tion R(1) into con-overlapping windows with shape (A x N. x S, dn.) to reduce the computation
FUNSD (Jaume al., 2019) is scanned document dataset for form understanding. It has 149 training samples and 50 test samples with various layouts.
2To make a fair comparison with haselines, we use the same binary classification training strategy.

---

FUNSD
XFUND
Structure Information
Model
Avg.
EN
ZH
JA
ES
FR
DE
PT
XLM-RoBERTa
0,2659
0.5105
0,5800
0,5295
0,4965
0.5305
0,5041
0,3982
0,4769
Text-only
InfoXLM
0.2920
0.5214
0.6000
0,5516
0,4913
0.5281
0,3262
0.4170
0,4910
LayoutXLM
0.5483
0.7073
0.6963
0.6896
0.6353
0.6415
0,6551
0.5718
0.6432
Local
XYLayonLM
0.7445
0.7059
0.7259
0.6521
0.6572
0.6703
0,5898
LiTT
0.6276
0.7297
0.7037
0.7195
0.6965
0.7043
0.6558
0,5874
0.6781
GOSE
0.5926
0.8631
0.8258
0.8375
0.7729
0. 0.8035
0.7780
0.7026
0.7720
Global
GOS
0.7697
0,8752
0.8096
0.8595
0.8646
0.8415
0.8023
0.7384
0.8201 1814201
Table I: Language-specific fine-tuning Fl accuracy FUNSD and XFUND (fine-tuning on X, testing on X). "Text- only" denotes pre-trained textual models without structure information, "Local" denotes pre-trained VrDU models with local features, and "Global" denotes using global structure information.
FUNSD
XFUND
Structure Information
Model
Avg.
EN
ZH
JA
ES
FR
DE
PT
XLM-RoBERTa
0.2659
0.1601
0.2611
0,2440
0.2240
0.2374
0,2288
0.1996
0.2276
Text-only
InfoXLM
0.2920
0.2405
0.2851
0,2481
0.2454
0.2193
0.2027
0.2049
0,2423
LayoutXLM
0.5483
0.4494
0,4408
0.4708
0,4416
0.4090
0,3820
0,3685
0.4388
Local
LiTT
0.6276
0.4764
0.5081
0.4968
0.5209
0.4697
0.4169
0.4272
0.4930
0.5926
0.5696
0.5556
0.5124
0.5295
0.4168
0.4325
D.4363
0.5056 s000
Global
GOSE
0.7697
0.6930
0.6805
0.7072
0.7145
0.6355
0.5997
0,5830
0.6729
(412
Table 2: Cross- looss-lnngual zero-shot transfer Fl accuracy on FUNSD and XFUND (fine-tuning on FUNSD, testing on X).
XFUND (Xu et al., 2021) is a multilingual form understanding benchmark. It includes languages with 1,393 fully annotated forms. Each language includes 199 forms. where the training set includes 149 forms, and the test set includes 50 forms.
and report the average performance under five sam- pling times for a fair comparison. More details of the training hyper-parameters can be found in Appendix A.
4.2 Language-speciffe Fine-tuning
4.1.2 Baselines
We compare the performance of GOSE applied the language-specifee fine-tuning task. The experi- mental results are shown in Table First, all VRE methods with structure information outperform the text-only models XLBRR and InfoXLM, which indicates structure information plays an im- portant role in the VrDU. Second, while pre-trained VrDU models have achieved significant improve- ment text-only models, method still outper- forms them by a large margin. This phenomenon denotes that incorporating global structure informa- tion generally helpful for VRE. Compared to the SOTA method LiLT (Wang et al., 2022a), GOSE achieves significant improvements on all language datasets and has an increase of 14.20% Fl accuracy on the average performance. Third, we further ob- serve that our GOSE is model-agnostic, which consistently improves diverse pre-trained models' relation extraction performance on all datasets. For example, GOSE has an improvement of 12.88% Fl accuracy on average performance compared to LayoutLLM (Xu et al., 2021).
We use the following baselines: (1) text-only pre-trained models without structure information: XLM-RoBERT (Conneau et al., 2020), InfoXLM (Chi al., 2021); (2) layout-aware pre-trained VrDU models with local structure information: LayoutXLM (Xu et al., 2021), XYLayouLMM (Gu et al.. 2022), LiLT (Wang et al., 2022a). All of the experimental results of these baselines are from their original papers directly, except for the results on the few-shot learning task 3.
4.1.3 Experiment Implementation
We use the bi-affine classifier and binary classifi- cation training loss over all datasets and settings following (Wang et al., 2022a) for fair compar- ison. The entity representation is the first token vector in each entity. For the few-shot learning task, we randomly sample training samples over each shot five times with different random seeds,
"e re-implement the results using the official code.

---

Componentts
FI Accuracy1
Method
66-
GIL
Spatial Prefix
GSKM
Z11
9.5
GOSE
0.7697
0.8752
w/o GII.
X
/
0.7384
0,8643
6.7
w/o Spuial Prefix PpeiialPrreix
V
x
/
0.7029
0.8161
w/o GSKM
X
X
x
0.6902 0.8037
107
35
Table 3: Ablation study of our model using LiLT as the backbone on the FUNSD and XUND (ZH). The symbol EN denotes FUNSD and ZH means chinese language.
Numoer of critrlong samples
Figure 3: Average performances of few-shot setting on FUNSD dataset. We report the results of random sampling five times over each shot.
structure knowledge can improve the generaliza- tion of the model.
4.3 Cross-lingual Transfer Learning
We evaluate GOSE on the cross-lingual zero-shot transfer learning task. this setting, the model is only fine-tuned on the FUNSD dataset (in English) and evaluated on each specific language dataset. We present the evaluation results in Table 2. It be observed that GOSE significantly outper- forms its competitors and consistently improve di- verse backbone encoders' relation extraction perfor- mance on all datasets. This verifies that GOSE can capture the common global structure information invariance among different languages and transfer it to other languages for VRE. We observe that the performance improvement of GOSE(LayouXLLM) is not as significant as GOSE(LiLT). This may be attributed to that the architecture of LiLT decou- pling the text and layout information makes it easier learn language-indee independent structural informa- tion, which is consistent with the observation of previous works (Wang et al., 2022a). We further evaluate GOSE on the Multilingual learning task, the results are shown in Appendix B.
4.5 Ablation Study
Effectiveness of individual components. We fur- ther investigate the effectiveness of different mod- ules in our method. we compare our model with the following variants in Table 3.
(1) w/o GIZ. In this variant, we remove the global interaction layer from GSKM. This change means that the GSKM module only performs local self- attention. The results shown in Table 3 suggest that our GIL can encourage the GSKM module to better exploit dependencies of entity pairs.
(2) w/o Spatial-Prefix, In this variant, we remove the spatial prefix-guided interaction. This change causes a significant performance decay. This sug- gests the injection of additional spatial layout infor- mation can guide the attention mechanism attend- ing to entity pairs with similar spatial structure and thus help the model learn global structural knowl- edge.
(3) w/o GSKM. In this variant, remove the GSKM module from GOSE. This change means the model only obtains context-aware relation fea- tures through a mean-pooling operation. The re- sults shown in Table indicate that although the mean-pooling operation can achieve a performance improvement, the GSKM module can mine more useful global structure information.
4.4 Few-shot Learning
Previous experiments illustrate that our method achieves improvements using full training sam- ples. We further explore whether GOSE could mine global structure information in the low-resource set- ting. Thus, we compare with the previous SOTA model LiLT on few-shot settings. The experimen- tal results in Figure 3 indicate that the average per- formance of GOSE still outperforms the SOTA model LiLT. Notably, our GOSE achieves com- parable average performance (67.82%) FUNSD dataset using only 64 training samples than LiLT does (62.76%) using full training samples, which further proves that our proposed method can more efficiently leverage training samples. This success may be attributed to the incorporation of global
Ablation Iteration Rounds. The highlight of our GOSE is mining global structure knowledge and refining entity embeddings iteratively. We ar- gue that these two parts can mutually reinforce each other: the integration of global structure knowledge can help refine entity embeddings. On the contrary, better entity embeddings can help mine more accu- rate global structure knowledge. Thus, we evaluate the influence of the iteration rounds. The results are shown in 5(a), which indicates GOSE usually achieves the best results within small number

---

TO:
Sam Zolot
TO:
Sam Zolot
Soctioon Son
Do Son
27
FROM
D. Landio
FROM:
D. Landro
DATE: N 9-Dec-97
DATE:
9-Dec-97
58 14
Presiure Deag
400 2a8
Pressure Brmp
890
530000
530000
24,
24.45
.GE MER.
FEER (8%) MBRS
Nadghe:
200
35/
5300G1
530003
Plant.
7% Ront
|Plaat.
76 Sant
RAARi
1581085B118
RRABE
15810858116
86-52-28
94-12-20
PRA 1R
A800
PIR 1800
Plag Hemp
(a) Conflicted Links
(b) Missing Links
Figure 4: The visualization of examples in FUNSD and XFUND(ZH), The left of each example is LiLT and the right is GOSE(LiLT)) The arrows blue, red and orange denote true positive, false positive and false negative (missed) relations respectively. Best viewed by zooming up.
0.5
of the entity "Section A", due to there are few top- to-bottom and one-to-many linking patterns.
0.64
Co.5
0.000
0.010
0.020
0,025
0,035
0.4-
(ra,, Sam zolod
(b)
Figure 5: performances our model using LiLT as the backbone. The symbol EN denotes FUNSD dataset and ZH means XFUND(in chinese). (a) Ablation of iteration rounds. (b) Ablation of global tokens.
CPD: Sam Zalleg
Landro)
ITS 08:-997
(PREat delay)
(PROM ajtandro)
(DATE: Sam Selay)
(DATE (aLande)
(DATE:
Figure The attention weight visualization between the entity pair (TO:, Sam Zolot) and other spatial prefix of entity pairs.
iteration rounds. In addition, we investigate the per- formance of different information under multiple iterations in Appendix C.
Visualization of Attention over Spatial Infor- mation. To illustrate the effect of our proposed spatial prefix-guided local self-attention. We cal- culate the attention scores over the spatial infor- mation for the document in Figure 5(a), i.c., the spatial-prefix guided attention weights using Equa- tion (6). As shown in Figure 6, the entity pair (TO:, Sam Zolot) pays more attention towards the entity pair (FROM: D.J.Landro) and (DATE: 2-DEC- 97). This phenomenon indicates that the injection of additional spatial layout information of entity pairs can guide the attention mechanism attending to entity pairs with similar spatial structure, thereby enhancing the capacity of the model to discern pre- cise dependencies between entity pairs.
Ablation of Global Tokens. We further investigate the effect of the number of global tokens in the GIL on our model. The results are shown Figure (b), which denotes GOSE can achieve optimal re- sults within a small number of global tokens, while keeping efficiency.
4.6 Further Analysis
Case Study. To better illustrate the effectiveness of global structure knowledge, conduct the spe- cific case analysis on the VRE task shown in Figure& 4. Through the visualization of examples, we can notice: (1) as shown in Figure 4(a), GOSE can greatly mitigate the prediction conflicted links which reveals that our method can capture global structure knowledge to detect conflicted in- teractions between entity pairs. (2) as shown in Figure 4(b), GOSE can learn long-range relations by analogy with the linking pattern of entity pairs, while keeping good recall. Notably, is also difficult for GOSE to predict long-range relations where is not sufficient global structure knowledge. For example, GOSE does not predict well relations
Related Works
5.1 Visual Relation Extraction
Visual relation extraction (VRE) aims at identifying relations between semantic entities from visually- rich documents. Early methods based on graph neural networks (Zhang et al., 2021b; Tang et al., 2021; Li et al., 2021b, 2022b, 2023c) learned node features by treating semantic entities as nodes

---

graph. Recently, most studies (Li al., 2020; Wang et al., 2022a: Gu al., 2022; Huang al., 2022b), used the self-supervised pre-training and fine-tuning techniques to boost the performance on document understanding. Although they have achieved sigmificant improvement on VIE tasks, es- pecially on the semantic entity recognition (SER) task (Peng et al., 2022). VRE remains largely un- derexplored and is also challenging task. In this paper, we focus on mining global structure knowl- edge to guide relation extraction. To the best of our knowledge, our work is the first attempt to exploit dependencies of entity pairs for this task.
7 Conclusion
In this paper, we present a general global struc- ture knowledge-guided relation extraction method for visually-rich documents, which jointly and it- eratively learns entity representations and mines global dependencies of entity pairs. To the best of our knowledge, GOSE is the first work lever- aging global structure knowledge to guide visual relation extraction. Concretely, we first use base module that combines with pre-trained model to obtain initial relation predictions. Then, further design relation feature generation module that generates relation features and a global structure knowledge mining module. These two modules perform the cecnerate-ccaptuu--icooppoo cess multiple times to mine and integrate accurate global structure knowledge. Extensive experimen- tal results on three different settings (e.g., standard fine- totiiniing, cross-linguat learning, low-resource setting) over eight datasets demonstrate the effec- tiveness and superiority of GOSE.
5.2 Efficient Transformers
pro-
Efficient transformers (Dong et al., 2019; Li et al., 2019; Tay et al., 2020; Beltagy et al., 2020; Ryoo al., 2021: Zhang et al., 2021a: Li et al., 2022c) are class of methods designed to address the quadratic time and memory complexity problems of vanilla self-attention. More similar to us are methods that leverage down-sampling to reduce the resolution of the sequence, such as window-based vision transformers (Liu et al., 2021; Yang et al., 2021; Huang et al., 2022a; Wang et al., 2022b). Different from existing methods, we propose the spatial prefix-guides attention mechanism, which leverages spatial layout properties of VrDs to guide GOSE to mine global structure knowledge.
Acknowledgments
We would like to express gratitude to the anony- mous reviewers for their kind comments. This work has been supported in part by the Zhejiang NSF (LR21F020004), Key Research and Development Projects Zhejiang Province (No. 2023C01030, 2023C01032), NSFC (No. 62272411), National Key Research and Development Program of China (2018AAA0001900), Ant Group and Alibaba- Zhejiang University Joint Research Institute of Frontier Technologies. Our work was also sup- ported by Scientific Research Fund of Zhejiang Provincial Education Department.
6 Discussion
Recently, The Multimodal Large Language Model (MLLM) (Li et al., 2022a; Yin et al., 2023) has emerged as a pioneering approach. MLLMs lever- age powerful Large Language Models (LLMs) as a brain to perform multimodal tasks. The sur- prising emergent capabilities of MLLM. such as following zero-shot demonstrative instructions (Li al., 2023b) and OCR-free math reasoning (Zhu al., 2023; Dai al., 2023), are rare in tradi- tional methods. Several studies (Xu et al., 2023; Liu et al., 2023) have conducted comprehensive evaluations of publicly available large multimodal models. These investigations reveal that MLLMS still struggle with the VIE task. In this paper, we introduce and empirically validate that global struc- tural knowledge is useful for visually-rich docu- ment information extraction. Our insights have the potential to shape the advancement of large model technology in the domain of visually-rich documents.
Limitations
The proposed work still contains several limitations to address in future work, as follows: Method. One limitation of our method is that it cannot capture global structure information from informative visual clues. The visual features visually-rich documents such as font size and color may provide diverse structure information. For example, section titles in resumes and job ads are often in fonts different from the content. We leave this for future work.
Task. We only evaluate the visual relation extrac- tion task covering diverse settings. Due to the lim- ited budget and computation resources, we cannot

---

afford evaluation on more tasks related visually- rich documents. We will plan to evaluate the pro- posed approach on more visual information extrac- tion tasks such as semantic entity recognition.
Wonseok Hwang, Jinyeong Yim, Seunghyun Park, So- hee Yang. and Minjoon Seo. 2021. Spatial depen- dency parsing for semi-structured document informa- tion extraction. In Findings of ACL.
Guillaume Jaume et al. 2019. FUNSD: A dataset for form understanding in noisy scanned documents. In ICDAR, volume 2. pages 1-6.
References
Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-documest transformer. arXiv:2004.05150.
Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2021a. StructuralL Structural pre-training for form understanding. In ACI.
Zewen Chi, Li Dong, Furu Wei, Nan Yang. Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, He- Yan Huang, and Ming Zhou. 2021. InfoXLM: An cnermattion-thooetii framework for tross-lingual language model pre-training. In NAACL-HLT, pages 3576-3588.
Juncheng Li, Minghe Gao, Longhui Wei, Siliang Tang, Wenqiao Zhang, Mengze Li, Wei Ji, Qi Tian, Tat- Seng Chua, and Yueting Zhuang. 2023a. Gradient- regulated meta-prompt learning for generalizable vision-language models. In Proceedings of the IEEF/CV International Conference on Computer Vision.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In ACL, pages 8440-8451.
Juncheng Li. Xin He. Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Sil- iang Tang. 2022a. Fine-grained semantically aligned vision-language pre-training. Advances in neural information processing systems, 35:7290-7303.
Lei Cui, Yiheng Xu, Tengchao Lv, and Furu Wei. 2021. Document AI: benchmarks, models and applications. arXiv preprint arXi::2111.08609
Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang. Wei Ji. Wenqiao Zhang, Tat- Seng Chua, Siliang Tang, and Yueting Zhuang. 2023b. Fine-tuning multimodal 1lms to follow zero- shot demonstrative instructions. arXiv preprint arXiv:2308.04155
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. 2023. In- structblip: Towards seneral-purpose lssiun-language models with instruction tuning.
Juncheng Li, Siliang Tang, Fei Wu, and Yueting Zhuang. 2019. Walking with mind: Mental imagery enhanced embodied qa. In Proceedings of the 27th ACM In- ternational Conference on Multimedia, pages 1211- 1219.
Li Dong, Nan Yang, Wenhui Wang. Furu Wei, Xi- aodong Liu, Yu Wang. Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. Proceedings of NeurlPS.
Juncheng Li, Siliang Tang, Linchao 7hu, Haochen Shi, Xuanwen Huang. Fei Wu, Yi Yang, and Yueting Zhuang. 2021b. Adaptive hierarchical graph reason- ing with semantic coherence for indo-andd-lanuugge inference. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 1867- 1877.
Zhangxuan Gu, Changhua Meng. Ke Wang. Jun Lan, Weigiang Wang, Ming Gu, and Liqing Zhang. 2022. Xylayoutim: Towards layout-aware multimodal net- works for visually-riet document understanding, In Proceedings of the IEEF/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 4583-4592.
Juncheng Li, Siliang Tang, Linchao Zhu, Wenqiao Zhang, Yi Yang. Tat-Seng Chua, and Fei Wu. 2023c. Variational cross-graph reasoning and adaptive struc- tured semantics learning for compositional temporal grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Towards a unified view of parameter-efficient transfer learning.
Tao Huang, Lang Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. 2022a. Lightvit: Towards light- weight convolution- vision transformers. arXiv preprint 2rxXx:2207.0555:
Juncheng Li, Xin Wang, Siliang Tang. Haizhou Shi, Fei Wu, Yueting Zhuang, and William Yang Wang. 2020. Unsupervised reinforcement learning of transferable meta-skills embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12123-12133
Yupan Huang. Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022h. I Layoutlmv3: 3: Pre-training for doc- ument ai with unified text and image masking. In Proceedings of the 30th ACM International Confer- ence on Multimedia.
Juncheng Li, Junlin Xie, Long Qian, Linchao Zhu, Siliang Tang, Fei Wu, Yi Yang, Yueting Zhuang,

---

and Xin Eric Wang. 2022b. Compositional tem- poral grounding with structured variational cross- graph correspondence learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3032-3041.
Yi Tay, Mostafa Dehghami, Dara Bahri, and Donald Met- zler. 2020. Efficient transformers: A survey. CoRR. 2092009........
Jiapeng Wang, Lianwen Jin, and Kai Ding. 2022a. LiLT: A simple yet effective Eonguage-indeeendee layout transformer for structured document understanding. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (ACL), pages 7747-7757.
Juncheng Li, Junlin Xie, Linchao Zhu, Long Qian, Sil- iang Tang, Wenqiao Zhang, Haochen Shi, Shengyu Zhang, Longhui Wei. Qi Tian, and Yueting Zhuang. 2022c. Dilated context integrated network with cross- modal consensus for temporal emotion localization videos. In Proceedings of the 30th ACM Interna- tional Conference on Multimedia.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wen- gang Zhou, Jianzhuang Liu, and Houqiang Li. 2022b, Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 17683-17693.
Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Yang Liu, Biao Yang, Mingxin Huang, Dezhi Peng. Mingyu Liu, Mingrui Chen, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. 2023. On the hidden mystery of ocr in large multimodal models.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu. Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2023. Lvim-ehub: A com- prehensive evaluation benchmark for large vision- language models.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10012-10022.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. LayoutLM: Pre-training of text and layout for document image understanding. pages 1192-1200.
In
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In ICLR.
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang. Yi- juan Lu, Dinei Florencio, Cha Zhang, and Furu Wei. 2021. LayoutXLM: Multimodal pre-training for multilingual visually-rich document understanding. arXiv preprint 2eXix:204408833:
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, nigh-performance deep learning library. In Proceedings of the Annual Conference on Neural Information Processing Systems, pages 8024-8035.
Jianwei Yang. Chunyuan Li, Pengchuan Zhang. Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. 2021. 1. Focal attention for long-range interactions in vision transformers. In Advances in Neural Information Processing Systems.
Shukang Yin. Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Fnnhong Chen. 2023. A survey on multimodal large language models.
Qiming Peng. Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang. Yuhui Cao, We- ichong Yin, Yongfeng Chen, Yin Zhang. Shikun Feng. Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2022. Ernie-aaoou: Layout knowledge enhanced pre- training for visually-rich document understanding. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 3744-3756.
Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021a. Poolingformer: Long document modeling with pool- ing attention. Proceedings of ICML.
Yue Zhang, Zhang Bo, Rui Wang, Junjie Cao. Chen Li, and Zuyi Bao. 2021b. Entity relation extraction as dependency parsing in visually rich documents. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 2759-2768.
Michael Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. Tokenlearner: Adaptive space-time tokenization for videos. In Ad- vances in Neural Information Processing Systems, pages 12786 -17977.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vasion-language understanding with advanced large language models.
Guozhi Tang, Lele Xie, Lianwen Jin, Jiapeng Wang. Jingdong Chen, 7hen Xu, Qianying Wang, Yaqiang Wu, and Hui Li. 2021. MatchVIE: Exploiting match relevancy between entities for visual information ex- traction. In ICCAI pages 1039-1045.

---

A Hyperparameters
0.8
All of our experiments are performed on one NVIDIA 3090 GPU with the Py (Paszke et al., 2019) framework. We use a mini-batch AdamW (Loshchilov and Hutter, 2018) optimizer with a weight decay of 0.1. The model is trained with batch size of 2. The window size is fixed to 64. In the Language-Specinng Fine-tuning experi- ments for all languages, the learning rate, steps set 6.25 10-6 10 accordingly on LiLT en- coder, while 2.5 x 10 --4 ,4 x 10 on the Layoutxlm encoder. In the multilingual learning experiments, we use the full trnguage-speciiieg data for train- ing, with total steps 1.6 x 10 In the cross-lingual zero-shot transfer learning experiments, we directly evaluate the model, which was trained in the previ- ous kunguage-speciffe experiments, on the XFUND dataset.
0.6
0.4
gicbel atruthrra rnsermation
(Bicks struature Jesturme cels
Berradnn rounds
Figure Fl performances of different information of GOSE (LiLT) under multiple iterations FUNSD dataset.
that (1) when the iterative learning process begins, both global structural features and entity representa- tions progressively enhance and mutually reinforee each other. (2) In scenarios where the number of iteration rounds stands at zero, i.c., without itera- tive learning. The performance of global structure information is poor. This may be because the initial entity representations obtained from the pre-trained model are not strong, thus the mined structural in- formation without iterative optimization is noisy. (3) As the number of itcration rounds increases certain point, the performance of the model de- creases. This phenomenon can attributed to too many iteration rounds that can cause mined glohal structural information to become over-smoothing thus affecting the final performance of our model.
B Multilingual Learning
We evaluate GOSE on the multilingual learning set- ting. In this setting, the model is fine-tuned with all 8 languages simultaneously and evaluated on each specific language. From the experimental results shown in Table 4, we can find that although this setting further improves the baseline model per- formance compared to the language-specific fine- tuning. our method GOSE once again outperforms its counterparts by large margin. We hold that the superior performance of our method GOSE can be attributed the fact that the previous method still does not learn sufficient global structure infor- mation multilingual learning. This finding also demonstrates that mining global structure informa- tion is beneficial for the VRE task.
Model
FUNSD XFUND (Avg.)
XIMMRRBERTa
0.3638
0.6727
Text-only
InfoXLM
0.3699
0.6495
LayoutXLM
0.6671
0,7988
Local
LiLT
0.7407
0.8228
G.SE LMr
0.7755
0,8656
Global
GOSELILT
0.9003
0.8994
Table 4:
Multilingual fine-tuning F1 accuracy
FUNSD and XFUND (fine-tuning on 8 languages all, testing on X).
Effect of Different Information
We show the performance of our GOSE (LiLT) un- der multiple iterations in Figure We can observe


Reviews:
[{"review_id": "zwI4sVC1Dm", "reviewer": "Reviewer_5vV3", "summary": "", "questions": "Please see the \"Reasons to Reject\" above.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 5vV3\n\nWe sincerely thank you for the valuable comments. We are encouraged to see that our work is recognized as novel and clear. We will explain your concerns point by point.\n\n> **Q1:** Lack of explanations of how the local- and global-structure information can be learned, and visualization is suggested to convenience audiences.\n\n**A1:** In our paper, the term \"local structure information\" refers to entity representations, which are derived from a pre-trained model serving as the base module. On the other hand, \"global structure information\" entails the exploration of associations between entity pairs. Notably, the interplay between local and global structure information is mutually reinforcing throughout the iterative learning process. We add an attention weights analysis to better illustrate how our GOSE accurately captures dependencies between entity pairs. We calculate the attention scores over the spatial information for the document in Figure 5(a), i.e., the spatial-prefix guided attention weights using Equation (6). We provide the results of attention weights over the spatial prefix in the following table. As shown in the following table,  the entity pair (TO, Sam Zolot)  pays more attention towards the entity pair (FROM:, D.J.Landro) and  (DATE:, 2-DEC-97).  This phenomenon indicates that the injection of additional spatial layout information of entity pairs can guide the attention mechanism attending to entity pairs with similar spatial structure, thereby enhancing the capacity of the model to discern precise dependencies between entity pairs.  As you nicely suggested, we will include the visualization of attention weights in the next version. \n\ni.  The spatial-prefix guided attention weights of the entity pair (TO, Sam Zolot) over spatial prefix of other entity pairs.\n\n|                 | (TO, Sam Zolot) | (TO, D.J.Landro) | (TO, 2-DEC-97) | (FROM:, Sam Zolot) | (FROM:, D.J.Landro) | (FROM:, 2-DEC-97) | (DATE:, Sam Zolot) | (DATE:, D.J.Landro) | (DATE:, 2-DEC-97) |\n| :-------------: | :-------------: | :--------------: | :------------: | ------------------ | ------------------- | ----------------- | ------------------ | ------------------- | ----------------- |\n| (TO, Sam Zolot) |   **0.0189**    |      0.0338      |     0.0318     | 0.0064             | **0.0097**          | 0.0340            | 0.0054             | 0.0064              | **0.0103**        |\n\n> **Q2:** There are many typos scattered, and careful proofreading is suggested. e.g. Line 402: \"we random sample training samples\" \"random\"-->\"randomly\" Line 413: \"which indicate structure information plays\" \"indicate\"-->\" indicates\" Line 437: \"It can be observed that GOSE ..... and consistently improve\" \"improve\"-->\"improves\" \n\n**A2:** Thank you for your review and guidance. We will proceed with careful proofreading to ensure the submitted paper is grammatically and spelling-wise impeccable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation of this paper is clear\n2. This paper is well-written and can be easily followed. \n3. The proposed local- and global- learning procedure seems novel.\n4. Extensive experiments with promising results.\n", "reasons_to_reject": "1. Lack of explanations of how the local- and glocal-structure information can be learned, and visualization is suggested to convenience audiences. \n\n2. There are many typos scattered, and careful proofreading is suggested. e.g.\nLine 402:   \"we random sample training samples\"   \"random\"-->\"randomly\"\nLine 413:    \"which indicate structure information plays\"   \"indicate\"-->\" indicates\"\nLine 437:    \"It can be observed that GOSE ..... and consistently improve\"   \"improve\"-->\"improves\"\n\n", "questions_to_authors": "Please see the \"Reasons to Reject\" above.", "paper_topic_and_main_contributions": "This paper studies visual relation extraction and introduces a novel method to learn the global structure of the document to guide the extraction.  This paper tackles two challenges to effectively mine and incorporate global structure knowledge, including the vast mining space for capturing dependencies between entity pairs and the noisy mining process due to the lack of direct supervision. Experimental results on cross-lingual learning and few-shot learning show the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "Please see the \"Reasons to Reject\" above."}, "extra_text": {}}, {"review_id": "3R8D8Mn7th", "reviewer": "Reviewer_ZM3X", "summary": "", "questions": "This paper presents very impressive results, with huge improvement over baseline methods. However, from the ablation study, the most useful part is the iterative learning. The performance of the method surprisingly dropped from F1=0.7697 to 0.1599 when removing the iteration steps. This raises concern on over-fitting, as multiple rounds of iterations can result in over-parameterization and overfitting, particularly when considering that the datasets used in the studies are relatively small and there seems no validation datasets for hyperparameter tuning. It would be useful to extract a subset of a training dataset as a validation dataset from to eliminate this concern. \n\nSome minor comments are: \n1. As one of the most important hyper-parameter, It is unclear how many iterations were used in the experiments. It might not be unexpected to see that the best performance was achieved within a small number of iterations, as a large number of iterations can diminish representational power of spatial information. Also, how are the global tokens T initialized? \n\n2. I wonder how the method would deal with entities that spread across multiple pages in a document, and how to adjust the distance and other functions for these cases.\n\n3. It is great to see that the proposed GOSE method is plug-and-play. In experiments, GOSE was used with LayoutXLM or LiLT. It would be useful to mention how GOSE can be plugged with other model in the Method section.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer ZM3X\n\nWe sincerely thank you for your comprehensive comments and constructive advice. We will explain your concerns point by point.\n\n> **Q1:** Suspicious improvement through iterative learning steps. This paper presents very impressive results, with huge improvement over baseline methods. However, from the ablation study, the most useful part is the iterative learning. The performance of the method surprisingly dropped from F1=0.7697 to 0.1599 when removing the iteration steps. This raises concern on over-fitting, as multiple rounds of iterations can result in over-parameterization and overfitting, particularly when considering that the datasets used in the studies are relatively small and there seems no validation datasets for hyperparameter tuning. It would be useful to extract a subset of a training dataset as a validation dataset from to eliminate this concern.\n\n**A1:** Thanks for your question. The performance of the method surprisingly dropped when removing the iteration steps is mainly due to the fact that we only use the mined global structural information to make predictions without fusing entity representations (using Equation (9)). **Since the initial entity representations obtained from the pretrained model is not strong, the mined structural information without iterative optimization is noisy**.  The iterative learning process can refine the mined global structural information and enhance the richness of entity representations. This concerted effort leads to a notable enhancement in the performance of model. \n\nTo illustrate this point more clearly, We extract a subset of the training dataset as a validation dataset to perform the hyper-parameter tuning. When tuning the iteration rounds, we keep other hyper-parameters constant. We add the experiment of different information of our GOSE (LiLT) under multiple iterations over FUNSD dataset in the following table. \n\n|                         Information                          | Rounds 0 | Rounds 1 | Rounds 2 |  Rounds 3  | Rounds 4 | Rounds 5 | Rounds 6 | Rounds 7 |\n| ---------------------------------------------------------- | :---------------------------------------------: | :------: | :------: | :--------: | :------: | :------: | :------: | :------: |\n|                Global structure features only                |                     0.1599                      |  0.5505  |  0.7135  | **0.7238** |  0.7071  |  0.6797  |  0.6150  |  0.6078  |\n| Entity representations  incorporating global structural information | 0.6276*  |  0.7274  |  0.7628  | **0.7697** |  0.7527  |  0.7511  |  0.7466  |  0.7426  |\n\nAs shown in above table, the first line demonstrates the performance of only using mined global structural information, the second line demonstrates the performance of using entity representations incorporating global structural information. We can observe that:\n\n(1) When iterative learning process begins, both global structural features and entity representations progressively enhance and mutually reinforce each other. For example, by executing one iteration round (rounds = 1), the performance of only global structure features improves from 0.1599 (absent iteration) to 0.5505. Similarly, the performance of entity representations advances from 0.6276 (without global structural information, rounds=0) to 0.7274 (incorporating global structural information, rounds=1). \n\n(2) As the number of iteration rounds increases to a certain point, the performance of the model decreases. This phenomenon can be attributed to too many iteration rounds can cause mined global structural information to become over-smoothing thus affecting the final performance of our model.\n\nIn addition to the above observations, we further extract a subset of a training dataset as a validation dataset to hyperparameter tuning to eliminate the raised concern on over-fitting.  Please refer to our reply to the Q4 of you. We will include these discussions in the next version according to your valuable suggestion.\n\n\n\n> **Q2:** Relatively small datasets.\n\n**A2:** We sincerely appreciate your thoughtful feedback on our work. We completely acknowledge the significance of datasets in establishing the generalizability of our results.  Thus, we choose widely recognized and publicly accessible datasets for the purpose of meaningful comparison.  Although the size of each dataset is relatively small,  we integrated FUNSD and XFUND datasets to explore more challenging settings. For instance, we evaluate our GOSE on the cross-lingual zero-shot transfer learning task (in Section 4.3). In this setting, the model is only fine-tuned on the FUNSD dataset and directly evaluated on the XFUND dataset. At the same time, we merged the FUNSD and XFUND datasets into a relatively large dataset to evaluate the multilingual learning task (in Appendix 2). In this setting, the model is fine-tuned with all 8 languages simultaneously and evaluated on each specific language.  These comprehensive settings demonstrate the robustness and effectiveness of our GOSE across diverse contexts. Furthermore, We notice some works [A, B] published results on large-scale yet private datasets. We will conduct experiments on larger datasets when these large-scale dataset are publicly available.  \n\n[A] Chargrid: Towards Understanding 2D Documents. Katti et al. EMNLP 2018\n\n[B] Entity Relation Extraction as Dependency Parsing in Visually Rich Documents. Zhang et al. EMNLP 2021.\n\n\n\n>**Q3:** Missing some details in experimental settings.\n\n**A3:** Thanks for the nice suggestion.  As mentioned in Section 4.1.3, we follow the experimental setup in LiLT  [C] for a fair comparison.  We  will further explain details of experimental settings in our paper.\n\n[C] LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding. Wang et al. ACL 2022.\n\n\n\n> **Q4:** As one of the most important hyper-parameter, It is unclear how many iterations were used in the experiments. It might not be unexpected to see that the best performance was achieved within a small number of iterations, as a large number of iterations can diminish representational power of spatial information. Also, how are the global tokens T initialized?\n\n**A4:** Thanks for your question. We extract a subset of the training dataset as a validation dataset to perform the hyper-parameter tuning. When tuning the iteration rounds, we keep other hyper-parameters constant. The optimal number of iteration rounds is set to 3 on FUNSD and 4 on XFUND respectively. We provide detailed results under different iteration rounds in the following tables. As you correctly understand, as the number of iteration rounds increases to a certain point, the performance of the model decreases.  The global tokens T are initialized zero. We will declare these more clearly in our paper.\n\ni. Results ( F1 accuracy) of GOSE under different iteration rounds over FUNSD dataset.\n\n\n\n| Iteration Rounds |   FUNSD    |\n| :--------------: | :--------: |\n|        1         |   0.7274   |\n|        2         |   0.7628   |\n|        3         | **0.7697** |\n|        4         |   0.7527   |\n|        5         |   0.7511   |\n|        6         |   0.7466   |\n|        7         |   0.7426   |\n\nii. Results ( F1 accuracy) of GOSE under different iteration rounds over XFUND dataset.\n\n| Iteration Rounds |     ZH     |     JA     |     ES     |     FR     |     IT     |     DE     |     PT     |    Avg.    |\n| :--------------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n|        1         |   0.8553   |   0.7998   |   0.8345   |   0.8116   |   0.8235   |   0.7747   |   0.7107   |   0.8016   |\n|        2         |   0.8612   |   0.8014   |   0.8460   |   0.8371   |   0.8311   |   0.7816   |   0.7156   |   0.8103   |\n|        3         |   0.8650   |   0.8045   |   0.8499   |   0.8408   |   0.8345   |   0.7930   |   0.7224   |   0.8151   |\n|        4         | **0.8752** | **0.8096** | **0.8595** | **0.8646** | **0.8415** | **0.8023** | **0.7384** | **0.8273** |\n|        5         |   0.8659   |   0.8081   |   0.8589   |   0.8554   |   0.8379   |   0.8017   |   0.7378   |   0.8236   |\n|        6         |   0.8662   |   0.8056   |   0.8504   |   0.8488   |   0.8319   |   0.7929   |   0.7296   |   0.8189   |\n|        7         |   0.8653   |   0.8024   |   0.8469   |   0.8425   |   0.8281   |   0.7874   |   0.7236   |   0.8137   |\n\n\n\n> **Q5:** I wonder how the method would deal with entities that spread across multiple pages in a document, and how to adjust the distance and other functions for these cases.\n\n**A5:**  Thanks for raising a concern about the multi-page document scenario. In response to your concern, we propose a feasible solution to tackle the multi-page VRE challenge. Our strategy involves treating this complex task as a sequence generation problem. First, we can use our GOSE to capture relevant global structure information from each page conditioned to the query entity separately. Then relevant global structure information from each page is concatenated to provide a decoder with a holistic structure representation of the multi-page document at the time of generating target entities.  Recently,  we notice that some works have begun to propose multi-page DocVQA benchmarks [D]. In the future, if there is a publicly available multi-page VRE benchmark, we will attempt to employ our GOSE for this task.\n\n[D] Hierarchical multimodal transformers for Multi-Page DocVQA. Tito et el.  Pattern Recognition 2023.\n\n\n\n> **Q6:** It is great to see that the proposed GOSE method is plug-and-play. In experiments, GOSE was used with LayoutXLM or LiLT. It would be useful to mention how GOSE can be plugged with other model in the Method section.\n\n**A6:**  Thanks for the suggestion.  The reason for plug-and-play is that GOSE only needs to receive entity representations as inputs. The entity representations can be derived from any pre-trained model. After receiving entity representations, our GOSE automatically mines the global structural information to enhance the entity representation. As illustrated in Section 3.1,  We can use other models as a visually-rich document encoder to encode the  scanned image of a document to obtain initial semantic entity representations in the same way that LiLT and LayoutXLM are used. We will adjust the presentation to make it easier to understand.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Good results, rich details, and reasonable methods", "reasons_to_reject": "Suspicious improvement through iterative learning steps; \nrelatively small datasets;\nmissing some details in experimental settings.", "questions_to_authors": "This paper presents very impressive results, with huge improvement over baseline methods. However, from the ablation study, the most useful part is the iterative learning. The performance of the method surprisingly dropped from F1=0.7697 to 0.1599 when removing the iteration steps. This raises concern on over-fitting, as multiple rounds of iterations can result in over-parameterization and overfitting, particularly when considering that the datasets used in the studies are relatively small and there seems no validation datasets for hyperparameter tuning. It would be useful to extract a subset of a training dataset as a validation dataset from to eliminate this concern. \n\nSome minor comments are: \n1. As one of the most important hyper-parameter, It is unclear how many iterations were used in the experiments. It might not be unexpected to see that the best performance was achieved within a small number of iterations, as a large number of iterations can diminish representational power of spatial information. Also, how are the global tokens T initialized? \n\n2. I wonder how the method would deal with entities that spread across multiple pages in a document, and how to adjust the distance and other functions for these cases.\n\n3. It is great to see that the proposed GOSE method is plug-and-play. In experiments, GOSE was used with LayoutXLM or LiLT. It would be useful to mention how GOSE can be plugged with other model in the Method section.", "paper_topic_and_main_contributions": "This paper studied Visual Relation Extraction (VRE) which aims to extract relationships between entities from visually-rich documents. \nThe authors proposed a GlObal Structure knowledge-guided relation Extraction (GOSE) framework which makes use of both local spatial layout and global structure information to extract relations, and use an iterative learning method to learning the most informative features. The experimental results show good performance of the proposed method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "\"). VRE remains\" --> \"), VRE remains\" in P8"}, "extra_text": {}}, {"review_id": "jA1GGmhH0P", "reviewer": "Reviewer_5V4t", "summary": "", "questions": "Q1: I notice that the extracted relations are tuples instead of triplets since the head entity itself actually represents a certain kind of relation. Is there any work in VRE that extracts relational triplets from the scanned images of documents? Are there any human-annotated datasets?  \nQ2: Perhaps the sharp performance drop (without Iteration) in Sec. 4.5 can originate from the significant decline of model parameters? ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 5V4t\n\nWe appreciate the reviewer for the valuable comments. Our response to the reviewer’s questions is as follows.\n\n> **Q1:** The paper does not discuss the performance of LLMs that are able to process multi-modal information (scanned image of a document coupled with a prompt) and then provide useful information, which renders it unclear to further estimate the paper’s contribution to the VRE task. \n\n**A1:** Thanks for raising this important consideration. Recent studies [E, F] have conducted comprehensive evaluations of publicly available large multimodal models. These  investigations reveal that large multimodal models primarily rely on semantic understanding for complex reasoning and exhibit inferior perception of fine-grained structure information without semantic. For example, for the VQA task,  InstructBLIP [G] achieves an impressive top-1 accuracy of 60.20% on the OCR-VQA benchmark. However, for the key information extraction (KIE) task, which consists of semantic entity recognition (SER) and visual relationship extraction (VRE), large multimodal models demonstrate notably poor performance. On the FUNSD benchmark which is used in our paper, InstructBLIP  achieves 1.03% F1 score,  MiniGPT-4 [H]  achieves 1.27% F1 score, and mPLUG-Owl [I] achieves 0.41% F1 score.  Furthermore, our paper introduces and empirically validates that the global structural knowledge is useful for visually-rich document information extraction. We firmly believe that our insights have the potential to guide the development of large model technology within the visually-rich document domain. As you nicely suggested, we will include this discussion in our paper.\n\n[E] On the Hidden Mystery of OCR in Large Multimodal Models. Liu et al. Arxiv 2023.\n\n[F] LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models. Xu et al. Arxiv 2023.\n\n[G] InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. Dai et al. Arxiv 2023.\n\n[H] MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. Zhu et al. Arxiv 2023.\n\n[I] mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality. Ye et al. Arxiv 2023.\n\n\n\n\n\n> **Q2:** The paper argues that the main contribution of GOSE is that dependencies between entity pairs are accurately captured and modeled, but it seems that more experiments are required to provide the evidence. It remains unclear for me about how the dependencies can work and boost the performance. I think, for example, it will be a bonus if authors visualize the attention weights over the spatial information (e.g., spatial prefix). \n\n**A2:**  Thanks for your important concern. We calculate the attention scores over the spatial information for the document in Figure 5(a), i.e., the spatial-prefix guided attention weights using Equation (6). We provide the results of attention weights over the spatial prefix in the following table. As shown in the following table,  the entity pair (TO, Sam Zolot)  pays more attention towards the entity pair (FROM:, D.J.Landro) and  (DATE:, 2-DEC-97).  This phenomenon indicates that the injection of additional spatial layout information of entity pairs can guide the attention mechanism attending to entity pairs with similar spatial structure, thereby enhancing the capacity of the model to discern precise dependencies between entity pairs. We will visualize these attention weights in the next version. \n\ni.  The spatial-prefix guided attention weights of the entity pair (TO, Sam Zolot) over other spatial prefix of entity pairs.\n\n|                 | (TO, Sam Zolot) | (TO, D.J.Landro) | (TO, 2-DEC-97) | (FROM:, Sam Zolot) | (FROM:, D.J.Landro) | (FROM:, 2-DEC-97) | (DATE:, Sam Zolot) | (DATE:, D.J.Landro) | (DATE:, 2-DEC-97) |\n| :-------------: | :-------------: | :--------------: | :------------: | ------------------ | ------------------- | ----------------- | ------------------ | ------------------- | ----------------- |\n| (TO, Sam Zolot) |   **0.0189**    |      0.0338      |     0.0318     | 0.0064             | **0.0097**          | 0.0340            | 0.0054             | 0.0064              | **0.0103**        |\n\n\n\n\n\n>**Q3:** The reproducibility of the paper may be a concern, as the authors did not mention whether the code or data will be publicly available for others to replicate or build upon this work.\n\n**A3:** Thanks for your concern. We have provided the data and code as part of the Supplementary Materials.  Additionally, we will include a statement within our paper affirming our commitment to making all data and code publicly accessible. We greatly value the opportunity to enhance the transparency and reproducibility of our research through these actions.\n\n\n\n\n\n\n\n> **Q4:** I notice that the extracted relations are tuples instead of triplets since the head entity itself actually represents a certain kind of relation. Is there any work in VRE that extracts relational triplets from the scanned images of documents? Are there any human-annotated datasets?\n\n**A4:** Thanks for your question. We appreciate your proper understanding of the visually-rich document relation extraction task. This task entails determining the presence of relationships between two semantic entities and subsequently extracting tuples comprising related entity pairs. Given the inherent definition of the VRE task, prevailing methodologies have predominantly directed their efforts towards effectively capturing and modeling the multimodal information embedded in scanned document images to achieve precise tuple extraction [J]. Due to fine-grained annotation requirements and the poor performances of LLMs in the visually-rich document domain [E, F], current high-quality datasets require manual annotation, which encompasses the FUNSD and XFUND datasets used in our paper.  Therefore,  Using machines to automatically annotate visually-rich documents may be a challenging and interesting topic. We appreciate that the reviewer points out a meaningful and interesting direction for future research.  We leave this for future work.\n\n[J] DOCUMENT AI: BENCHMARKS, MODELS AND APPLICATIONS. Cui et al. Arxiv 2021.\n\n\n\n\n\n> **Q5:** Perhaps the sharp performance drop (without Iteration) in Sec. 4.5 can originate from the significant decline of model parameters?\n\n**A5:**  Thanks for your question.  The sharp performance drop (without Iteration) in Sec. 4.5 is mainly due to the fact that we only use the mined global structural information to make predictions without fusing entity representations (using Equation (9)). Since the initial entity representation obtained from the pretrained model is not strong, the mined structural information without iterative optimization is noisy.  The iterative learning process can refine the mined global structural information and enhance the richness of entity representations. This concerted effort leads to a notable enhancement in the performance of model. \n\nTo illustrate this point more clearly, We extract a subset of the training dataset as a validation dataset to perform the hyper-parameter tuning. When tuning the iteration rounds, we keep other hyper-parameters constant. We add the experiment of different information of our GOSE (LiLT) under multiple iterations over FUNSD dataset in the following table.\n\n|                         Information                          |                    Rounds 0                     | Rounds 1 | Rounds 2 |  Rounds 3  | Rounds 4 | Rounds 5 | Rounds 6 | Rounds 7 |\n| :----------------------------------------------------------: | :---------------------------------------------: | :------: | :------: | :--------: | :------: | :------: | :------: | :------: |\n|                Global structure features only                |                     0.1599                      |  0.5505  |  0.7135  | **0.7238** |  0.7071  |  0.6797  |  0.6150  |  0.6078  |\n| Entity representations  incorporating global structural information | 0.6276*  |  0.7274  |  0.7628  | **0.7697** |  0.7527  |  0.7511  |  0.7466  |  0.7426  |\n\nAs shown in above table, the first line demonstrates the performance of only using mined global structural information, the second line demonstrates the performance of using entity representations incorporating global structural information. We can observe that:\n\n(1) When iterative learning process begins, both global structural features and entity representations progressively enhance and mutually reinforce each other. For example, by executing one iteration round (rounds = 1), the performance of only global structure features improves from 0.1599 (absent iteration) to 0.5505. Similarly, the performance of entity representations advances from 0.6276 (without global structural information, rounds=0) to 0.7274 (incorporating global structural information, rounds=1). \n\n(2) As the number of iteration rounds increases to a certain point, the performance of the model decreases. This phenomenon can be attributed to too many iteration rounds can cause mined global structural information to become over-smoothing thus affecting the final performance of our model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experimental results are sufficient to demonstrate both the effectiveness and efficiency of GOSE.\nThe paper is well-written and clearly presents the motivation and ideas of GOSE.", "reasons_to_reject": "The paper does not discuss the performance of LLMs that are able to process multi-modal information (scanned image of a document coupled with a prompt) and then provide useful information, which renders it unclear to further estimate the paper’s contribution to the VRE task.\nThe paper argues that the main contribution of GOSE is that dependencies between entity pairs are accurately captured and modeled, but it seems that more experiments are required to provide the evidence. It remains unclear for me about how the dependencies can work and boost the performance. I think, for example, it will be a bonus if authors visualize the attention weights over the spatial information (e.g., spatial prefix).\nThe reproducibility of the paper may be a concern, as the authors did not mention whether the code or data will be publicly available for others to replicate or build upon this work.", "questions_to_authors": "Q1: I notice that the extracted relations are tuples instead of triplets since the head entity itself actually represents a certain kind of relation. Is there any work in VRE that extracts relational triplets from the scanned images of documents? Are there any human-annotated datasets?  \nQ2: Perhaps the sharp performance drop (without Iteration) in Sec. 4.5 can originate from the significant decline of model parameters? ", "paper_topic_and_main_contributions": "This paper explores the long-neglected global spatial/structure information in the VRE task by tackling two technical challenges, proposes GOSE to incorporate the information, and demonstrates the advantages of GOSE such as data efficiency, cross-lingual transfer, and SOTA performance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper targets a practical setting of IE on visually riched documents. Instead of pure text, the inputs can be pdf or some other visual based file. The paper is well motivated and organized, and the experiments show the effectiveness.After serious discussion and consideration, the reviewers' main concerns are about the huge improvements by iterative component, small datasets, and comparison with LLMs. The authors actively provide more evidence which we think has mostly solved the concerns. Nevertheless, some reviewers are not excited about the idea, suggesting the effectiveness yet not much insights.

Overall, we appreciate both the efforts of reviewers and authors. This work is ready to publish, and the only concern is the novelty. We hope the authors can continue to improve the paper according to the comments and your response.
Decision: Accept-Findings