Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction
Authors: Chong Zhang
Ya Guo
Yi Tu
Huan Chen
Jinyang Tang
Huijia Zhu
Qi Zhang
Tao Gui
Venue: EMNLP 2023 Main (2023)
Keywords: visually-rich document understanding
information extraction
named entity recognition
TL;DR: Introducing Token Path Prediction, a simple method to address reading order issues in information extraction from visually-rich documents, achieving SOTA performance in various VrD tasks including NER, entity linking and reading order prediction.

Abstract: Recent advances in multimodal pre-trained models have significantly improved information extraction from visually-rich documents (VrDs), in which named entity recognition (NER) is treated as a sequence-labeling task of predicting the BIO entity tags for tokens, following the typical setting of NLP. However, BIO-tagging scheme relies on the correct order of model inputs, which is not guaranteed in real-world NER on scanned VrDs where text are recognized and arranged by OCR systems. Such reading order issue hinders the accurate marking of entities by BIO-tagging scheme, making it impossible for sequence-labeling methods to predict correct named entities. To address the reading order issue, we introduce Token Path Prediction (TPP), a simple prediction head to predict entity mentions as token sequences within documents. Alternative to token classification, TPP models the document layout as a complete directed graph of tokens, and predicts token paths within the graph as entities. For better evaluation of VrD-NER systems, we also propose two revised benchmark datasets of NER on scanned documents which can reflect real-world scenarios. Experiment results demonstrate the effectiveness of our method, and suggest its potential to be a universal solution to various information extraction tasks on documents.

Reading Order Matters:
Information Extraction from Visually-rich Documents by Token Path Prediction
Chong Zhang Ya Guo Yi Tu?, Huan Chen², Jinyang Tang?,
Huijia Zhu?, Qi Zhang Tao Gui
Sccoool of Computer Science, Fudan University, Shanghai, China
2 "Ant Info Security Lab, Ant Group Inc., Hangzhou, China
3 IInstitute of Modern Languages and Linguistics, Fudan University, Shanghai, China (chongzhang20, qz, tgui fefudan edu. cn (guoya. gy, qianyi. ty, chenhuan. chenn, jinyang, tjy, hui jia. ..laa..zji zhj)eaatgrouu.ccom
Abstract
OF STORES SUPPLIED
ACCOUNT
Recent advances in multimodal pre-trained models have significantly improved informa- tion extraction from visually-rich documents (VrDs), which named entity recognition (NER) is treated sequence-labeling task of predicting the BIO entity tags for tokens, following the typical setting of NLP. However, BIO-tagging scheme relies on the correct order model inputs, which not guaranteed in real-world NER on scanned VrDs where text are recognized and arranged by OCR systems. Such reading order issue hinders accurate marking of entities by BIO-tagging scheme, making it impossible for sequence- labeling methods to predict correct named entities. To address the reading order issue, we introduce Token Path Prediction (TPP), a simple prediction head to predict entity mentions as token sequences within documents. Alternative to token classification, TPP models the document layout complete directed graph of tokens, and predicts token paths within the graph as entities. For better evaluation of VID-NER systems, also propose two revised benchmark datasets of NER on scanned documents which can reflect real-world sce- narios. Experiment results demonstrate the effectiveness of our method, and suggest its potential to be universal solution various information extraction tasks on documents.
Ided Toxt (Ondered)
NAME
ACCOUNT
STORES
SUPTRRD
BB
X
Actual (Bloddarad)
STORES
XAME
ACCOUNT
(a) Entity annotation by BIO-tags. With disordered inputs, BIO-tags fail to assign a proper label for each word to mark entities clearly.
OF STORES SUPPLIED
NAME OF ACCOUNT
Ideal Text (Undered)
NAMP
ACCOUNT
OF STORES SUIPPT
(Rborders))
(b) Entity annotation as word sequences within the document. The entity annotations are not affected by model input order.
Figure document image with its layout and entity annotations. (a) Sequence-labeling methods are not suitable for VrD-NER as model inputs are disordered in real situation. (b) We address the reading order issue treating VrD-NER as predicting word sequences within documents.
and comprehension of scanned VrDs are necessary in various scenarios, including legal, business, and financial fields (Stanislawek al., 2021: Huang et al., 2019; Stray and Svetlichnaya, 2020). Re- cently, several transformer-basee multimodal pre- trained models (Garncarek et al., 2020; Xu et al., 2021 aa; Li et al., 2021a: Hong et al., 2022: Huang et al., 2022: Tu et al., 2023) have been proposed. Known as document transformers, these models can encode text, layout, and image inputs into unified feature representation, typically by marking each input text token with its xy-coordinate on the document layout using an additional 2D positional embedding. Thus, document transformers can adapt to a wide range VrD tasks, such as Named
1 Introduction
Visually-rich documents (VrDs), including forms, receipts and contracts, are essential tools for gathering, carrying, and displaying information in the digital era. The ability to understand and extract information from VrDs is critical for real- world applications. In particular, the recognition
"This work done when the author intern Ant Group.
'Equal contribution. Coorrepoondigg author.

---

Entity Recognition (NER) and Entity Linking (EL) (Jaume al., 2019; Park et al., 2019).
FRSH SHEHH BAY 125
Suntital
30,000
230.000
TOTAL (Oty 1. (00000000000000 0000
TOTAL CHANGE
However, in the practical application of infor- mation extraction (IE) from scanned VrDs, the reading order issue is known as a pervasive problem that lead to suboptimal performance of current methods. This problem is particularly typical in VrD-NER, task that aims to identify word sequences in document as entities of predefined semantic types, such as headers, names and addresses. Following the classic settings of NLP, current document transformers typically treat this task as sequence-labeling problem, tagging each text token using the B10-tagging scheme (Ramshaw and Marcus, 1999) and predicting the entity tag for tokens through token classification head. These sequence-labeling sabsling-based methods assumes that each entity mention is a continuous and front-to-back word sequence within inputs, which is always valid in plain texts. However, for scanned VrDs in real-world, where text and layout annotations are recognized and arranged by OCR systems, typically in a top-to-down and left-to-right order, the assumption may fail and the reading order issue arises, rendering the incorrect order of model inputs for document transformers, and making these sequence-labeling methods inapplicable. For instance, depieted in Figure/ 1, the document contains two entity mentions, namely "NAME OF aCcOUNT" and "# OF STORES SUPPLIED". However, due to their layout positions, the OCR system would recognize the contents as three segments and arrange them in the following order:(1) OF STORES"; NAME OF ACCOUNT"; (3)SSUPPIEDD" Such disordered input leads to significant confusion in the BIO- tagging scheme, making the models unable to assign proper label for each word to mark the entity mentions clearly. Unfortunately, the reading order issue is particularly severe in documents with complex layouts, such as tables, multi-column contents, and unaligned contents within the same row or column, which are quite common in scanned VrDs. Therefore, we believe that the sequence- labeling paradigm is not practical approach address NER on scanned VrDs in real-world scenarios.
BOX
Figure 2: The reading order issue of scanned documents in real-world scenarios. Left:: The entity "TOTAL 230,000" disordered when reading from top to down, The entity "CASH CHANGE" lies in multiple rows and is separated to two segments. Right: The entity "TOTAL 30,000" is interrupted by "(Qty 1.00y" when reading from left to right. All these situations result in disordered model inputs and affect performance of sequence- labeling based VrD-NER methods. More real-world examples of disordered layouts are displayed in Figure in Appendix.
and can be adapted to various VrD-IE and VrD understanding (VrDU) tasks, such as NER, EL, and reading order prediction (ROP). Specifically, when using TPP for VrD-NER, we model the token imputs as complete directed graph of tokens, and each entity as token path, which is group of directed edges within the graph. We adopt a grid label for each entity type to represent the token paths as n * n binary values of whether two tokens are linked or not, where n is the number of text tokens. Model learns to predict the grid labels by binary classification in training, and search for token paths from positively token pairs in inference. Overall, TPP provides viable solution for VrD-NER by presenting suitable label form, modeling VrD-NER as predicting token paths within graph, and proposes a straightforward method for prediction. This method does not require any prior reading order and is therefore unaffected by the reading order issue. For adaptation to other VrD tasks, TPP is applied to VrD- by predicting a global path of all tokens that denotes the predicted reading order. TPP is also capable of modeling the entity linking relations by marking linked token pairs within the grid label, making it suitable for direct adaptation to the VrD-EL. task. Our work is related to discontinuous NER in NLP as the tasks share similar form, yet current discontinuous NER methods cannot be directly applied to address the reading order issue since they also require a proper reading order of contents.
To address the reading order issue, introduce Token Path Prediction (TPP), a simple yet strong prediction head for VrD-IE tasks. TPP is compat- ible with commonly used document transformers,
For better evaluation of our proposed method, we also propose two revised datasets for VrD-NER. In current benchmarks of NER on scanned VrDs, such as FUNSD (Jaume et al., 2019) and CORD

---

(Park al., 2019), the reading order is manually- corrected, thus failing to reflect the reading order issue in real-world scenarios. To address this limitation, we reannotate the layouts and entity mentions of the above datasets to obtain two revised datasets, FUNSD-r and CORD-r, which accurately reflect the real-world situations and make it possible to evaluate the VrD-NER methods in disordered scenarios. We conduct extensive experiments by integrating the TPP head with different document transformer backbones and report quantitative results on multiple VrD tasks. For VrD-NER, experiments on the FUNSD-r and CORD-r datasets demonstrate the effectiveness of TPP, both as an independent VrD-NER model, and as pre-processing mechanism to reorder inputs for eequence-labeling models. Also, TPP achieves SOTA performance on benchmarks for VrD-EL and VrD-ROP, highlighting its potential as universal solution for information extraction tasks on VrDs. The main contribution of our work are listed as follows:
together with its layout. In sequence-labeling based NER with document transformers, BIO-tagging scheme assigns BIO-tag for each text token to mark entities: B-ENT/I indicates the token to be the beginning/inside token of an entity with type ENT, and O indicates the word does not belong to any entity. In this case, the type and boundary of each entity is clearly marked in the tags. The BIO-tags are treated the classification label of each token and predicted by a token classification head of the document transformer. As illustrated in introduction, eqquence-labeling methods are typical for NER in NLP and is adopted by current VrD-NER methods, but is not suitable for VrD- NER in real-world scenarios due to the reading order issue.
Reading-order-awarr Methods Several studies have addressed the reading order issue on VrDs in two directions: (1) Task-specific models that directly predict the reading order, such Lay- outReader, which uses a seuueenceto-sequenee approach (Wang et al., 202 However, this method is limited to the task at hand and cannot be directly applied to VrD-IE tasks. (2) Pre-trained models that learn from supervised signals during pre-training to improve their awareness of reading order. For instance, ERNIE-Layout includes pre-training objective of reading order prediction (Peng et al., 2022); XYLayouLM enhances the generalization of reading order pre-training by generating various proper reading orders using an augmented XY Cut algorithm (Gu et al., 2022). However, these methods require massive amount of labeled data and computational resources during pre-training. Comparing with the above methods. our work is applicable to multiple VrD-IE tasks of VrDs, and can be integrated with various document transformers, without the need for additional supervised data or computational costs.
1. We identify that sqguence-abbeing-benr
VrD- NER methods is unsuitable for real-world sce- narios due the reading order issue, which is not adequately reflected by current benchmarks.
2. We introduce Token Path Prediction, a simple yet strong approach to address the reading order issue in information extraction on VrDs.
3. Our proposed method outperforms SOTA meth- ods in various VrD tasks, including VID-NER, VrD-EL, and VrD-ROP. We also propose two revised VrD-NER benchmarks reflecting real- world scenarios of NER on scanned VrDs.
2 Related Work
Sequence-labeling based NER with Document Transformers Recent advances of pre-trained techniques in NLP (Devlin al., 2019; Zhang al., 2019) and CV (Dosovitskiy et al., 2020; Li et al., 2022c) have inspired the design of pre- trained representation models in document AI, in which document transformers (Xu et al., 2020, 2021a; Huang et al., 2022; Li et al., 2021a,c; Hong et al., 2022; Wang et al., 2022; Tu et al., 2023) are proposed to act layout-aware representation model of document in various VrD tasks. By the term document transformers, we refer to transformer encoders that take vision (optional), text and layout input of a document as a token sequence, in which each text token is embedded
Discontinuous NER The form of discontinuous NER is similar to VrD-NER, as it involves identi- fying discontinuous token sequences from text as named entities. Current methods of discontinuous NER can be broadly categorized into four groups: (1) Sequence-labeling methods with refined BIO- tagging scheme (Tang al., 2015; Dirkson et al., 2021), (2) 2D grid prediction methods (Wang al., 2021a; Li et al., 2022b: Liu et al., 2022), (3) Sequence-to-seqqunec methods (Li al., 2021b; He and Tang, 2022), and (4) Transition- based methods (Dai et al., 2020). These methods all rely

---

Grid Labds
LOP STORTS
OF ACCOUNT
SUPPLIED OF
M2
Mi STORES
NAME
ACCOUNT
SUPPLIED
Document Transformer
with Tokae Path Prediction
Prodictions
Class Imbalanced for Binary Classifiation
Quection:
M2 W2 My
Prediction: OF STORUS SUPPLIED
Token Pathe W. 20 My 20
X
107
Prediction: ACCOUNT
Figure 3: An overview of the training procedure of Token Path Prediction for VID-NER. A TPP head for each entity type predicts whether an input tokens links to an another. The predict result viewed as an adjacent matrix which token paths decoded entity mentions. The overall model is optimized by the class-inbalance loss.
on a proper reading order to predict entities from front to back, thus cannot be directly applied to address the reading order issue in VrD-NER.
a complete directed graph with the Np tokens [wili=1,...Ng D as vertices. This graph consists of n² directed edges pointing from each token to each token. For each entity sy {ej, (wa..... ). the word sequence can be represented by path (w,, Wjas.... Wha- Wje) within the graph, referred as token path.
3 Methodology
3.1 VrD-NER
The definition of NER on visually-rich documents with layouts is formalized follows. A visually- rich document with Np words is represented as D [w,, b.)..=,,...NNo. where w; denotes the i-th word document and b; = (xf+ v x! ut) denotes the position of wi in the document layout. The coordinates (xf, v2) and (x), w²) correspond to the bottom-left and top-right vertex of w, 's bounding box, respectively. The objective of ViD-NER is predict all the entity mentions (81)...,8,) within document D, given the pre- defined entity types & {ei)==t....Ne- Here, the j-th entity in is represented as 8j {e,, (W w. w)){. where ej E & is the entity type and (w,,,.... w),) ) a word sequence, where the words are two-by-two different but not necessar- ily adjacent. It is important to note that sequence- labeling based methods assign BIO-label to each word and predict adjacent words (w,, Wy+1;-..) as entities, which is not suitable for real-world VrD- NER where the reading order issue exists.
TPP predicts the token paths in a given graph by learning grid labels. For each entity type, the token paths of entities are marked by a n n grid label of binary values, where is the number of text tokens. In specific, for each edge in every token path, the pair of the beginning and ending tokens is labeled as in the grid label, while others are labeled as For example Figure 3, the token pair "(NAME, OF)" and "(OF, ACCOUNT)" are marked in the grid label. In this way, entity annotations can be represented as Ng grids of n * n binary values.
The learning of grid labels is then treated Ng binary classification tasks which can be implemented using any classification model. In TPP, we utilize document transformers to represent document inputs as token feature sequences, and employ Global Pointer (Su et al., 2022) as the classification model to predict the grid labels by binary classification. Following (Su et al., 2022), the weights are optimized by class- imbalance loss to overcome the class-imbalance problem, there of n² labels in each grid. During evaluation, we collect the Ne predicted grids, filtering the token pairs predicted to be positive. If there are multiple token pairs with same beginning token, we only
n² samples,
most n positive labels out
3.2 Token Path Prediction for VrD-NER
Token Path Prediction, we model VrD-NER as predicting paths within a graph of tokens. Specifically, for a given document D. we construct

---

AGE:
Under 25 2.2
Reading
including selecting adequate data resources and the annotating process.
25-34
-23.0
Under 32 3.0
4.1
Motivation
ANGE
FUNSD (Jaume et al., 2019) and CORD (Park et al., 2019) are the most popular benchmarks of NER on scanned VrDs. However, these benchmarks are biased towards real-world scenarios. In FUNSD and CORD, segment layout annotations are aligned with labeled entities. The scope of each entity the document layout is marked by bounding box that forms the segment annotation together with each entity word. We argue that there are two problems their annotations that render them unsuitable for evaluating current methods. First, these benchmarks do not reflect the reading order issue of NER on scanned VrDs, as each entity corresponds to a continuous span in the model input. In these benchmarks, each segment models continuous semantic unit where words are correctly ordered, and each entity corresponds exactly to one segment. Consequently, each entity corresponds to a continuous and front-to-back token span the model input. However, in real-world scenarios, entity mentions may span across segments, and segments may be disordered, necessitating the consideration of reading order issues. For instance, as shown in Figure 5, the entity "Sample Requisition FForm 02:02:06]" is located in a chart cell spanning multiple rows; while the entity is recognized as two segments by the OCR system since OCR- annotated segments are confined to lie in a single row. Second, the segment layout annotations in current benchmarks vary in granularity, which is inconsistent with real-world situations. The scope of segment in these benchmarks ranges from single word multi-row paragraph, whereas OCR-annotated segments always correspond words within single row.
32
3.3
Under
Under
25
25
25
25
34
34
Extly Litking Atnolation
Beading Order Atnoution
The grid label design of TPP for VrD-EL and VrD-ROP. For VrD-EL., Under 25/25-34 is linking 3.2/3.0 so each token between the two entities are linked, For VrD-ROP, the reading order is represented as a global path including all tokens starting from an auxiliary beginning token 
, in which the linked token pairs are marked.
Figure 4:
keep the pair with highest confidence score. After that, we perform a greedy search to predict token paths as entities. Figure 3 displays the overall procedure of TPP for VID-NER.
In general, TPP is a simple and easy-to- implement solution to address the reading order issue in real-world VrD-NER.
3.3 Token Path Prediction for Other Tasks
We have explored the capability of TPP address various VrD tasks, including VrD-EL and VrD- ROP. As depicted in Figure 4, these tasks are addressed by devising task-specific grid labels for TPP training. For the prediction of VrD-EL, we gather all token pairs between every two entities and calculate the mean logit score. Two entities are predicted to be linked if the mean score is greater than O. In VID-ROP, we perform beam search on the logit scores, starting from the auxiliary beginning token to predict global path linking all tokens. The feasibility of TPP on these tasks highlights its potential as a universal solution for VrD tasks.
Therefore, we argue that a new benchmark should be developed with segment layout anno- tations aligned with real-world situations and entity mentions labeled on words.
4 Revised Datasets for Scanned VrD-NER
4.2 Dataset Construction
In this section, we introduce FUNSD-r and CORD- r, the revised VrD-NER datasets to reflect the real- world scenarios of NER on scanned VrDs. We first point out the existing problems of popular benchmarks for NER on scanned VrDs, which indi- cates the necessity of us to build new benchmarks. We then describe the construction of new datasets,
As illustrated above, current benchmarks cannot reflect real-world scenarios and adequate bench- marks are desired. That motivates us to develop new NER benchmarks of scanned VrDs with real- world layout annotations.
We achieve this by reannotating existing bench- marks and select FUNSD (Jaume et al., 2019)

---

datasets proposed in our work. The performance of methods are measured by entity-level Fl on these datasets. Noticing that previous works on FUNSD and CORD have used word-level Fl as the primary evaluation metric, we argue that this may not be entirely appropriate, as word-level FI only evaluates the accuracy of individual words, yet entity-level Fl assesses the entire entity, including its boundary and type. Compared with word-level F1, entity-level F1 is more suitable for evaluation on NER systems in pratical applications. For VrD- EL, the experiments are conducted on FUNSD and the methods are measured by FI. For VrD-ROP, the experiments are conducted on ReadingBank. The methods are measured by Average Page-level BLEU (BLEU for short) and Average Relative Distance (ARD) (Wang et al., 2021b)
Pebacco
Prodeetoon LLLERR
Fl Prodection VAROER
Spectel Deuntrrenets
Sccoy t0 Hoy
7ip eslution
Spcey 30 Hot tokadao rnsesson Aed mo 20d-00 dalieeny
2.0 190 601-00 on 13:11 no sanee)
Saliveny auraying ofrectence 750 to an bott abould be 6.1 mnlitg.
butt
Figure 5: A comparison of the original (left) and revised (right) layout annotation. The revised layout annotation reflects real-world situations, where (1) character-level position boxes are annotated rather than word-level boxes, (2) every segment lies in single row, (3) epatially-ajjacee words are recognized into segment without considering their semantic relation, and (4) missing annotation exists.
and CORD (Park al., 2019) for the following reasons. First, FUNSD and CORD are the most commonly used benchmarks on VrD-NER as they contains high-quality document images that closely resemble real-world scenarios and clearly reflect the reading order issue. Second, the annotations on FUNSD and CORD are heavily biased due to the spurious correlation between its layout and entity annotations. In these benchmarks, each entity exactly corresponds
5.2 Baselines and Implementation Details
For VrD-NER, compare the proposed TPP with requence-labeling methods that adopts document transformers integrated with token classification heads. As these methods suffer from the reading order issue, we adopt a VrD-ROP model as the pre-processing mechanism to reorder the inputs, mitigating the impact of the reading order issue. We adopt LayoutLMv3-baas (Huang et al., 2022) and LayoutMask (Tu et al., 2023) the backbones integrate with TPP or token classification heads, as they are the SOTA document transformers in base-level that use "vision+text+layout" and "text+layout" modalities, respectively. For VrD- EL and VrD-ROP, we introduce the baseline methods with whom we compare the proposed TPP in Appendix B. The implementation details of experiments illustrated in Appendix C.
segment, which leaks the ground truth entity labels during training by the 2D positional encoding of document transformers, as tokens of the same entity share the same segment 2D position information. Due to the above issues, we reannotate the layouts and entity mentions on the document images of the two datasets. We automatically reannotate the layouts using an OCR system, and manually reannotate the named entities word sequences based on the new layout annotations to build the new datasets. The proposed FUNSD-r and CORD-r datasets consists of 199 and 999 document samples including the image, layout annotation of segments and words, and labeled entities. The detailed annotation pipeline and statistics of these datasets is introduced in Appendix A. We publicly release the two revised datasets GitHub.
5.3 Evaluation on VrD-NER
In this section, we display and discuss the per- formance of different methods on VID-NER in real-world scenario. In all, the effectiveness of TPP is demonstrated both as an independent VrD- NER model, and as a pre-processing mechanism to reorder inputs for sequence-labeling models. By both means, TPP outperforms the baseline models on two benchmarks, across both of the integrated backbones. It is concluded in Table that: (1) TPP is effective model, surpassing the performance of sequence- labeling methods on two benchmarks for real- world VrD-NER. Since sequence-labeling methods are adversely affected by the disordered inputs,
5 Experiments
5.1 Datasets and Evaluation
independent VrD-NER
The experiments in this work involve VrD-NER, VrD-EL, and VrP-ROP. For VrD-NER, the experi- ments are conducted on the FUNSD-r and CORD-r

---

Backbone
Method
Pre.
Cont. (%)
Backbone
Method
Pre.
Cont. (%)
F1
None
95.74
78.77
None
92.10
82.72
Sequence
Sequence
LR
95.53
78.37
LRc
82.10
70.33
Mvy3
Labeling
LayoutLMv3
Labeling
TPP &
97.29
79.72
TPPc
92.43
83.24
TPP
None
80,40
TPP
None
91,85
None
95.74
77.10
None
92.10
81.84
Sequence
Sequence
IR
95.53
77.24
LRc
82.10
68.05
LayoutMask
Labeling
LayoutMask
Labeling
TPPR
97.29
80.70
TPPc
92,43
81.90
None
78.19
TPP
None
89.34
(a) VrD-NER on FUNSD-r
(b) VrD-NER on CORD-r
Table I: The VrD-NER performance of different methods on FUNSD-r and CORD-r, Pre. denotes the pre-processing mechanism used to re-arrange the imput tokens, where LRP/TPP, denotes that input tokens are reordered by LayouReeddrrTPP-forr www.eeeadderPP-f--------O-O VrD-ROP model, LR and TPPR are trained on ReadingBank, and LRc and TPPc are trained on CORD. Cont. denotes the continuous entity rate, higher for better pre- pee-prcesssigg mechanism. The best Fl score and the best continuous entity rates marked in bold. Note that TPP-for- VrD-NER TPP-for-ViD-NTR methods do not leverage any reading order information from ground truth annotations pre-processing mechanism predictions.
their performance is constrained by the ordered degree of datasets. In contrast, TPP is not affected by disordered inputs, as tokens are arranged during its prediction. Experiment results show that TPP outperforms sequence-labeling methods on both benchmarks. Particularly, TPP brings about +9.13 and +7.50 performance gain with LayoutLMv3 and LayoutMask the CORD-r benchmark. It is noticeable that the real superiority of TPP for VrD-NER might be greater than that reflected by the results, as TPP actually learns a more challenging task comparing to sequence-labeling. While sequence-labeling solely learns to predict entity boundaries, TPP additionally learns to predict the token order within each entity, and still achieves better performance. (2) TPP is a desirable pre-processing mechanism reorder inputs for sequence-labeling models, while the other VrD- ROP models are not. According to Table Lay- outReader and TPP-for-VrD-ROO are evaluated as the pre-proee preepeocessnng mechanism by the continuous entity rate of rearranged inputs. As a representative of current VrD-ROP models. LayoutReader does not perform well as pre-processing mechanism, the continuous entity rate of inputs decreases after the arrangement of LayoutReader disordered datasets. In contrast, TPP performs satisfactorily. For reordering FUNSD-r, TPP R brings about a +1.55 gain on continuous entity rate, thereby enhancing the performance of sequence- labeling methods. For reordering CORD-r, the desired reading order of the documents is to read row-by-row, which conflicts with the reading order of ReadingBank documents that are read column-
by-column. Therefore, for fair comparison on CORD-r, we train LayoutReader and TPP on the original CORD to be used as pre pre-processing mechanisms,, denoted as LRc and TPPc- As illustrated in Table 1, TPPc also improves the continuous entity rate and the predict performance of sequence-labeling methods, comparing with the LayoutReader alternative. Contrary to TPP, we find that using LayoutReader for pre-processing would result in even worse performance on the two benchmarks. We attribute this to the fact that LayoutReader primarily focuses on optimizing BLEU scores on the benchmark, where the model is only required to accurately order consecutive tokens. However, according to Table 4 in Appendix A, the average entity length is 16 and words in the datasets, and any disordered token within an entity leads its discontinuity in sequence-labeling. Consequently, LayoutReader may occasionally mispredict long entities with correct input order, due to its seq2seq nature which makes it possible to generate duplicate tokens or missing tokens during decoding.
For better understanding the performance of TPP, we conduct ablation studies to determine the best usage of TPP-for-ViD-NER VrD-NER by configuring the TPP and backbone settings. The results and detailed discussions can be found in Appendix D.
two
5.4 Evaluation Other Tasks
Table 2 and 3 display the performance of VrD- EL and VrD-ROP methods, which highlights the potential of TPP as a universal solution for VrD-IE tasks. For VrD-EL, TPP outperforms MSAU-PA -PA,,

---

Case
Ground Truth
Sequence-labeling
Token Path Prediction
A4
A4
A4
Multi-row Entity
/73/79 10/6/80, Update
5/3 79
//3/79 10/6/80, Update
10/6/80, Update
Multi-column Entity
FOTAL 1E SDLVESS.P0
0000000
TOTAL BORR LVEGB.30
TOTAL V2 1E
0000000 1000000
100,000
120,000
DO VE53 309
Long Entity
SURMISSION DATE
SURMUSSION DATE
SURVISSION DATE
Entity Type Identification
JUNE
SEPT
SEPTII
NOVS
Figure 6: Case study of Token Path Prediction for VrD-NER, where entities of different types distinguished color, and different entities of the same type are distinguished by the shade of color.
Method
Avg. Page-level BL.EU (%)
Order
Method
GNN+MLP (Carbonell 2021)
r=100%
r=50%
r=0%
SPADE (Hwang et ul., 2021)
41.3
LayoutReader
97.65
97.88
98.19
OCR
Doc2Graph (Gemelli et al., 2022)
53.36
TPP
98.18
98.16
LayoutXLM (Xu et al., 2021b)
54.83
LayoutReader
97.72
97.70
17.83
Shfl.
SERA (Zhang et al., 2021)
65.96
TPP
98.16
98.09
98.12
BROS (Hong al., 2022)
71.46
MSAU-PAF (Dang et al., 2021)
75
(a)
The average page-level BLFUU of methods (higher is better).
79.23
ARD
Order
Method
r=100% %
r=50%
r=0%
Table 2: The VrD-EL performance of different methods on FUNSD. The best result is marked bold.
LayoutReader
2.50
2.24
1.75
OCR
TPP
0.29
0.35
0.37
LayourReader
2.48
2.46
72.94
Shfl.
a competitive method, by large margin of 4.23 on Fl scores. The result suggests that the current labeling scheme for VrD-EL provides adequate information for models to learn from. For VrD- ROP, TPP shows several advantages comparing with the SOTA method LayoutReader: (1) TPP is robust to input shuffling. The performance of TPP remains stable when train/cvaluation inputs are shuffled as TPP is unaware of the token input order. However, LayoutReader is sensitive to input order and its performance decreases sharply when evaluation inputs are shuffled. (2) In principle, TPP is more suitable to act as pre- processing reordering mechanism compared with LayoutReader. TPP surpasses LayoutReader by significant margin on ARD among six distinct settings, which is mainly attributed the paradigm differences. Specifically, TPP-for- VrD-ROP guar- antees to predict a permutation of tokens as the reading order, while LayoutReader predicts
TPP
0.37
0.39
0.39
(b) The average relative distance of methods (lower better).
The VrD-ROP performance of different methods on ReadingBank. For the Order setting, OCR denotes that inputs arranged left-to-right and top- to-bottom in evaluation. Shfl. denotes that inputs are shuffled evaluation. is the proportion of shuffled samples in training. The best results are marked in bold.
Table 3:
the reading order by generating a token sequence and carries the risk of missing some tokens the document, which results in a negative impact the ARD metric, and also make LayoutReader unsuitable for
pre-proccssing reordering mechanism. (3) TPP surpasses LayoutReader among five out of six settings on BLEU. TPP is unaware of the token input order, and the performance of it among different settings is influenced only by random aspects. Although

---

LayoutReader has higher performance in the setting where train and evaluation samples are both not shuffled, is attributed to the possible overfitting of the encoder of LayoutReader on global ID signals under this setting, where global ID signals strongly hint the reading order.
performance on VrD-EL and VrD-ROP tasks. We also propose two revised VrD-NER benchmarks to reflect the real situations of NER on scanned VrDs. In future, we plan to further improve our method and verify its effectiveness on more VrD- IE tasks. We hope that our work will inspire future research to identify and tackle the challenges posed by VrD-IE in practical scenarios.
5.5 Case Study
For better understanding the strength and weakness of TPP, we conduct a case study for analyzing the prediction results by TPP in VrD-NER. As discussed, HPP-for-VrD-NNE
Limitations
Our method has the following limitations:
make predictions by modeling token paths. Intuitively, TPP should be good at recognizing entity boundaries in VrDs. To verify this, we visualize the predict result of interested cases by TPP and other methods in Figure 6. According to the visualized results, TPP is good at identifying the entity boundary and makes accurate prediction of multi-row, multi- column and long entities. For instance, in the Multi- row Entity case, TPP accurately identifies the entity "5/3/79 10/6/80, Update" in complex layouts. In the Multi-columm Entity case, TPP identifies the entity "TOTAL 120,000" with the interference of the injected texts within the same row. In the Long Entity case, the long entity as paragraph is accurately and completely extracted, while the sequence-labeling method predicts the paragraph two entities.
1. Relatively- Computation Cost: Token Path Prediction involves the prediction of grid labels, therefore is more computationall-eavily than vanilla token classification, though it is still simple and generalizable method for VrD- IE tasks. Towards training on FUNSD-r, LayourLMy3/LLyouMMesk with Token Path Prediction requires 2.45x/2.75x longer training time and 1.56x/1 1.7x higher peak memory occu- pancy compared to vanilla token classification. The impact is significant when training Token Path Prediction datasets.
large-scale
2. Lack of Evaluation on Real-world Benchmarks: To illustrate the effectiveness of Token Path Prediction on information extraction of real- world scenarios, we rely on adequate down- stream datasets. However, benchmarks is still missing of other IE tasks than NER, and of other languages than English. We appeal for more works to propose VrD-IE and VrDU benchmarks that reflect real-world scenarios, and the demonstration could be more concrete of our method to be capable of various VrD tasks, be compatible with document transformers of different modalities and multiple languages, and be open to future work of layout-aware representation models.
Nevertheless, TPP may occasionally misclassify entities as other types, resulting in suboptimal performance. For example, in the Entity Type Identification case, TPP recognizes the entity boundaries of "SEPT 21" and "NOV 9" but fails to predict their correct entity types. This error can be attributed to the over-reliance of TPP on layout features while neglecting the text signals.
6 Conclusion and Future Work
In this paper, we point out the reading order issue in VrD-NER, which can lead suboptimal perfor- mance of current methods in practical applications. To address this issue, we propose Token Path Prediction, a simple and easy-to-implemennt method which models the task as predicting token paths from a complete directed graph of tokens. TPP can be applied to various VrD-IE tasks with unified architecture. We conduct extensive experiments to verify the effectiveness of TPP on real-world VrD- NER, where it serves both as an independent VrD- NER model and as a pre- prepprocssiin mechanism to reorder model inputs. Also, TPP achieves SOTA
Acknowledgements
The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of Shang-
China 222.6666077761976566222222222
hai Rising-Star Program (23QA1400200) Natural
Science Foundation of Shanghai (23ZR1403500) Program of Shanghai Academic Research Leader
under grant 22XD1401100.

---

References
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4583-4992.
Manuel Carbonell, Pau Riba, Mauricio Villegas, Alicia Foméé, and Josep Lladós. 2021. Named entity recognition and relation extraction with graph neural networks in semi structured documents. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 9622-9627, IEEE
Yuxin He and Buzhou Tang. 2022. Setgner: General named entity recognition as entity set generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3074-3085
Xiang Dai. Sarvnaz Karimi, Ben Hachey, and Cecile Paris. 2020. An effective transition-hased model for discontinuous ner. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5860-5870.
Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang. Daehyun Nam, and Sungrae Park. 2022. Bros: A pre-trained language model focusing text and layout for better key information extraction from documents. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 10767- 10775.
Tuan Anh Nguyen Dang, Duc Thanh Hoang, Quang Bach Tran, Chih-Wei Pan, and Thanh Dat Nguyen. 2021. End-to-end hierarchical relation extraction for generic form understanding. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 5238-522400 5238-5245.15 IEEE.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutlmv3: Pre-training for document AI with unified text and image masking. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 14, 2022, pages 4083-4091. ACM.
Tuan-Anh Nguyen Dang and Dat-Thani Nguyen. 2021. End-to-end information extraction by character-level embedding and multi-stage attentional u-net. arXiv preprint arxiv: 20X:::::6600000 200.005...
Zheng Huang. Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. 2019. lcdar2019 competition scanned receipt ocT and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516-1520 IEEE
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT. pages 4171-4186.
AR Dirkson, Suzan Verberne, Wessel Kraaij. E Holder- ness, A Jimeno Yepes, A Lavelli, AL Minard, J Puste- jovsky, and F Rinaldi. 2021. Fuzzybio: proposal for fuzzy representation of discontinuous entities. In Proceedings of the 12th International Workshop on Health Text Mining and Information Analysis, pages 77-82. Association for Computational Linguistics.
Wonseok Hwang, Jinyeong Yim, Seunghyun Park, Sohee Yang. and Minjoon Seo. 2021. Spatial dependency parsing for semi-structured document information extraction. In Findings of the Associ- ation for Computational Linguisties: ACI-IJCNLP 2021, pages 330-343.
Hiroshi Inoue 2019.
Multi-sample dropout for accelerated training and better generalization. arXiv preprint arXiv:1905.0978..
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11 1929.
Guillaume Jaume, Hazim Kemal Ekenel, and Jean- Philippe Thiran. 2019. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference Document Analysis and Recognition Workshops (ICDARW), volume 2, pages 1-6. IEEE
Lukasz Garncarek, Rafal Powalski, Tomasz Stanis- lawek, Bartosz Topolski, Piotr Halama, Michal Turski, and Filip trali'nski, 2020. Lambert: Layout- aware language modeling for information extraction. In IEEE International Conference Document Analysis and Recognition.
Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi. 2019. Pifpaf: Composite fields for human pose esti- mation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11977-11986.
Andrea Gemelli, Sanket Biswas, Enrico Civitelli, Josep Lladós, and Simone Marinai. 2022. Doc2graph: a task agnostic document understanding framework based on graph neural networks. arXiv preprint arXiv:2208.1 1666.
Chenliang Li. Bin Bi, Ming Yan. Wei Wang, Songfang Huang, Fci Huang, and Luo Si. 2021a. Structurallm: Structural pre-training for form understanding, In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Pupers). pages 6309-6318.
Zhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, and Liging Zhang. 2022. Xylayoutmm: Towards layout-aware multimodal networks for visually-rich document understanding.

---

Chenxia Li, Weiwei Liu, Ruoyu Guo, Xiaoting Yin. Kaitao Jiang, Yongkun Du, Yuning Du Lingfeng Zhu, Baohua Lai, Xiaoguang Hu, et al. 2022a. Pp-ocrv3: More attempts for the improvement of ultra lightweight oer system. arXiv preprint arXiv:2206,03001.
2021. Kleister: key information extraction datasets involving long documents with complex layouts. In Document Analysis and Recognition-ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5-10, 2021, Proceedings, Part I, pages 564-579. Springer.
Fei Li, ZhiChao Lin, Meishan Zhang, and Donghong Ji. 2021b. A span-based model for joint overlapped and discontinuous named entity recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4828.
Jonathan Stray and Stacey Svetlichnaya. 2020. Project deepform: Extract information from documents.
Jianlin Su, Ahmed Murtadha, Shengfeng Pan, Jing Hou, Jun Sun, Wanwei Huang, Bo Wen, and Yunfeng Liu. 2022. Global pointer: Novel efficient span- based approach for named entity recognition. arXiv preprint arx.v:2208.03054
Jingye Li. Hao Fei, Jiang Liu, Shengqiong Wu, Meishan Zhang. Chong Teng. Donghong Ji, and Fei Li. 2022b. Unified named entity recognition as word-word relation classification. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pages 10965-10973.
Buzhou Tang, Qingcai Chen, Xiaolong Wang, Yonghui Wu, Yaoyun Zhang. Min Jiang, Jingqi Wang. and Hua Xu. 2015. Recognizing disjoint clinical concepts in clinical text using machine learning- based methods. In AMIA annual symposium proceedings, volume 2015, page 1184. American Medical Informatics Association.
Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei. 2022c. Dit: Self-supervised pre-training for document image transformer. In MM '2: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 14, 2022. pages 3530-3539. ACM.
Yi Tu, Ya Guo, Huan Chen, and Jinyang Tang. 2023. Layoutmask: Enhance text-layout interaction in multi-modal pre-training for document understand- ing. arXiv preprint arXiv:2305.1872
Jiapeng Wang, Lianwen Jin, and Kai Ding. 2022. Lilt: A simple yet effective language- innaaggeeddeeedeentt layout transformer for structured document understanding. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pages 7747-7757.
Yulin Li, Yuxi Qian. Yuechen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, and Errui Ding. 2021c. Structext: Structured text understanding with multi-modal transfommers. Proceedings of the 29th ACM International Conference on Multimedia, pages 1912- 1920.
Yucheng Wang, Bowen Yu, Hongsong Zhu, Tingwen Liu, Nan Yu, and Limin Sun. 2021a. Discontinuous named entity recognition as maximal clique discov- ery. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 764-774.
Jiang Liu, Donghong Ji, Jingye Li, Dongdong Xie, Chong Teng, Liang Zhao, and Fei Li. 2022. Toe: A grid-tagging discontinuous ner model enhanced by embedding tag/word relations and more fine-grained tags. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:177-187.
Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jacheung Surh, Minjoon Sco, and Hwalsuk Lee. 2019. Cord: consolidated receipt dataset for post-ocr parsing. In Workshop on Document Intelligence at NeurlPS 2019.
Zilong Wang, Yiheng Xu, Lei Cui, Jingho Shang, and Furu Wei. 2021b. Layoutreader: Pre- Pre-training of text and layout for reading order detection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4744-474.
Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengji Huang, Teng Hu, Weichong Yin. Yongfeng Chen, Yin Zhang, al. 2022. Emni-- layout: Layout knowledge enhanced pre-training for visually-rich document understanding. arXiv preprint arxiv:2210.06155.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che. et al. 2021a. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2579-2591.
Lance A Ramshaw and Mitchell P Marcus. 1999. Text chunking using transformation-baat learning. Natural language processing using very large corpora, pages 157-176.
Tomasz Stanislawek, Filip Graliński, Anna Wröblewska, Dawid Lipinski, Agnieszka Kaliska, Paulina Ros- alska, Bartosz Topolski, and Przemyslaw Biecek.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. Layoutlm: Pre- training of text and layout for document image

---

understanding. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 192--1200.
A
The Annotation Pipeline and Statistics Revised Datasets
The annotation pipeline is introduced as follows. First, we collect the document images of FUNSD and CORD, and re-annotate the layout automati- cally with an OCR system. We choose PP-OCRv3 (Li et al., 2022a) OCR engine it is the SOTA solution of general-purposs OCR and can extend multilingual settings. We operate each document image of FUNSD and CORD, keeping layout annotations with more than 0.8 confidence, and filtering document images with more than 20 valid words. After that, manually annotate entity mentions the new layouts word sequences, based the original entity annotations. Entity mentions that camnot match word sequence on the new layout are deprecated. After the annotation, we divide the train/validation//ess splits according to the original settings to obtain the revised datasets. In all, the proposed FUNSD-r and CORD-r datasets consists of 199 and 999 document samples including the image, layout annotation of segments and words, and labeled entities. Table 4 lists the detailed summary statisties of the proposed datasets. The total number of segments, words, entities and entity types, the average length segments and entities, and the sample number of data splits are displayed. Additionally, the continuous entity rate is introduced as the rate of entity whose tokens are continuous in the order of segment words. When fed into model, the continuous entities correspond continuous token spans within inputs and are able be predicted by requence-labelinn
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang. Yijuan Lu, Dinei Florencio, Cha Zhang, and Furu Wei. 2021b. Layoutxim: Multimodal pre-training for multilingual visually-richh document understanding. arXiv preprint arXiv:2104.065: 004.6666..
Wenwen Chengquan Zhang, Haoyu Cao. Wei Hua, Bohan Li, Huang Chen, Mingyu Liu, Mingrui Chen, Jianfeng Kuang. Mengjun Cheng, al. 2023. ledar 2023 competition on structured text extraction from visually-rich document images. arXiv preprint arXiv:2.306.03287.
Yue Zhang. Zhang Bo, Rui Wang, Junjie Cao, Chen Li, and Zuyi Bao. 2021. Entity relation extraction as dependency parsing in visually rich documents, In Proceedings of the 2021 Conference Empirical Methods Natural Language Processing, pages 2759-2768.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. Ernie: Enhanced language representation with informative entities. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 1451-
methods. Therefore, the continuous entity rate indicates the ordered degree of the document layouts in the dataset.
B Baselines for VrD-EL and VrD-ROP
For VrD-EL, we compare the proposed TPP with the following strong baseline models:
GNN-based document encoders: GNN+MLP (Carbonell et al., 2021) is the pioneer to adopt GNN as the document encoder for better mod- eling of document inputs. Doc2Graph (Gemelli et al., 2022) is an enhanced GNN-based docu- ment encoder for VrD tasks.
Transformer-based document encoders: Lay- outXLM (Xu et al., 2021b) is pre-trained document encoder for multilingual document inputs. BROS (Hong et al., 2022) a pre-trrined document encoder focuses on achieving better

---

Case
Disordered Layout
Desired Reading Order
Reading Order by OCR Result
Multi-row Content
it R
#4
S dull ($#)
B # +
Seal with Askew or Overlapped Texts
#
# ($*)
* X08 ,
X08 U8 ;: fid: 1301 &
Print Distortion
$ # 01 W 13
$ T #h a
17 # ii As Ab: #
A 1
Design Luyout with Various Font Sizes
FU
(
+ #1# (#z bb:
3b: #15 (#c)
$ H
$ A X &o KLLA A A
Skewness due Photography
E 68040 JC Σ 16980
M 68040 16980
36 46525
i2 # 46522 3 1
Semantic-driven Reading Order
36 A
# a E 43.22
a + At 43.22
Figure 7: Examples of disordered layouts real-world scenarios. These examples are taken from (Yu et al., 2023).
of
# of
Avg. Length of Segment
# of Entities
Avg. Length of Entity
Cont.
# of Entity of Samples
Dataset
Segments
Words
(%)
Types
(Train/Val/Test)
FUNSD-r
10,091
166,040
16.45
7,924 12,582
15.21
95.74
149/-/50
CORD-
12,582
123,153
7.55
9.20
92.10
799/100/100
Table 4: Statistics of the proposed datasets. Cont. denotes the continuous entity rate, as the rate of entity whose tokens are continuous the order of segment words. Higher continuous entity rate indicates to more orderly layouts.
performance on key information extraction tasks. barsing-based methods: SPADE (Hwang al., 2021) treats VrD-EL spatial-aware depen- dency parsing problem and adopts spatial- enhanced language model to address the task. SERA (Zhang et al., 2021) also treats the task as dependency parsing, and adopts a biaffine parser together with a document encoder to address the task.
methods in Table 2 and 3. The backbone of all compared methods of base-level.
C Implementation Details
The experiments in this work are divided into two parts, as experiments on VrD-NER and on other tasks.
For VID-NER, we adopt LayoutLMv3-bune (Huang et al., 2022) and LayoutMask (Tu et al., 2023) as the backbones to integrate with Token Path Prediction. The maximum sequence length of textual tokens for both of them is 512. We use Global 1D and Segment 2D position information in LayoutLMv3, and Local ID and Segment 2D in LayoutMask, according to the preferred settings. We adopt positional residual linking and multi- dropout to improve the efficiency and robustness in training TPP. Positional residual linking adds the ID positional embeddings of each token to the
Detection- methods: MSAU-PAF (Dang et al., 2021) tackles the task by integrating the MSAU architecture for detection (Dang and Nguyen, 2021) and the PIF-PAF mechanism for link prediction (Kreiss al., 2019).
For VrD-ROP, we compare the proposed TPP with LayoutReader (Wang et al., 2021b), which is a sequence-to-seqqeenee model achieving strong performance on VrD-ROP. We refer to the reported performance in the original papers of these baseline

---

backbone outputs to enhance 1D position signals that is crucial to the task. Multi-dropout duplicates the last fully-connect layer of TPP. Following the setting of (Inoue, 2019), the duplicated layers are trained with different dropout noises and their weights are shared, which enhances model robustness towards randomness. The effectiveness of these mechanisms are discussed in Appendix D. In fine-tuning, we generally follow the original setting of previous VrD-NER works (Huang et al., 2022; Tu et al., 2023). We use an Adam optimizer, with 1 % linear warming-up steps, a 0.1 dropout rate, and a le-5 weight decay. For Token Path Prediction, the learning rate is searched from (3e- 5, 5e5, 8e-5). On FUNSD, the best learning rate is 5e-5/5e-5 for
Position
Backbone
FUNSD-r
CORD-r
ID
2D
Global
Segment
80.40
91.85
LayourLMvv
None
Segment
74.15
89.91
Global
Word
77.86
90.45
Local
Segment
78.19
89.34
LayoutMask
Glohal
Segment
66.24
87.36
None
Segment
67.81
82.27
Local
Word
75.96
89.45
(a) Ablation study on different 1D and 2D positional encoding choices of backhones.
Backbone
Method
FUNSD-r
CORD-r
TPP
80.40
91.85
Lv
(w/o mul.)
79.61
90.99
(w/o mul. & res.)
78.93
91.14
TPP (w/o mul.)
78.19
89.34
while on CORD the learning rate is 3e-5/8e-5, respectively. fine-tuning the comparing token classification models, we all set the learning rate to 5e-5. We adopt positional residual linking and multi-dropout in Token Path Prediction. We fine-tune FUNSD-r and CORD-r by 1,000 and 2,500 steps on Tesla A100 GPUs, respectively, with a batch size of 16. The maximum number of decoded entities is limited to 100.
LayoutMask
75.34
87.96
(w/o mul. & res.)
65.53
89.55
(b) Ablation study the desiga of Token Path Prediction. Positional residual linking and multi-dropout abbreviated and mul., respectively.
Table 5: Ablation studies TPP-for-VrD-NER position encoding choices of backbones and model design.
Besides to VrD-NER, we conduct experiments on VrD-EL and VrD-ROP adopting LayoutMask backbone. For VrD-EL, TTPP-fo-VrID-PR is fine- tuned by 1000 steps with learning rate of 8e-5 and batch size of 16, with the learning rate of the global pointer weights is set 10x for better convergence. For VrD-ROP, TPP-for- VrD-ROP is fine-tuned by 100 epochs with learning rate of 5e-5 and batch size of 16, following the settings of (Wang et al., 2021b). The beam size set during decoding. For all experiments, we choose the model checkpoint with the best performance on the validation set, and report its performance on the test set.
brings about the best performance. Although TPP is unaware of the order of tokens, the incorporation of better ID positional signals can enhance the contextualized representation generated by the backbones, leading to improved performance. For CORD-r, we note that some occasions, using word-level 2D position information brings about better performance. This is because short entities are the majority in CORD-r, especially entities with only one char, such as numbers and symbols. For these entities, word-level 2D position information brings about a direct hint the model in training and prediction, resulting in better performance.
For the second question, we conduct an ablation study by removing the positional residual linking and multi-dropout of TPP. We observe from Table 5b that: (1) Positional residual linking and multi- dropout are both essential due to their effectiveness on the results of FUNSD-r, T. (2) Positional residual linking enhances the 1D position signals in model inputs. Comparing with vanilla TPP, TPP with posi- tional residual linking behaves better FUNSD-r but slightly worse on CORD-1. This is because segments CORD-r are relatively short, when inputs are disordered, ID signals are much more noisy and negatively affects model performance
D Ablation Studies to TPP VrD-NER
For better understanding of TPP on VID-NER, we propose two questions: (1) from which setting TPP benefits most of positional encoding of backbones, and (2) by what means positional residual linking and multi-dropout be helpful to TPP. We conduct the ablation studies to answer the questions.
For the first question, we alter the choices of ID and 2D positional encoding of backbones, and the results are reported in Table 5a. For FUNSD-r, the preferred positional encoding setting backbones

---

since they do not provide sufficient information. The results verify the function of positional residual linking as a enhancement mechanism of ID signals. (3) Multi dropout enhances model robustness to random aspects. TPP outperforms its variant without dultiid among different backbones and different benchmarks.


Reviews:
[{"review_id": "EnGxZdll7z", "reviewer": "Reviewer_kJnu", "summary": "", "questions": "Question A: Any insight as to why using LayoutReader for preprocessing results in even lower continuity and F1 scores, than when using standard sequence labeling LayoutLM on unmodified dataset.\n\nQuestion B: In table 3 it appears different shuffling ratios r apply to OCR-ordered input, which I understand is not shuffled. How can this be?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your reviews and your appreciation to our work. \n\nWe notice your suggestion that our work could also be named as token trajectory prediction (TTP), which we think is also reasonable. To maintain consistency within the ongoing OpenReview discussion, we retain the original name of our method as Token Path Prediction (TPP).\n\nYour concerns are addressed as follows:\n\n\n\n**Q1:** About Reasons To Reject Point 1, \"the practical impact of TPP may be limited as entity continuity rates are relatively high in revised datasets\"\n\n**A1:** In fact, TPP becomes even more useful in practical applications, as there would be massive cases of document layouts that exhibit intensive disorder, such as (1) information extraction on scanned cards/passports with multi-row contents, seals with askew texts, and registration forms and tables with print distortion, for document analysis; and (2) visual qa on infographics and hand-written documents for document understanding. Moreover, in these documents, the disordered entities are usually critical ones (eg. dates, seal texts, etc. ), and are difficult to be corrected by regular matching. **It is this practical reality that primarily drives our motivation.** In the revised version, we will provide more real-world document examples with severe disordered layouts to further support our claims. \n\nBesides, compared with real-world scanned documents, FUNSD-r and CORD-r have neat layouts and thus have higher entity continuity rates. Despite this advantage,  sequence-labeling methods on them still suffer from significant performance drop. This observation indicates the seriousness of the reading order issue in practical scenarios. \n\n\n\n**Q2:** About Reasons To Reject Point 2, \"the possible data leakage in experiments\"\n\n**A2:** By \"Margin of improvement for CORD-r seems suspiciously higher than for FUNSD\", we think you refer to the performance of \"Document Transformer + None Pre-processing + TPP-NER\", abbriviate as \"None+TPP\", which is 91.85/89.34 with two backbones, and is much higher than None+SL (82.72/81.84). However, **None+TPP does not leverage any ground truth reading order annotations of CORD**, indicating that its superior performance is not a result of data leakage. As illustrated in Section 5.5, we believe it is the ability of identifying the entity boundary of TPP that attributes to the superior performance of None+TPP. \n\nHowever, the comparison between LR+SL and TPP_C+SL is somehow unfair, as the latter is aware of the ground truth reading order annotations of CORD, while the former is not. Due to it, we finetuned LayoutReader on CORD, noted as \"LR_C\", adopt it as the pre-processing mechanism, and the results are as follows: \n\n- Entity continuity rate after LR_C pre-processing: 82.10.\n- The F1 score of LR_C+SL on CORD: 70.33 for LayoutLMv3, and 68.05 for LayoutMask.\n\nWe will replace the original results of LR+SL in Table 1b by the above results in the revised version. Note that the newly-reported results are still lower than that without pre-processing mechanisms. This phenomenon is illustrated below. \n\n\n\n**Q3:** About Question A, \"why using LayoutReader for preprocessing results in worse performance\"\n\n**A3:** According to [1], LayoutReader primarily focuses on optimizing BLEU scores on the benchmark, where the model is only required to accurately order 4 consecutive tokens. However, according to Table 4 in the appendix of our paper, the average length of entities is 16 and 7 words in the datasets. In this case, any disordered token within an entity leads to discontinuity. Consequently, LayoutReader may occasionally mispredict long entities with correct input order, due to its seq2seq nature which makes it possible to generate duplicate tokens or missing tokens during decoding.\n\nTo address this issue more clearly, we will reorganize the content and provide clearer illustrations of this phenomenon in the revised version.\n\n\n\n**Q4:** About Question B, \"confusion on the settings of Table 3\"\n\n**A4:** For Table 3, we adopt the identical settings as LayoutReader [1]: the rate r% implies that r% train samples are randomly ordered, while the rest train samples remain in OCR order; the abbreviation OCR and Shfl. denotes that the validation samples are in OCR/random order. \n\nTable 3 illustrates the superiority of TPP for reading order prediction, as its performance is generally better than LayoutReader, and is invariant of the shuffle rate r%. \n\n\n\n**Q5:** About suggestions on Typos, Grammar Style And Presentation Improvements\n\n**A5:** We thank you for pointing out the typos. These mistakes would be corrected in the revised version.\n\n \n\nReferences:\n\n1. Wang, Zilong, et al. \"Layoutreader: Pre-training of text and layout for reading order detection.\" arXiv preprint arXiv:2108.11591 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposal of a reading order invariant labeling scheme TPP for VDU is well motivated and is novel, as far as I can tell.\n\nThe authors' solution of using TPP as grid label prediction is simple but sound and offers a number of advantages including the added ability to perform entity linking. They make appropriate adjustments to the training, such as using class-imbalance loss to accommodate the new labeling scheme.\n\nThey make reasonable modifications to the annotations of the existing datasets to highlight the shortcomings of reading order dependence in the presence of OCR errors; and conduct a compelling evaluation of their approach in multiple settings (as a standalone NER and as preprocessing for standard sequence labeling), which shows significant gains of using their method across NER, EL, and ROP tasks.\n\nTheir ablation and case studies appear compelling as well.", "reasons_to_reject": "While conceptually the TPP makes a lot of sense, its practical impact may be limited as entity continuity rates seem still relatively high even in modified datasets. \n\nRegarding TPP training for reading order prediction, the original CORD dataset is used for training, which could be causing data leakage. (Margin of improvement for CORD-r seems suspiciously higher than for FUNSD)", "questions_to_authors": "Question A: Any insight as to why using LayoutReader for preprocessing results in even lower continuity and F1 scores, than when using standard sequence labeling LayoutLM on unmodified dataset.\n\nQuestion B: In table 3 it appears different shuffling ratios r apply to OCR-ordered input, which I understand is not shuffled. How can this be?", "paper_topic_and_main_contributions": "The authors point out and provide multiple examples of the shortcomings of the traditional sequence tagging scheme when applied to entity extraction from visually rich document, such as reliance on correct reading order, which is hard to obtain of realistic OCR. \n\nThey propose an alternative to sequence labeling for NER, token path prediction (TPP) that is invariant to reading order where all tokens in the document are treated as nodes of a graph and entities correspond to paths through this graph.\n\nTo do TPP, they modify the token standard classification head for binary grid label prediction.\n\nThey employ their approach as a standalone system for NER and entity linking (EL) as well as, reading order prediction (ROP) for preprocessing the input for standard sequence labeling.\n\nFinally, they showcase performance gains on modified versions of FUNSD and CORD to include raw reading order errors and entity discontinuities. They show moderate to dramatic improvement over standard LayoutLMv3 and a collection of base lines including ones specifically designed to handle reading order issues.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n\ntokens. In specific, for each edge in every token 282\n\n268 we modeling"}, "extra_text": {}}, {"review_id": "KbgCh3RANA", "reviewer": "Reviewer_bY5f", "summary": "", "questions": "QA:  I think it would be interesting to see how the proposed the model preform on the original datasets (not the revised one) and comparing to the previous work. ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your reviews and your appreciation to our work. Your concerns are addressed as follows:\n\n**Q1:** About Reasons To Reject Point 1 and Question 1, \"no fair comparisons with the existing methods\"\n\n**A1:** It is unable to make fair comparisons between TPP and other methods on FUNSD and CORD due to the following reasons: (1) Other methods leverage the spurious correlation between segment layout and entity annotations, leading to falsely high performance. As discussed in Section 4.1, it is demonstrated that each entity corresponds exactly to one segment in FUNSD and CORD. The segment layout annotations of FUNSD and CORD leaks the entity boundary information, and previous models take advantage of this. (2) The task for TPP is harder to tackle VrD-NER. TPP is unaware of the input order of tokens. Therefore, in addition to detecting entity boundaries like other methods, TPP needs to arrange the token order within the entity as well. \n\nHowever, **fair comparisons on public benchmarks are conducted** for VrD-EL and VrD-ROP tasks, where TPP outperforms other strong baseline methods and achieves SOTA performance, even without any specialized design or optimization for these particular tasks. We believe that this is enough to prove the competitiveness of TPP among other methods. \n\n\n\n\n\nAdditionally, there are two clarifications that need to be addressed:\n\n1. Re-emphasizing our motivation\n\n- In response to your review comments, we would like to re-emphasize the main motivation behind our paper, to aware of the difficulties of VrD tasks in real-world. Throughout the article, we have consistently emphasized the criticality of the reading order issue in real-world VrD tasks, particularly VrD-NER. To our best knowledge, this issue has not been adequately addressed to date. Besides, we note that current public benchmarks (FUNSD, CORD, ...) fail to highlight this problem, as they only annotate entity labels on continuous word sequences. Consequently, existing models are affected that they focus on improving performance on these benchmarks, but none have successfully resolved the reading order issue. \n\n- Our work aims to draw the attention of researchers to this phenomenon, and inspire the development of new VrD-NER models to meet the need of practical applications. To accomplish this, we re-formulate the task and introduce two benchmarks (FUNSD-r and CORD-r) as well as one baseline method (TPP). By doing so, we hope to foster discussions within the field and encourage the development of solutions for real-world VrD tasks, but not overfitting benchmarks.\n\n2. About the concern of the reproducibility of our work\n\n- We notice that this review addresses some concern about the reproducibility of our work. The two proposed datasets, codes and model checkpoints would be released after this paper is accepted. \n\n- Besides, all experimental details have been thoroughly presented in Appendix C to demonstrate that our experimental results are trustworthy and reproducible, which we believe reflects our utmost dedication and precision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a token path prediction method to dealing with the reading order issue in the visually-rich documents for NER tasks. The proposed method outperforms SOTA methods in various VrD tasks. ", "reasons_to_reject": "The experiments are conducted on the revised dataset, so there is no fair comparisons with the existing methods. ", "questions_to_authors": "QA:  I think it would be interesting to see how the proposed the model preform on the original datasets (not the revised one) and comparing to the previous work. ", "paper_topic_and_main_contributions": "The paper proposes a method to deal with the reading order issue in the NER tasks on visually-rich documents. Also the authors present two revised benchmark datasets of NER tasks on scanned documents. Experimental results show the proposed method is efficient. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "K7k8mgePhl", "reviewer": "Reviewer_crwh", "summary": "", "questions": "1) How could you add visual channel/input to your model? Would it benefit?\n2) Can we overcome the constructing graphs of tokens on the plain page by utilising visual embeddings?\n3) How is your work related to the LAMBERT model and its input types, described in [1]?\n4) Can you explain Figure 5? I cannot see much difference; maybe you can point out particular bounding boxes or describe them in the figure description.\n5) Please, add the continuation of the sentence in line 1021 in the appendix.\n\n[1] Garncarek, Ł. et al. (2021). LAMBERT: Layout-Aware Language Modeling for Information Extraction. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12821. Springer, Cham. https://doi.org/10.1007/978-3-030-86549-8_34\n", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your reviews and your appreciation to our work. \n\nWe also thank you for recommending a related work, LAMBERT [1], which is an important work of self-supervised pre-trained document transformer with text+layout inputs at the era it is proposed. It achieves great performance on several VrD-IE tasks.  We thank you for sharing this interesting work, and we will add the citation of this work in the revised version. \n\nYour concerns are addressed as follows:\n\n\n\n**Q1:** About Question 1, \"would it benefit to add visual inputs\"\n\n**A1:** Token Path Prediction (TPP) is a VrD-NER task head and is compatible with commonly used document transformers. Whether to use visual inputs is decided by the upstream document transformers but not TPP. For your question, leveraging visual inputs would be beneficial as in Table 1, those results of LayoutLMv3 are generally better than those of LayoutMask. \n\n\n\n**Q2:** About Question 2, \"overcoming the construction of token graphs by adding visual inputs\"\n\n**A2:** As illustrated above, we can choose whether to use visual inputs, but merely using visual inputs is not enough to solve VrD-NER with reading order issue. The solution relies on the modeling and decoding strategy of TPP. \n\n\n\n**Q3:** About Question 3, \"the relation between our work and LAMBERT and its input types\"\n\n**A3:** As stated in the response of Question 1, TPP can be integrated together with LAMBERT, where LAMBERT serves as the upstream representation model of document inputs, and TPP acts as the downstream task model. Regrettably, due to the limited time of author response period, we are unable to present experiment results to demonstrate the effectiveness of LAMBERT+TPP on the VrD tasks. However, considering that LAMBERT shares the same input modalities and possesses a similar size of LayoutMask, the performance gain of LAMBERT by incorporating TPP should be comparable to the results on LayoutMask. \n\n\n\n**Q4:** About Question 4, \"precise explanation of Figure 5\"\n\n**A4:** We apologize for the lack of clarity, and we will reorganize the content of Section 4.2 to illustrate this figure more clearly in the revised version. There are basically 3 differences between the original and revised layouts:\n\n1. The original layout annotates word-level position boxes, yet the revised layout annotates character-level position boxes. The latter is more in line with the outputs generated by OCR systems. \n2. In the original layout, segments may span multiple rows, but in the revised layout, every segment lies in a single row. For example, the long paragraph \"Spray 50 lbs ... 6.5 mg/cig. \" is annotated as a whole segment in the original layout, but split to 7 segments in the revised layout, with each row being annotated as an individual segment. \n3. The segment annotations on the original layout are manually labeled, ensuring a generally precise alignment with the semantic units of texts. However, in the revised layout, the segment annotations are labeled by text-region detection models and do not consider the semantic meaning of texts. As a result, (1) some unrelated texts may be grouped together in the same segment, such as \"Filter Production\" and \"Wicker\"; and (2) some short texts, like the symbol \" on the right, may not be recognized by the OCR system. \n\n\n\n**Q5:** About Question 5, \"one sentence is unfinished in the Appendix\"\n\n**A5:** We thank you for pointing out this mistake and sincerely apologize for it. This mistake would be corrected in the revised version. The complete sentence at line 1021 should be, \"This is because segments in CORD-r are relatively short, when inputs are disordered, 1D signals are much noisier and could not provide sufficient information.\"\n\n\n\nReference:\n\n1. Garncarek, Łukasz, et al. \"Lambert: Layout-aware language modeling for information extraction.\" International Conference on Document Analysis and Recognition. Cham: Springer International Publishing, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the Authors carefully analysed the topic and challenges in visually rich document recognition tasks.\n- then, they carefully revised the existing two datasets (I.e. FUNSD, CORD) and re-annotated them.\n- then, they proposed a new solution that defeated other benchmarks.\n- the paper is very clear, with vivid examples and descriptions even for less-informed readers - I appreciate it very much!", "reasons_to_reject": "I have not found any.", "questions_to_authors": "1) How could you add visual channel/input to your model? Would it benefit?\n2) Can we overcome the constructing graphs of tokens on the plain page by utilising visual embeddings?\n3) How is your work related to the LAMBERT model and its input types, described in [1]?\n4) Can you explain Figure 5? I cannot see much difference; maybe you can point out particular bounding boxes or describe them in the figure description.\n5) Please, add the continuation of the sentence in line 1021 in the appendix.\n\n[1] Garncarek, Ł. et al. (2021). LAMBERT: Layout-Aware Language Modeling for Information Extraction. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12821. Springer, Cham. https://doi.org/10.1007/978-3-030-86549-8_34\n", "paper_topic_and_main_contributions": "At first, the Authors pointed out problems with current datasets and 2 benchmarks on visually rich document recognition tasks (NER, EL< ROP).\nThen, they propose a new method for token path prediction and explain their approach and revised datasets for those tasks.\n", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I have not found any, only in line 1021 - not ended one sentence."}, "extra_text": {"Missing_References": "This work is somehow related: [1] Garncarek, Ł. et al. (2021). LAMBERT: Layout-Aware Language Modeling for Information Extraction. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12821. Springer, Cham. https://doi.org/10.1007/978-3-030-86549-8_34\n\nI hope the authors can compare and it can add additional interesting perspective to the paper."}}]

Author rebuttal:

Meta-review: The paper pointed out problems with current datasets and 2 benchmarks on visually rich document recognition tasks (NER, EL< ROP). The authors propose a new method for token path prediction and explain their approach and revised datasets for those tasks. The proposal of a reading order invariant labeling scheme TPP for VDU is well motivated and is novel,
Decision: Accept-Main