DocTrack: A Visually-Rich Document Dataset Really Aligned with Human Eye Movement for Machine Reading
Authors: Hao Wang
Qingxuan Wang
Yue Li
Changqing Wang
Chenhui Chu
Rui Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: visually-rich document
dataset
eye tracking
human reading order
preordering
TL;DR: A dataset of visually rich documents containing human eye movement information; preordering models can output human-like reading orders

Abstract: The use of visually-rich documents in various fields has created a demand for Document AI models that can read and comprehend documents like humans, which requires the overcoming of technical, linguistic, and cognitive barriers. Unfortunately, the lack of appropriate datasets has significantly hindered advancements in the field. To address this issue, we introduce DocTrack, a visually-rich document dataset really aligned with human eye-movement information using eye-tracking technology. This dataset can be used to investigate the challenges mentioned above. Additionally, we explore the impact of human reading order on document understanding tasks and examine what would happen if a machine reads in the same order as a human. Our results suggest that although Document AI models have made significant progresses, they still have a long way to go before they can read visually richer documents as accurately, continuously, and flexibly as humans do. These findings have potential implications for future research and development of document intelligence.

DocTrack: A Visually-Rich Document Dataset Really Aligned with Human Eye Movement for Machine Reading
Hao Wang', Qingxuan Wang', Yue Li', Changging Wang', Chenhui Chu and Rui Wang*
School of Computer Engineering and Science, Shanghai University, China 2Grauuate School of Informatics, Kyoto University, Japan DDpartment of Computer Science and Engineering, Shanghai Jiao Tong University, China (wang-hao, wqx, ly_ces, wcqing)@shu. edu. cn chu@i. koriiikkooouucccc ac. jp, wangrui j ju. edu.cn cn
Abstract
Rirheript Banker Fornje
OCR Output Order
The use of visually-rich documents (VRDs) in various fields has created demand for Doe- ument Al models that can read and compre- hend documents like humans, which requires the overcoming of technical, linguistic, and cognitive barriers. Unfortunately, the lack of appropriate datasets has significantly hindered advancements in the field. To address this is- sue, we introduce DOCTRACK, VRD dataset really aligned with human eye-movement in- formation using eye-tracking technology. This dataset can be used investigate the chal- lenges mentioned above. Additionally, we explore the impact of human reading order on document understanding tasks and exam- inc what would happen if machine reads in the same order as a human. Our results sug- gest that although Document AI models have made significant progress, they still have long way to go before they can read VRDs as accurately, continuously, and flexibly as hu- mans do. These findings have potential impli- cations for future research and development of Document Al models. The data is available at https://git thub. com/hint lab/ doctrack,
TOBACOO SCIENCE
Zeldritiin NX
AUTHOn
Jnizes f. Faiiiano,
Fiffact Trleess sa the satechol. Hleld In
Clarrette Burie*
Hiasseript Berion
Human Reading Order
SCIENCE
ttnit Kilk
Stach 18, 1858
Jodzn g. Ballianes, Bichard Yorra Janes Wolds
Petfict of jétretee
Hield
Glommtte Sooky"
Figure 1: Case comparison of the default serialized in- order output by the OCR engine with the real hu- man reading order. It clearly reveals the significant dif- ferences in processing behavior between machines and humans. The red numbers indicate the ordinal numbers the reading sequence. For best visibility, is recom- mended to zoom in on the results.
Identifying and understanding VRDs is a time- consuming and laborious task due to their diver- sity and complexity (Wang et al., 2021a). Tex- tual information alone is insufficient for extracting key information from diverse document types, ne- cessitating multimodal approach that considers the consistency and correlation of multiple modal- ities, including text, visual, and layout, through joint modeling. Examples of modern document AI models for VRD understanding include the LayouLLM series models (Xu et al., 2020, 2021; Huang et al., 2022) and StructText series models (Li et al., 2021; Yu et al., 2023a).
1 Introduction
With the continuous development of information technology, our access to information is becoming increasingly diverse. Among the various formats, the proportion of visual information in daily doc- uments such tables, graphs, diagrams, etc., is the rise (Ceci et al., 2007; Jaume et al., 2019). Therefore, effectively utilizing such visual infor- mation has become a hot research topie the NLP research community and introduces a new chal- lenge of understanding Visually-Richs Documents (VRDs) (Liu et al., 2019; Yu et al., 2021), which are documents that contain substantial visual com- ponents.
While these models can obtain fine-grained multimodal document representations and achieve promising results in downstream VRD (VRD) un- derstanding tasks, they lack the ability to gener- ate a serialized input order from a given document
"Corresponding author

---

that fits into the Transformer architecture. As re- sult, they typically utilize simple rules, such left- to-right or top-to-bottom, or directly use the input order generated by the OCR tool in the previous step (Lee et al., 2021) I) to serialize inputs. However, these input orders are quite different from the read- ing order that humans are used to (See Figure 1 for an example). The different input orders sig- nificantly affect the performance of document Al models on the downstream document understand- ing tasks, which often overlooked.
order generation methods, which refer techniques for generating machine reading or- ders that mimic human reading patterns in VRDs. We explore various techniques for generating these reading orders and propose a practical preordering pipeline that leverages these generated reading orders to improve document understanding tasks.
3. We conduct both intrinsic and extrinsic eval- uations to analyze the performance of the human-like reading order generation model and measure its impact on downstream tasks. The observations suggest that human reading order may not be suitable for reading VRDs.
To address this issue, Wang et al. (2021b) con- struct the ReadingBank dataset containing the lo- cal priority order using the order of the XML source code of Word documents. However, it re- mains questionable whether this reading order is consistent with the actual human reading order and whether it actually beneficial for machine com- prehension tasks. Therefore, state-of-the-art Docu- ment Al models lack deep understanding of spa- tial relationships and document structure, resulting in limited performance when dealing with VRDs, c.g., forms and infographies.
2 Human Reading Order
The human reading order plays a significant role in the comprehension of VRDs. Human reading order refers to the direction and sequential order in which people scan documents as they read the texts. Generally, people read left-to-right top-to- bottom to comprehend the text and obtain informa- tion. This order is also related the way the text is written in most languages and scripts, includ- ing English and Modern Chinese. However, other reading order patterns exist. For example, tradi- tional Chinese and Japanese are usually written vertically from right to left, while Arabic is written horizontally from right to left (Rayner, 1998). As result, people naturally scan their eyes in this or- der while reading (Henderson and Ferreira, 1993). Although the reading order of text generally follows a relatively fixed pattern, VRDs usually present information in a mixture of modalities (e.g., text, images, graphics, etc.) with com- plex two-dimensional layout and semantic struc- ture. Readers typically choose the appropriate reading order by considering the spatial structure of text content, image information, and the typo- graphic position of the document in combination. This natural temporal processing can help readers better understand the content and intent of the doc- ument and establish connections between different modalities. For example, when presented with a diagram, readers typically first visually analyze it as whole to get general idea of what it describes. From there, they usually focus on the most obvi- ous data labels first, and then gradually scan the rest the data and labels to gain a more complete understanding of the information presented.
To this end, we propose DOCTRACK, benchmark dataset containing various types of real-world documents aligned with human eye- movement information. Specifically, we propose a preordering pipeline to integrate human reading order within modern document Al models. The integration process occurs during inputting docu- ment contents after OCR parsing but before feed- ing them into the downstream task. We also ex- plore different approaches to generate human-like reading orders, including default OCR tools, Z- pattern, simple rules, and AI models that utilize multimodal information. Our objective to evalu- ate the impact of different reading orders on down- stream document comprehension tasks and iden- tify the most effective reading order for existing multimodal document AI models. This provides insight into the similarities and differences be- tween human and machine reading patterns.
In summary, this paper makes three contribu- tions:
1. We construct a benchmark dataset, namely DOCTRACK, aligned with human eye move- ment information. To our knowledge, Doc- bench-
TRACK is the first human-annoter mark dataset for the purpose of research on VRD reading order generation.
2. We investigate different human-like reading

---

3 Eye Tracking in NLP
4 Dataset
4.1 Data Collection
3.1 Basic Notions
Our work aims to evaluate how reading order im- pacts the comprehension of VRDs. To achieve this, randomly select documents mainly from three available datasets: FUNSD (Jaume al., 2019), SeaBill (Zhang et al., 2022), and Info- graphic (Mathew et al., 2022). These datasets are widely used and provide diverse examples of complex structured and graphic documents for our analysis. We reuse all document images in
Eye tracking is an important technique for study- ing eye movements during human reading. Hu- man reading is a complex cognitive activity, and eye movement trajectories can visualize the read- ing process and are important for understanding how humans acquire and comprehend knowledge. Intuitively, different visual tasks result in varying scan (i.e., eye movement) patterns. Studying these patterns can help us understand the mechanisms of human cognitive processing. Numerous research in neuroscience has established a strong associa- tion between eye-tracking data and language com- prehension activity in human brains (Henderson and Ferreira, 1993). The eye movement trajectory is typically described an irregular curve, mainly composed of two alternating eye movement ac- tions: saccade and fixation. A saccade is when our eyes rapidly jump from one word to the next in the text, representing the shift attention. Fixa- tion refers to the situation we sometimes focus on a word and remains stationary during the visual task for period of time. In addition, the scan pat- tern and speed of the eye movement trajectory can be affected by various factors, such font size, line spacing, contrast, etc. of the text.
WRAK
STRUCTURED
INFOGRAPH
TOTAL
Pattern
norm-2
local priori
cross&
Train
doc
149
160
100
409
ent
7,441
10,024
12,650
30,115
22,512
16,055
24,364
62,931
doc
50
50
30
130
ent
2,332
3,430
3,794
9,556
tok
8,973
7,022
7,308
23,123
Table I: Data statistics in the DOCTRACK dataset cate- gorized by the type of VRDs. We also give the primary scan patterns in each subset.
the FUNSD dataset (Jaume et al., 2019). Given that these documents are less structured than the other datasets and the eye movement tracks pri- marily to conform to the normal-Z reading order pattern, we rename this sub-dataset as "WeAK." We have selected number of structured tabu- lar documents from the SEABILL dataset that contains detailed information related to the ship- ment of goods. This information includes the name of the consignor and consignee, the type and quantity of the shipment, etc. The subset com- prises 160 training samples and 50 testing sam- ples, totaling 13,454 semantic entities, denoted as "STRUCTURED." We also select 100 training samples and 30 test samples from the Infographic dataset (Mathew et al., 2022). This subset contains a large number of graphic components, it is more diverse and complicated than the other two sub- sets, called "INFOGRAPH." Table shows details of the dataset and statistics.
3.2 Applications
cognitive-motivate natural language process- ing (NLP), several studies have investigated the impact of eye-tracking data on NLP tasks, for example. designing machine learning models for NLP tasks such as part-of-speech tagging (Bar- rett et al., 2016), term extraction (Yaneva al., 2017), and syntactic parsing (Kim, 2009). Later, researchers combine eye-tracking data with word embeddings in neural models to improve NLP tasks, including sentiment analysis (Mishra et al., 2017) and named entity recognition (NER) (Hol- lenstein and Zhang, 2019) or revising neural atten- tion (Barrett et al., 2018: Sood et al., 2020; Tak- maz et al., 2020). Recent studies (Bhattacharya al., 2020; Ren and Xiong, 2021; Ding et al.. 2022; Khurana al., 2023) attempt to align text features and cognitive signals to identify their dif- ferences and commonalities.
4.2 Analysis of Eye Movement Patterns
We observe mainly four types of reading patterns among these documents. Figure 2 illustrates hu- man reading behaviors when reading the docu- ments from the DOCTRACK dataset.
Normal-Z. The normal-Z order, also called

---

COVID 99: High Mortality Rangs
Linked To
Stide ong
GHIE
St
bor
and
(a)
(c)
(d)
Australia is Warming Faster
tha the Global Veeagee
Annut Jean temperahunn anomaly Angollla world Celciu.
+1.52
1,5
cecelake
1.0
Figure 2: Scatterplots showing eye movement patterns while humans are reading can be organized in various ways: (a) normal-Z order; (b) local priority order; (c, d, e) modsl interaction order, (f) visual instruction order. Note that these eye movement data have not yet been aligned with the OCR output. The full version of the images can be found in Appendix A.2.
"zigzag pattern," refers to the typical eye move- ment pattern that occurs when people read pure text weakly structured documents. This zigzag pattern is efficient way for the human visual sys- tem process and understand text. Specifically, our eyes move from left to right along line of text and then jump back to the beginning of the next line during reading, creating a zigzag scan trajectory of eye movements. Figure 2(a) shows the diagram of eye movement sequences during human reading, which often resembles the shape of the letter "Z"
of data graphs, they typically show interactive ra- dial eye movements. During reading, the eyes fol- low radial path between the graph and the asso- ciated text. For example, when people read pie chart, they usually focus first on the pie portion of the chart and then scan along its perimeter find the corresponding text. This back-and-forte movement results in a cross-modal interaction pat- tern. The unique shape and layout structure of pie charts contributes to this pattern because the graph- ical portion often contains the most important in- formation that people attend to first. In contrast, the text label section provides detailed information that people need to scan step by step in order to un- derstand. Therefore, eye tracking presents a radial pattern in these types of infographics.
Local priority. Local-priority eye-movement pat- terns are common cognitive strategy that humans use when dealing with the hierarchical layout structure of forms or tables. Usually, as shown in Figure 2(b), by prioritizing local information, we focus on the content inside a tabular cell first and then shift our attention to other elements around the cell while looking up in the tabular or form document. This facilitates quick reading and re- trieval of informative text obtain the needed in- formation.
Visual instruction. When reading flowcharts, an eye movement pattern characterized by backtrack- ing often develops, readers must understand and compare different texts in the boxes. Flowcharts commonly contain great deal of information and detail presented in hicrarchical, logical manner. Readers typically focus first on the chart's over- all structure and theme, then progress gradually to each detail. When encountering text that requires
Cross-modal interaction. When people read in- fographics that contain pies, charts, or other types

---

Overall Distribution
Cross-modal interaction details
ing adjacent reading sequence or the ordinal number between two gaze points.
Nomala
Train
Train
Local printty
3. We delete gaze points that are repeatedly turned the eye multiple times and keep only the ordinal numbers of the first eye mo- ment.
Teat
Graerodd Intrrtie
Use
Yassaammdion
Figure 3: The statistics of four types of reading pat- terns.
By adopting the above corrections, ensure the accuracy of the recorded reading trajectory.
4.4 Annotation Agreement
contextual referencing, readers may re-read earlier sections aid in better understanding the informa- tion's meaning. This self-correcting, backtracking eye movement pattern reflects the information pro- cessing of the reader and can help with better com- prehension and memory retention of the informa- tion. Figure shows the statistics of the four types of reading patterns found in DOCTRACK
In our final experimental setup, we assign two out of the five participants label the same subset of data. Subsequently, for each document file, we compare the labeling results from these two partic- ipants. We then conduct a voting process among the five participants to select the most appropriate labelingdocument--eelliie aligns with everyone's expectations. This chosen labeling is ultimately considered as the final data.
4.3 Eye-tracking Experiments
To conduct cye-tracking experiments, we ran- domly divide the dataset into five parts and recruit five participants.' All participants are instructed to read the data on an HP 24-inch 1080P display. We record the participants' eye trajectories while they reading using Tobii TX300 and Tobii Stu- dio. These devices can record the visual move- ment trajectory of each subject and convert into digital data. Data extracted from the eye tracker includes fixation points, fixation time, frequency, eye saccade distance, and pupil size. These mea- surements can help researchers gain insights into participants' internal cognitive processes.
Reading Order Integration
5.1 Rule-based Heuristic Methods
Multi-modal information extraction methods rely accurate sequence detection of documents. However, inconsistencies in OCR engines can lead to variances in reading order. To address this issue, Li al. (2022) introduce position offset threshold to standardize reading order and deal with OCR s instabilities. The text boxes are sorted from top to bottom, and if the distance between two boxes in the Y direction is smaller than the threshold, then their order is determined based their X direction order. Besides, Gu et al. (2022) the bounding box according to the two coordi- nates of the -axiis and then performs the X-axis search and the right-down search with Y +X com- bined consideration. By changing the order of to- ken input into the model, both approaches achieve good results. The proposed rule-based sorting ap- proaches conform to the basic cognition of hu- mans when reading, resulting in accurate informa- tion extraction. Therefore, leveraging the reading order generated by rule-based sorting approaches can significantly improve the accuracy of multi- modal information extraction.
The high sampling frequency of an eye tracker may result in an unsmooth trajectory with recorded gaze points. In addition, gaze points dur- ing eye movements may deviate or miss due to pe- ripheral vision (see Appendix A.1). To improve the accuracy of the eye tracker when recording the reading trajectory, we do as follows:
1. In the case of missing gaze points (case 1), if the gaze point hits the periphery of the known OCR bounding box within certain Euclidean distance, we use the ordinal num- ber of the peripheral gaze point as the reading sequence index of the current bounding box.
5.2 Modal-based Preordering Methods
2. In the case of missing gaze points (case 2). the ordinal number from the surround-
5.2.1 Proposed Pipeline
To follow human reading behaviors, we use a process called "preordering" (reordering-as-
'All participants are graduate or undergraduate students.

---

Atomic comparison model
PC-1
0g
dist
rEPc-al > r(h]
Downstream Tasks
...
Transformer/BERT LayoutLM...
OCR
conca
Classifier
Initial Order
Sequence Preordering
Human-like Order
text
position
Input
Input
Figure 5: Atomic reading order comparison model for neighboring bounding boxes. The symbol ">" indi- cates that the former bounding box might be read by humans after the latter.
Figure 4: The proposed preordering pipeline for utiliz- ing human-like reading orders in VRD understanding tasks.
preprocessing), a term borrowed from the domain of statistical machine translation (Xia and Mc- Cord, 2004; Collins al., 2005; Neubig et al., 2012; Nakagawa, 2015). This process involved reorganizing the inputs the order they would be read by human. By this, we make casier for the reader to understand the procedure of aligning the multimodal sequential input features with human reading order and thus enabling to evaluate the impact of reading order the VRD understanding tasks.
positions learn the order of each bounding box. Specifically, at first, the given bounding box coor- dinates (Zups Mup- ddown Mdown) (representing the top-left and bottom-right coordinates) generated by the OCR tool are used to compute the cen- troid coordinates (Ix, of each bounding box. We combine all the bounding box centroid coor- dinates in pairs, then feed the centroid coordinates of the combined bounding boxes directly into multi-layer Transformer network. This enables us to predict the spatial relationship between the two bounding boxes, i.e., which one should come before or after in the human reading order.
5.2.2 Atomic Comparison Models
We obtain the basic features of different modali- ties of documents through different encoders and imitate human reading order according to these features. Therefore, we propose four different models for reading order generation. Each model uses information from single multiple modal- ities simultaneously: Box, Text, Text+Box, and Text+Box+Image.
b, : by Transformer( yr: Vz.
(2)
Text. We use BERT to encode the texts, and since each bounding box contains take the latent representation at the first token position the embedding for bounding box.
more tokens,
b, by BERT(T; t2)
(3)
Each model takes into account different factors that influence how humans prioritize and read el- ements in VRDs, including the position of the el- ement, the text within the element, and the visual region associated with it. By utilizing these mod- cls, we more accurately evaluate the impact of reading order human comprehension of such documents.
Text+Box. To jointly encode the text and 2D- positions inputs, we use LayoutLM.. Similar to the operation in the Text section, we take the first latent representation the input to the classifier. The final model can be formulated follows:
by : by = LayoutLM(t; ty; ty B. M5)
(4)
0. if r(b:] < r[by]
Text+Box+Image We use LayoutLMv2 joint encode the text, image and 2D-position within the bounding boxes as follows:
p = (bb))b))
(I)
1. if r(b.] > r[b,]
where b, and by denote the i-th and j-th bounding boxes, respectively, [bx refers the index ID of the bounding box in the predicted reading se- quence, and [bry] < r[b;)] indicates that b; appears before by in the reading order sequence.
b; : by Lvy0ut tyz Me, M5; I, I1
(5)
where we consider the ROIs of document images, denoted as I and I,. We take the first latent en- coding to represent each bounding box.
Box. In this model, we use only two-dimensional

---

WEAK
STRUCTURED
INFOGRAPH
OVERALL
Modality
T
p
pt
T
P
T
Box
0.4521
0.4731
0.7017
0.7411
0.6761
0.7413
0.5992
0.6366
TEXT
0.5369
0.5740
0.8965
0,9717
0,9046
0,9171
0.7589
0.8052
TEXT+BoX
0.5316
0.5749
0.9269
0.9766
0.9717
0.9977
0.7837
0.8256
TEXT+BOX+IMACE
0.6293
0.6893
0.8930
0.9690
0.9575
0.9954
0.8052
0.8665
MISSING RATE
38.16%
12.98%
9.55%
Table Correlation coefficient scores of Kendall's tau and Spearman's rank p between resorted input sequences generated by using different preordering methods and golden human eye movement orders. Bold indicates the best and underline indicates the second best.
Algorithm Moddelbbaaae based Sequenece Preordering
actly what reading order is needed for a machine document intelligence model, i.e., what input or- der enhances machine document comprehension. make of extrinsic evaluation assess the quality of temporal order on human-like reading order generation.
Input: the serialized input sequence [bı,-..,00 that contains multimodal features;
Output: the sorted input sequence r;
bi...., bi]:
for t 0 to do
for j 0 to do
p
f(r, r-+1); Calling the atomic comparison model to determine the precedence order.
6.1 Intrinsic Evaluation
if p < 0.5 then
Specifically, compare machine-generated in- put orders with ground truths, i.e., human read- ing orders, to evaluate the model's performance in the intermediate task reading order gener- ation. We measure the correlation between the machine-generated input order and the reading or- der of human experts by calculating Spearman's 's rank and Kendall's tau scores to assess the accu- racy of the machine-generatee reading order and the interaction between different modalities and different types of documents.
swap Ij with rý-13 > Consistent with Bubble sort.
return r, New sorted sequence.
5.2.3 Sequence Preordering Algorithm
We test four atomic comparison models within the model-based sequence preordering algorithm before-and-after judgment. The preordering algo- rithm takes as input the atomic comparison models outputs (0/1 sequence) to construct an adjacent ma- trix. Thus, the preordering algorithm is variant of the commonly used Bubble Sort algorithm, which outputs the sorted input sequence by rearranging the bounding box positions in the sequence. Refer Algorithm 1 for more details.
Table shows the sequence correlation eval- uation results of four different models on three datasets. Among them, the average result of the model based on Text+Box+Image is the best, and the result based on the bounding box is the worst, which also verifies multimodal features have an important impact on document ranking. The re- sults in the above table show that humans read VRDs with weak table structure, and humans still use visual features such as font background to help model the temporal sequence, and visual features are less important in the case of table structure. and basic text and position enough. However, infographics with more images and visual features require more powerful models that can capture text, location, and visual features. For more de- tails see Appendix A.3.
6 Experiments and Analysis
To gain a deeper understanding of the differences between human reading order and machine pro- cessing order, we conduct both intrinsic and ex- trinsic evaluations using the DOCTRACK dataset. By comparing different modal fusions, the intrin- sic evaluation allows the reading order generation model, which in turn helps us to understand the extent to which infor- mation from each modality influences the reading order built by humans. In order to determine ex-
assess the quality of

---

MODALITY
TYPE
WEAK
STRUCTURED
INFOGRAPH
PREORDER
Text
PPs
Img
R/H/
P
R (98))
FI (%8)
(%6)
(
(86)
ANLS ($)
EYE
57.75
60.23
58.70
57.63
59.13
58.37
4.01
EYE+
60.52
60.77
60.47
60.75
60.83
60.79
4.65
DEFAULT-OCR
56.69
62.11
60.33
58.99
60.01
59.51
3.82
BERT
Z-ORDER
R
64.09
65.28
64.66
63.44
62.78
63.11
5.88
XYLAYOUT
60.16
60.84
60.19
59.24
60.08
59.65
3.71
MODEL-B
60.19
61.98
60.92
59.01
60.98
59.98
2.94
MODEL-T
62.80
63.16
62.87
61.04
60.22
60.62
2.99
MODEL-T+
M
61,30
62.43
61.80
60.52
60.77
60.47
3.14
MODEL-T+B+1
62.74
64.51
63.45
62.43
62.17
62.80
3.21
82.13
85.11
83.58
77.84
74.14
75.94
14.43
87.38
83.82
77.44
74.39
75.88
15.69
LAYOUTLM22
DEPAULT-OCR
R
86.94
80.95
83.44
78.56
73.02
75.69
12.50
Z-ORDER
88.00
84,46
86.06
78.05
74.77
76.37
18.09
84.01
78.41
76.61
12.38
MODEL-B
87.23
85.16
86.13
78.45
73.01
75.63
12.98
MODEL-T
85,35
82.40
83.77
77.64
77.14
77.39
12.67
MODEL-T+B
86.76
86.61
86.57
80.24
74.84
77.45
14.70
MODEL-T+B+1
87.00
87.01
86.98
79.74
75.58
77.60
14.70
63.82
65.98
EYE++
91.47
91.19
91.33
69.22
65.57
67.35
18. 84.64
LAYOUT
DEPAULT-OCR
90,96
92.00
91.48
72.96
66.71
69.70
18.22
94.63
77.27
68.24
72.47
20.58
XYLAYOUT
90.10
89.69
89.90
71.05
66.73
68.82
17.63
MODEL-B
M
92.37
91,95
92.16
75.81
70.00
72.79
17.76
MODEL-T
91.86
93.09
92.47
74.01
67.56
70.64
17.52
MODEL-T+B
92.39
92.85
77.17
70.98
73.94
18.01
MODEL-T+BB.
93.98
93.14
93.56
74.17
68.97
71.47
18.24
Table 3: Extrinsic evaluation on the DOCTRACK dataset. DEFAULT-OCK represents the original order, EYE and BYE++ represent the original eye movement order and the smoothed eye movement order, Z-ORDER (Yu al., 2023b) and XYLAYOUT (Gu et al., 2022) are two orders generated by expert experience, MODEL-B, MODEL- T, MODEL-T+B and MODEL-T+B+ represent atomic comparison models such as Box, Text, Text+Box, and Bext=Box+ respectively. R/H/M refers to the order generated rules, humans, and models.
6.2 Extrinsic Evaluation
DQA. Document Question Answering (DQA) is a challenging task VRD understanding that re- quires machines to understand both the visual and textual content of a document image and to an- swer questions about it. Table 3 lists the Aver- age Normalized Levenshtein Similarity (ANLS) scores the INFORGRAPH subset of text-only baseline BERT, layout-aware multimodal base- lines LayoutLMV2 and LayoutLMv3. We observe that LayoutLMM2 and LayoutLMv3 models out- perform text-only baselines (BERT) by a large margin. While integrating human reading order enhances the state-of-the document Al model in downstream tasks, it does not always outper- form the human-like reading order generated ing rule-based approach. This suggests that true human reading order may not be necessary to enhance existing machine document models.
Semantic entity recognition. The purpose of this study is to analyze the impact of reading or- der on the VRD understanding using the Seman- tic Entity Recognition (SER) task for the WFAK and STRUCTURED subset of documents. Table shows the results. We find that the impact of read- ing order on the SER task varies depending on document AI model used. When using BERT, the simple Z-order works best, and the effect of each order is better than the effect of the original order. In the multimodal Layour and LayoutL.Mv3 models, multimodal ordering works best, slightly better than Z-order. These results suggest that human reading order and machine ordering order have a strong influence on the SER task, and dif- ferent models have different degrees of sensitivity to these orders.

---

There are several possible reasons for this. First, most of the datasets used by existing document AI models are sorted by simple rules and therefore are better suited for the orders generated by using simple rules such as Z-pattern. Additionally, in- dividual human reading orders may be very noisy unless large human eye-movement dataset is con- structed by collecting a significant amount of hu- man eye-movement data.
Ethics Statement
This study has received institutional cthics ap- proval and complies with the Declaration of Helsinki and subsequent revisions. Informed con- sent obtained from all participants before the study began, and they were informed that they had the right withdraw from the study at any time. Personal identities are removed from the data to ensure anonymity. The participants have also ap- proved eye-tracking data release for research pur- poses.
7 Conclusion
We investigate the impact of human reading order on Document Al models for VRD understanding tasks. We propose different methods to generate human-like reading orders, along with a practical preordering pipeline that can leverage the gener- ated reading orders. Our observations suggest that true human reading order may not always be suit- able for reading VRDs. The dataset we construct can help in designing better document Al models and human reading robots in the future.
References
Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Sógaard. 2018. Sequence classification with human attention. In Proceedings of the 22nd Conference Language Learning, CoNLL 2018, Brussels, Bel- gium. October 31 November 1, 2018, pages 302- 312. Association for Computational Linguistics.
Computational Natural
Maria Barrett, Joachim Bingel, Frank Keller, and An- ders Søgaard. 2016. Weakly supervised part-of- speech tagging using eye-tracking data. In Proceed- ings of the 54th Annual Meeting of the Association for Computational Linguistics, ACI. 2016, August 7- 12, 2016, Berlin, Germany, Volume 2: Short Papers. The Association for Computer Linguistics.
Limitations
In this work, we focus the impact of human eye- tracking order and machine reading ordering for VRD understanding. Due to the complexity of eye movement characteristics, when the participants were doing eye movement experiments, they were required to ignore the eye movements information, such as the fixation time of each fixation point, back gaze, and the number of fixations. There- fore, our next step will be to explore the impact of more eye movement gaze information the inde- pendent understanding of VRDs. In addition, due the high annotation cost, the annotation has not been done by multiple annotators. Therefore, the Imner-agreemet rate is not available for the cur- rent dataset.
Nilavra Bhattacharya, Somnath Rakshit, Jacek Gwiz- dka, and Paul Kogut. 2020. Relevance prediction from eye-movements using semi-interr semi-interpretable con- volutional neural networks. In CHIIR '20: Con- ference Human Information Interaction and Re- trieval, Vancouver, BC, Canada, March 14-18, 2020. pages 223-233. ACM.
Michelangelo Ceci, Margherita Berardi, and Donato Malerba. 2007. Relational data mining and IIP for document image understanding. Appl. Artif. Intell., 117435):317-34.
Michael Collins. Philipp Koehn, and Ivona Kučerová. 2005. Clause restructuring for statistical machine translation. In Proceedings of the 43rd Anmual Meet- ing of the Association for Computational Linguis- tics (ACL '05), pages 531-540, Ann Arbor, Michi- gan, Association for Computational Linguistics.
Acknowledgements
This work was supported by National Natu- ral Science Foundation of China (Young Pro- gram: 62306173, General Program: 62176153),
Xiao Ding, Bowen Chen, Li Du, Bing Qin, and Ting Liu. 2022. CogBert: Cognition-guided pre-trained language models. In Proceedings of the 29th Inter- national Conference Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, Oc- tober 12-17, 2022, pages 3210-3225. International Committee on Computational Linguistics.
JSPS KAKENHI Program (JP23H03454), Shang- hai Sailing Program (21YF1413900), Shanghai Pujiang Program (21PJ (06800,, Shanghai Mu- nicipal Science and Technology Major Project (202102))Z00002,, the Alibaba-AIR Program (22088682), and the Tencent AI Lab Fund (RBFR2023012).
Zhangxuan Gu, Changhua Meng. Ke Wang. Jun Lan, Weiqiang Wang, Ming Gu, and Liqing Zhang. 2022.

---

Xylayoutim: Towards layout-aware multimodal net- works for visually rich document understanding. In IEFF/CVF Conference on Computer Vision and Put- tern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 4573-4582. IEFE
Chenxia Li, Ruoyu Guo, Jun Zhou, Mengtao An. Yun- ing Du, Lingfeng Zhu, Yi Liu, Xiaoguang Hu, and Dianhai Yu. 2022. Pp-structurev2: A stronger docu- ment analysis system. CoRR, abs/2210.0539)
Yulin Li. Yuxi Qian, Yuechen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao. Junyu Han, Jingtuo Liu, and Errui Ding. 2021. Structext: Struc- tured text understanding with multi-modal trans- formers. In MM 21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021, pages 1912 1920. ACM.
JM Henderson and F Ferreira. 1993. Tyee move- ment control during reading: fixation measures reflect foveal but not parafoveal processing diffi- culty. Canadian journal of experimental psychology Revue canadienne de psychologie experimentale, 47(2)2201221.
Xiaojing Liu, Feiyu Gao, Qiong Zhang, and Huasha Zhao. 2019. Graph convolution for multimodal in- formation extraction from visually rich documents. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 2 (Industry Papers), pages 32-39, Minncapo- lis, Minnesota. Association for Computational Lin- guistics.
Nora Hollenstein and Ce Zhang. 2019, Entity recog- nition at first sight: Improving NER with eye move- ment information. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies. NAACL-HLT 2019, Minneapo- lis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 1-10. Association for Compu- tational Linguistics.
Minesh Mathew, Viraj Bagal, Ruben Tito, Dimosthenis Karatzas, Emest Valveny, and C. V. Jawahar. 2022. Infographicvqa. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022, pages 2582- 2591. IEEE
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutlmv3; Pre-training for doe- ument Al with unified text and image masking. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 14, 2022, pages 4083-4091 ACM.
Abhijit Mishra, Diptesh Kanojia, Seema Nagar, Kuntal Dey, and Pushpak Bhattacharyya. 2017. Leverag- ing cognitive features for sentiment analysis. CoRR. abs/1701.05552
Guillaume Jaume, Hazim Kemal Ekenel, and Jean- Philippee Thiran. 2019. FUNSD: A dataset for form understanding in noisy scanned documents. In 2nd International Workshop on Open Services and Tools for Document Analysis. OST@ ICDAR 2019, Sydney, Australia, September 22-25, 2019, pages 1-6. IEEE
Tetsuji Nakagawa. 2015. Efficient top-down BTG pars- ing for machine translation preordering. In Proceed- ings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Interna- tional Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers). pages 208-218, Beijing, China. Association for Computational Lin- guisties.
Varun Khurana, Yaman Kumar, Nora Hollenstein, Ra- jesh Kumar, and Balaji Krishnamurthy, 2023. Syn- thesizing human gaze feedback for improved NLP performance. In Proceedings of the 17th Confer- ence of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 1887-1900. Associa- tion for Computational Linguistics.
Graham Neubig, Taro Watanabe, and Shinsuke Mori. 2012. Inducing a discriminative parser to optimize machine translation reordering. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 843-853, Jeju Is- land, Korea, Association for Computational Linguis- tics.
Sung-Dong Kim. 2009. Syntactic category prediction for improving translation quality in english-korean korean machine tramslation. In Proceedings of the 23rd Pa- cific Asia Conference on Language, Information and Computation, PACLIC 23, Hong Kong. China, De- cember 3-5, 2009, pages 710-717. City University of Hong Kong Press.
K. Rayner. 1998. Eye movements in reading and infor- mation processing: 20 years of research. Psycholog- ical bulletin, 124(3):374222.
Chen-Yu Lee, Chun-Liang Li, Chu Wang, Renshen Wang, Yasuhisa Fujii, Siyang Qin, Ashok Popat, and Tomas Pfister. 2021. ROPE: Reading order equivari- ant positional encoding for graph-based document information extraction. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- une 2: Short Pupers), pages 314-321, Online. As- sociation for Computational Linguistics.
Yuqi Ren and Deyi Xiong. 2021. Cogalign: Learning to align textual neural representations to cognitive language processing signals. In Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/UCNLP 2021, (Volume 1: Long Papers), Vir- tual Event, August 1-6, 2021, pages 3758-3769. As- sociation for Computational Linguisties.

---

Ekta Sood, Simon Tannert, Philipp Müller, and dreas Bulling. 2020. Improving natural language processing tasks with human gaze-guided neural at- tention. In Advances in Neural Information Process- ing Systems 33: Annual Conference on Neural In- formation Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
shop on Innovative Use of NLP for Building Edu- cational Applications, BEA@ EMNLP 2017, Copen- hagen, Denmark, September 8, 2017, pages 121- 132. Association for Computational Linguisties.
Wenwen Yu, Ning Lu, Xianbiao Qi, Ping Gong, and Rong Xiao. 2021. PICK: Processing key infor- mation extraction from documents using improved graph learning-come convolutional networks. In 2020 25th International Conference on Pattern Recogni- tion (ICPR), pages 4363-4370.
Ece Takmaz, Sandro Pezzelle, Lisa Beinborn, and Raquel Fernández. 2020. Generating image descrip- tions via sequential cross-modal alignment guided by human gaze. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4664 4677. Association for Computa- tional Linguistics.
Yuechen Yu, Yulin Li, Chengquan Zhang, Xiaoqiang Zhang. Zengyuan Guo, Xiameng Qin, Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang. 2023a. Structextv2: Masked visual-textual prediction for document image pre-training.
Jiapeng Wang, Chongyu Liu, lianwen Jin, Guozhi Tang, Jiaxin Zhang, Shuaitao Zhang, Qianying Wang. Yaqiang Wu, and Mingxiang Cai. 2021a. To- wards robust visual information extraction in real world: New dataset and novel solution. In Pro- ceedings the AAAI Conference on Artificial Intelli- gence., volume 35, pages 2738-2745, AAAI Press.
Yuechen Yu, Yulin Li, Chengquan Zhang, Xiaoqiang Zhang, Zengyuan Guo, Xiameng Qin. Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang. 20236. Structextv2: Masked visual-textual prediction for document image pre-training. In The Eleventh Inter- national Conference on Learning Representations.
Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, and Furu Wei. 2021b. LayoutReader: Pre-training of text and layout for reading order detection. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 4735- 4744, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Junwei Zhang. Hao Wang, and Xiangfeng Luo. 2022. Dual-VIES Dual-level graph attention network for vi- sual information extraction. In The 19th Pacific- Rim International Conference Artificial Intelligence. pages 422 434.
A
Appendix
Fei Xia and Michael McCord. 2004. Improving statis- tical MT system with automatically learned rewrite patterns. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguis- tics, pages 508-514, Geneva, Switzerland. COL- ING.
A.1 Visualization of Human Reading Order
Figure 6 shows the missing gaze points during hu- man reading and after smoothing.
A.2 Human Reading Patterns
Figure 7 shows four patterns of human reading or- ders.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei A. F. Floréncio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021. Layoutlmy2: Multi-modal pre-training for visually-rich document understanding. In Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 11th In- ternational Joint Conference Natural Language Processing, ACI/JCCNP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 2579-2591. Association for Computational Linguis- tics.
A.3 Missing Gaze Points Visualization
Figure 8 shows the sequence of model generation and the sequence of human eye gaze.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming 7hou. 2020. Layoutlm: Pre- training of text and layout for document image un- derstanding. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing, Virtual Event, CA, USA, August 23-27, 2020, pages 1192-1200, ACM.
Victoria Yaneva, Constantin Orasan, Richard Evans, and Omid Rohamian. 2017. Combining multiple cor- pora for readability assessment for people with cog- nitive disabilities. In Proceedings of the 12th Work-

---

Allewab Grade Subsiittions for Export OPCL
Standerd Groak Morkho
Allowable Sebsiibeiionsy
curd lenott
SPCPs o ReS2
KC44-26
Tew o10
(a) missing gaze points
Food Lamedo
Allowable Grade Suhasbitinn Expert CPCL.
Standand Giodle Markis
Allmweble Sebsifhccone
oxed Spnol
Soree Trasso
RPOB Wr085
BC4815
Tow Rat Wert
2002
ROM RR1 dos
(b) fixed data visualization
Figure 6: Example of a document image with missing gaze points (-1) and fixed data in the WEAK subset.

---

METTSA, IUTA OAE sulme
(a) normal Z order
(b) local priority order.
COVID 9: High Mortality Pates Linked To Care
of soñaaee costs pad from each funting sage
Tro to cose
Parcht
Styrkott oame
gan svi
Norway
12% bor
km
Fro
gueent
gariningg
Austrars
1-2K
More
saving
from 21% 2018-19
statista
(c) cross-modal interaction (pie chart).
(d) cross cross-modal interaction (bar chart).
Australia is Warming Faster tharthe Global ve age
rucc tion
Adult Adteanced life
Annust mean temperature anomaly Au
and the world (in Celciu..
2019
Beanc
50
10
11
civeggence temperature
modsl interaction (line chart).
(f) visual instruction order.
Figure 7: A scatterplot showing four patterns of human eye movement while reading.

---

Share bollege bosta paid fromleesh funding Sourcee
44%
8% Stukeen Thetin wrk Sevinge
Paront hocme
13% bo rowiing
25%
Schollashio ahiis
8%
1%
Poitant borrtwving
Relilves
Mivke lonechird 37%) lartial udid Bloofioll Eavi See
E28 from 221 M MY 2019:12
(a) Sequence diagram of human eye movements while reading
(b) Scatterplot of human eye movement when reading.
Share of wwo.oolllleocooooooooooo coets paid from esch funding source
44% PaRkntiosAaa whidsarhha
8% nocmm
65 3 savingg
13% Studlrt bondwwing
28%
ships
8% Falbot bonrSwing
1%
Reldvves
Shird Affforlleed loollll Eavingelanshum [Rag bogs upfinim 20% in Rr12018-19
(c) Sequence diagram of reading order generated by MODEL-BBT+h
(d) Scatterplot of reading order generated by MODEL-
B+T+I.
Figure Case comparison human reading order and reading order generated by MODEL-B+T+I.


Reviews:
[{"review_id": "GZ1dUPJTLT", "reviewer": "Reviewer_ku22", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are delighted that the reviewer has accurately  understood our motivation and highlighted the significance and necessity of creating a dataset for human reading order on visually-rich documents, utilizing eye-tracking techniques.\n\n***Response-Q1 \"The sample size ... is small\".***:\n\n- **The experimental process is time-consuming and laborious**. Each time the experiment is conducted, we need to immobilize the participant's body and perform the experiment in a closed environment. The long time labeling process can be challenging for regular individuals to tolerate, resulting in many volunteers opting to quit after only a few labeling sessions. Consequently, the authors had to undertake the data labeling themselves.\n- Unlike other NLP task labeling, **the variance in labeling results among different subjects in the eye-tracking experiment is substantial**, making it nearly impossible to achieve consistent results across two subjects. The labeling outcomes vary significantly among individuals, and having too many participants can adversely impact the labeling results, making it difficult for the machine to learn effectively.\n- **Despite its small size, we appreciate your understanding of the importance of our dataset.** We would like to clarify that our work concentrates on human reading order and its potential positive impact on developing machine document understanding models. We hope that reviewers will give due consideration to our efforts, as the dataset, though small, serves as a crucial foundation for the entire project. The dataset's significance lies not only in its detailed construction process but also in the numerous valuable attempts and intriguing findings derived from it. We believe that these aspects make our paper particularly meaningful to the community and appealing to readers.\n\n\n***Response-Q2: Typos***\n\nWe will correct them.\n\n***Response-Q3: VQA->DQA***\n\nAgree, we will change VQA in the subtitle of Section 6.2 to document question answering.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- An eye tracking aligned dataset for VRD to understand how humans and machines attend to reading", "reasons_to_reject": "The sample size of human participants is five, a small number", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a dataset for multimodal reading comprehension aligned with eye-tracking data. Specifically, the authors investigated \nthe connection between human reading order and how it impacts the comprehension of visually-rich documents (VRD).", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typo\nBuddle --> Bubble sort on page 6, Section 5.2.3 \n\nSection 6.2\nVQA: A more appropriate subheading will be visual document question understanding [1] or document question answering [2]\n\nReferences\n1. https://rrc.cvc.uab.es/?ch=17&com=introduction\n2. https://huggingface.co/tasks/document-question-answering\n"}}, {"review_id": "eyGMYYC1UC", "reviewer": "Reviewer_2oER", "summary": "", "questions": "1. What is the OCR engine used in the paper? \n2. Is the reading order phrase-level? This means the eye-tracking technique is used to sort different phrases instead of words. If so, there is a question, how can we guarantee the quality of the word grouping in the OCR engine. \n3. Why can the human's reading order benefit the machine to understand the document. As in figure 1, the key value pair of author field is actually reversed.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments!\n\n***Response-Q1: Annotation Agreement***\n\nIn our experimental setup, we assign two out of the five participants to label the same subset of data. Subsequently, for each document file, we compare the labeling results from these two participants. We then conduct a voting process among the five participants to select the most appropriate labeling（document-level）that aligns with everyone's expectations. This chosen labeling is ultimately considered as the final data.\n\n***Response-Q2: metrics***\n\nWe are using Kendall's Tau and Spearman's rank correlation coefficient. We will add the details into the Appendix.\n\n- Kendall's Tau is a statistical method used to measure the correlation between two variables. It is mainly used to assess the degree of correlation between the ordering or rank of two variables.The value of Kendall's Tau ranges from -1 to 1, where:1 indicates perfect positive correlation,0 indicates irrelevant,-1 indicates perfect negative correlation.The formula for the Kendall's Tau rank correlation coefficient is as follows:\n\n$$\n\\tau=s^{\\frac{1}{2}}\\sqrt{\\frac{(n_1-1)(n_1-2)}{(n_1-1)n_2+(n_2-1)(n_2-2)}}\n$$\n\nwhere $\\tau$ is the Kendall correlation coefficient, $s$ is the sample standard deviation, and $n_1$ and $n_2$ are the frequencies of the two variables respectively,i.e. the number of occurrences of each value taken.\n\n- Spearman's rank correlation coefficient is a nonparametric statistical measure of the degree of association between two sets of rank data. It calculates the correlation based on the rank of the original data (rather than the original data values). The formula for the Spearman rank correlation coefficient is as follows:\n\n$$\n\\rho=1-6\\frac{d^2_1}{n}-6\\frac{d^2_2}{n}+6\\frac{d_1d_2}{n}\n$$\n\nwhere $\\rho$ is the Spearman correlation  coefficient,$d_1$ and $d_2$ are the rank data for each of the two variables, and $n$ is the sample size.\n\n***Response-Q2: OCR engine used***\n\nWe use Tesseract and Google Vision OCR engines.  \n\n***Response-Q4: phrase-level order***\n\nThe reading order in our study is determined at the segment level, which can alternatively be referred to as the phrase-level. As we only modify the segment-level order of the output generated by the OCR engine, there is no need to be concerned about word grouping, as it has already been addressed. Consequently, our sorted results are guaranteed to maintain consistency with the OCR outcomes.\n\n***Response-Q5: sometimes reading order reversed***\n\nThe reviewer raises an important point. When it comes to human reading, it is indeed inevitable that occasional reversals may occur; however, these instances are typically isolated, and the overall reading process remains orderly. On the other hand, for machines, the optimal reading order for humans may not necessarily be the most efficient or effective approach. It is indeed intriguing to consider that the reading order of documents encountered during LMM pre-training tends to follow a norm order resembling a Z-shape. This particular order appears to be more compatible and user-friendly for these LMMs, enhancing their overall performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1 . The reading order is a big issue for visually-rich document understanding. A wrong reading order could make it impossible for simple LMs to get the right results. In this paper, the eye tracking is a strategy to get the ordering information. \n2. The experiments show that the new ordering information surpass existing rule-based ordering techniques.", "reasons_to_reject": "A few important points are missing in the paper:\n1. During the annotation, did the authors compare the agreement between different annotators? \n2. It is also great to know more about the metric method in the table 2.\n3. The OCR is part of the preprocessing and the paper didn't mention which OCR engine they are using. Some OCR engines already integrate the reading order in their algorithms so the reading order result can be much better than what is shown in figure 1.\n4. It is not intuitive to ask machines to understand the document in the same order as humans. As shown in figure 1, in the human's reading order, the author's key and values are reversed (label 6 and 7)", "questions_to_authors": "1. What is the OCR engine used in the paper? \n2. Is the reading order phrase-level? This means the eye-tracking technique is used to sort different phrases instead of words. If so, there is a question, how can we guarantee the quality of the word grouping in the OCR engine. \n3. Why can the human's reading order benefit the machine to understand the document. As in figure 1, the key value pair of author field is actually reversed.", "paper_topic_and_main_contributions": "This paper proposes to improve the visually-rich document understanding task through the reading order. The new ordering comes from the eye tracking techniques and this new order makes it possible for the language model to understand the documents in a similar way as real human. Experiments show that the current layout-aware LMs can detect the new reading order and the performance is improved with the help of this new ordering, compared with other rule-based ordering techniques.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "kxg4QMwJSe", "reviewer": "Reviewer_hWn3", "summary": "", "questions": "1. Can WEAK conforming to the normal-Z pattern be regarded as the easiest part of the dataset and InfoGraphics the most difficult one requiring more complicated modeling? If so, models can better fit into WEAK preordering when doing intrinsic evaluations. However, from Tab. 2, correlation coefficient scores on STRUCTURED and INFOGRAPHICS (especially INFOGRAPHICS) are much higher than WEAK. Why would this happen? Is there any further explanation? \n2. For extrinsic evaluations, why do EYE and EYE++ show such different behaviors to Model-T and Model-T+B on STRUCTURED and INFOGRAPH as Model-T(T+B) has achieved extremely high correlations with labels in the preordering? \n3. How often does the missing gaze point be fixed by the strategies in 4.3? How to measure and guarantee the accuracy of the recording with these strategies? Some measurements on a small subset can be more convincing. Is there any possibility that the dataset can be affected by these strategies so that WEAK is harder to fit? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "At the suggested by the reviewers, we conducted further experiments. We discovered that the rate of missing gaze points significantly impacts the results of both intrinsic evaluation and extrinsic experiments, and may be the primary cause of different behaviors. \n\n***Response-Q1: The models did not yield the expected performance on the WEAK subset preordering during intrinsic evaluations.*** \n\nThis issue indeed requires clarification.\nsome explanation can be found in ***Response-Q3***. \n\n\n***Response-Q2:EYE/EYE++ different behaviors to Model-T/Model-T+B***: \n\nWe think the explanation for this question can also be found in ***Response-Q3***.  \n \n***Response-Q3:missing-gaze-point-rate***\n\nWe greatly appreciate this question, and we must admit that we had not observed this phenomenon previously!\n\nFor missing gaze points (in **eye**), we use -1 as a placeholder. Given a document order of [1, -1, 2, 5, -1, 6], we determine the current order of missing points based on the points before and after them. We then insert the missing points into the gaze point sequence behind them according to their coordinates, resulting in [1, 2, 2, 5, 5, 6] (in **eye++**). This approach makes the document order easier to fit rather than more difficult.\n\nFollowing the reviewer's advice, we calculated the ratio of missing gaze points for each specific subset by determining the proportion of -1 points relative to the total, see result below. \n\n|                  |  WEAK  | STRUCTURED | INFOGRAPH |\n| ---------------- | :----: | :--------: | :-------: |\n| missing rate | **38.16%** |   12.98%   |   9.55%   |\n\n\nAfter an extensive discussion, we endeavored to provide explanations to the best of our knowledge as follows:\n\n1) Due to the involuntary fluttering of human eyes while reading, the gaze point may deviate from the intended text area. \n\n2) The text segments in the document image are dispersed across the page, resulting in numerous missing points and a high rate of omissions.\n\n3) The document layout within the WEAK subset appears simplistic and seemingly arbitrary, leading to a significant discrepancy between the order comprehended by machines (z) and humans (arbitrary).\n\n4) We also inquired with specific participants, who reported that working on the WEAK dataset was not particularly laborious compared to other two subsets; however, they experienced a heightened level of mental concentration when handling the other two subsets.\n\nTherefore, we believe that the combination of all the aforementioned factors has led to the emergence of questions 1, 2, and 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* A new resource for real human eye movements aligned in the document reading modeling. \n* A good point of view to find the discrepancy between real human behavior and the current models' modeling based on rules. \n* A work with good writing can provide a comfortable reading experience. ", "reasons_to_reject": "* The appeared results of some experiments contradict the dataset's setup and assumptions. No further analyses or experiments are adapted to explain them. See Questions For The Authors. \n* More details about the dataset construction are required. ", "questions_to_authors": "1. Can WEAK conforming to the normal-Z pattern be regarded as the easiest part of the dataset and InfoGraphics the most difficult one requiring more complicated modeling? If so, models can better fit into WEAK preordering when doing intrinsic evaluations. However, from Tab. 2, correlation coefficient scores on STRUCTURED and INFOGRAPHICS (especially INFOGRAPHICS) are much higher than WEAK. Why would this happen? Is there any further explanation? \n2. For extrinsic evaluations, why do EYE and EYE++ show such different behaviors to Model-T and Model-T+B on STRUCTURED and INFOGRAPH as Model-T(T+B) has achieved extremely high correlations with labels in the preordering? \n3. How often does the missing gaze point be fixed by the strategies in 4.3? How to measure and guarantee the accuracy of the recording with these strategies? Some measurements on a small subset can be more convincing. Is there any possibility that the dataset can be affected by these strategies so that WEAK is harder to fit? ", "paper_topic_and_main_contributions": "This work towards document reading aims to align the eye movement during a human's actual reading with existing resources and models, revealing the findings that natural human eye movement modeling is beyond the current VRD models' abilities to make the process more human-like, where these models still depend highly on rules and other assumptions. \n\nThis work constructs a real human eye movement dataset for further research of document AI systems. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "Gaza -> Gaze ? in the appendix. "}, "extra_text": {}}, {"review_id": "U42saC0Jee", "reviewer": "Reviewer_sdSf", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Response-Q1***: It's great to see that the reviewer raise the issue of considering the importance of validation and development sets. Taking into account the limited number of annotations available, currently, there is no designated validation or development set. \n\nNevertheless, validation sets in machine learning can be employed for cross-validation, such as by allocating a portion of the training set to serve as a validation set. \n\nAs for additional dev/valid set, we appreciate this valuable suggestion. We will contemplate further annotating data to establish an additional validation set and release it in the not-too-distant future.\n\n***Response-Q2***:  We have committed to openly sharing all preordering models, data sets, and naturally, baseline systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n\n\n\n1. The paper introduces DOC TRACK, a first-of-its-kind dataset that aligns VRDs with human eye movement data. This unique benchmark can pave the way for new research in the field of VRD reading order generation.\n2. The proposed preordering pipeline, which incorporates human-like reading orders into AI models, is an interesting and practical solution to improve VRD understanding.\n3. The authors conduct both intrinsic and extrinsic evaluations, providing a thorough analysis of the impact of human-like reading orders on VRD comprehension tasks.\n\n\n", "reasons_to_reject": "\n\n1. According to Table 1, the dataset does not have a valid/dev set, which may add some difficulties in tuning the system.\n2. It would be better to open-source the baseline systems to further facillitate the community.\n\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The research paper addresses a key challenge in the field of AI - understanding Visually-Rich Documents (VRDs). VRDs are documents with substantial visual components such as tables, graphs, diagrams etc., which are increasingly common due to advancements in information technology. Document AI models are employed to comprehend these VRDs, but their understanding is often limited by the inability to accurately follow human-like reading order and their struggle to comprehend the complex interplay of text, visuals, and layout. To address these limitations, the authors introduce DOC TRACK, a unique VRD dataset that incorporates human eye-movement information gathered using eye-tracking technology. The dataset aims to explore how human reading patterns might influence document comprehension tasks, presenting a comparison between the human reading order and how AI models read and comprehend documents.\n\nThe paper makes three main contributions:\n\n1. It presents DOC TRACK, the first-of-its-kind dataset, aligned with human eye movement data. This benchmark dataset is expected to facilitate research on reading order generation in VRDs, thereby enhancing our understanding of how humans and machines differ in their approach to reading and interpreting VRDs.\n\n2. The authors explore various methods to generate human-like reading orders and suggest a practical preordering pipeline. This pipeline integrates human reading order into the AI model workflow, which has the potential to improve the models' document understanding capabilities.\n\n3. Lastly, the authors conduct a detailed evaluation of the impact of human-like reading order on AI models' performance in document understanding tasks. These evaluations help determine the efficacy of human reading order when used by AI in understanding VRDs, which may lead to more refined AI models in the future. Their findings indicate that despite improvements in Document AI models, they still have substantial progress to make before they can accurately and effectively understand VRDs like humans do.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a foundational dataset that combines machine document comprehension with human eye movement data for visually-rich documents. Most reviewers have expressed considerable enthusiasm for this novel dataset, recognizing its significance in enhancing the comprehension of visually-rich documents, a notion substantiated by the performance improvements showcased in the paper's experiments.

However, the reviewers have raised pertinent issues that we encourage the authors to address:

1. The experimental procedure is notably time-intensive and demanding in terms of labor. It would be beneficial for the authors to explicitly acknowledge this as a limitation of their paper. Moreover, providing insights and recommendations for future research and researchers embarking on similar endeavors could be a valuable addition.

2. The issue of agreement among annotators warrants attention. While the paper elucidates the methodology employed to determine the final annotations, it is equally important to declare the level of agreement among the annotators. For instance, when instructing annotators to rate a sample on a scale of 1 to 5, it should be noted whether the average score of 3 results from all annotators assigning a score of 3 or if it arises from a split where half the annotators rate it as 5 and the other half as 1. Reporting such discrepancies in annotators' judgments is crucial.

3. To address the queries posed by the reviewers, it is advisable to consider incorporating additional experiments into the paper, as suggested by reviewer hWn3. Expanding the experimental scope could help provide comprehensive answers to these questions and further bolster the paper's contributions.
Decision: Accept-Findings