GenKIE: Robust Generative Multimodal Document Key Information Extraction
Authors: Panfeng Cao
Ye Wang
Qiang Zhang
Zaiqiao Meng
Venue: EMNLP 2023 Findings (2023)
Keywords: Key information extraction
Multimodal generative model

Abstract: Key information extraction (KIE) from scanned documents has gained increasing attention because of its applications in various domains. Although promising results have been achieved by some recent KIE approaches, they are usually built based on discriminative models, which lack the ability to handle optical character recognition (OCR) errors and require laborious token-level labeling. In this paper, we propose a novel generative end-to-end model, named GenKIE, to address the KIE task. GenKIE is a sequence-to-sequence multimodal generative model that utilizes multimodal encoders to embed visual, layout and textual features and a decoder to generate the desired output. Well-designed prompts are leveraged to incorporate the label semantics as the weakly supervised signals and entice the generation of the key information. One notable advantage of the generative model is that it enables automatic correction of OCR errors. Besides, token-level granular annotation is not required. Extensive experiments on multiple public real-world datasets show that GenKIE effectively generalizes over different types of documents and achieves state-of-the-art results. Our experiments also validate the model's robustness against OCR errors, making GenKIE highly applicable in real-world scenarios.

The key information extraction (KIE) task aims to extract structured entity information (e.g. key- value pairs) from scanned documents such as re- ceipts (Huang al., 2019), forms (Jaume al., 2019), financial reports (Stanislawek et al., 2021), etc. This task is critical to many document under- standing applications, such as information retrieval and text mining (Jiang, 2012), where KIE frees the GenKIE: Robust Generative Multimodal Document Key Information Extraction
Panfeng Cao*, Ye Wang Qiang Zhang Zaiqiao Meng*
University of Michigan
National University of Defense Technology
Zhejiang University
School of Computing Science, University of Glasgow uumiggccmmmmeeo umi ch edu, wangyel Senyyel@@nndt...m edu. cn, qiang.zzarn zhang. cs@ z ju. edu. cn, zaiqiao www..mmeeggggg...o.om. ac uk
panfengel
Abstract
Discrimintion:: poedec taps sin sequenco classi fratimm
Irput Imago
veoe your 500x Ta deD
Juan SAGil
Baseline
tan woon yarn
Key information extraction (KTE) from scanned documents has gained increasing attention be- cause of its applications in various domains. Al- though promising results have been achieved by some recent KIE approaches, they are usually built based on discriminative models, which lack the ability to handle optical character recognition (OCR) errors and require laborious token-level labelling. this paper. we propose a novel generative end-to-end model, named GenKIE, to address the KIE task. GenKIE is a soquence-to-sqquemee ative model that utilizes multimodal encoders embed visual, layout and textual features and decoder to generate the desired output. Well-designed prompts are leveraged to incor- porate the label semantics as the weakly su- pervised signals and entice the generation of the key information. One notable advantage of the generative model is that it enables au- tomatic correction of OCR errors. Besides, token-level granular annotation is not required. Extensive experiments multiple public real- world datasets show that GenKIE effectively generalizes over different types of documents and achieves state-of off-the-art results. Our ex- periments also validate the model's robustness against OCR errors, making GenKIE highly applicable in real-world scenarios
muspiia Caneiiie the Coll
Compeny
BOOK TA.K. HHD
Aikrns
37./0
OCR
Gonneativen ando-sepennivell gerarata tokeans dia prompte
compasy BOOK
BHD
GeniE
aridoos N0 NI ANNAI
RooK TA
10 35-35,37
SAZA.
Prompt company s?? adensss in?
Compeny
BOOKTA BHD
Extracted Text
Addinesx
$4.0)
Figure I: An example of document KIE, Two entity types (i.e. company and address) are extracted from the document image. The baseline model (Xu et al., 2021) predicts correct labels, but the extracted entity values are still wrong due to the OCR errors (marked in red). GenKIE auto-corrects OCR errors and generates the correct entity values. The multimodal features can be used to distinguish ambiguous entity types (e.g. term 'TAMAN DAYAN' in the red circles of the input image).
multimodal gener-
business from manually processing great number of documents and saves significant amount of time and labour resource (Huang et al., 2019).
The KIE task is often tackled by a pipeline of approaches (Huang et al., 2019), including opti- cal character recognition (OCR) and named en- tity recognition (NER). The OCR technique, ex- emplified by Tesseract?, is employed to discern textual and layout attributes from the input images, namely the scanned documents. The NER model (Mohit, 2014) is used discriminatively extract salient details from the derived texts, such as pin- pointing specific entities from the text and layout features based on the annotated beginning-inside. outside (BIO) tags via a sequence classification strategy. A plethora of methodologies adhering to this framework have emerged, leveraging synergy of multimodal features textual, visual, and layout data-from the document image. Notably, contem- porary KIE models like StrucText (Li et al., 2021), ) BROS (Hong et al., 2020), and LayoutLMv2 (Xu et al., 2021) exhibit commendable results through hnttps:/ttesseract -ocr wit.hb..tbboo.
1 Introduction
*Corresponding author. OOur code and pretrained model are publicly available at ht https:/ / // thub.com// com/G asgow-AM AT4 oMed/ GenKIE.

---

the use of multimodal pretrained frameworks. However, one limitation of these models that they highly rely on the OCR model to extract texts from scanned documents and inevitably suffer from OCR errors. These OCR errors in texts will eventu- ally render wrong entity information. As shown in Figure I, although the classification- based model tags the entities correctly, the result is still wrong due to the OCR errors. Another limitation is that the semantic ambiguity in the document is hard to be captured by the existing models. For exam- ple, as shown in Figure 1, two image patches of the same text (TAMAN DAYAN) in the receipt have different entity types. It is difficult for some exist- ing approaches to improve performance by using only textual information. Other signals such as lay- out and visual information play critical role in identifying the correct entity type (Xu et al., 2021). Therefore effective incorporation of multimodal features is indispensable to improve the model per- formance for the KIE task.
We propose effective prompts that can adapt different datasets and multimodal feature embedding that deals with semantic ambigu- ity in documents. Our model generalizes on unseen documents with complex layouts.
Extensive experiments on real-world KIE datasets show that GenKIE has strong robust- ness against OCR errors, which makes more applicable for practical scenarios compared to classification basdd models.
2
Related Works
2.1 Conventional KIE Methods
Traditional document KIE methods (Dengel and Klein, 2002; Schuster et al., 2013) depend on prede- fined templates or rules to extract key information from scanned documents. Due to the extensive manual effort and specialized knowledge required to design these specific templates for each entity type, these approaches are not suitable for effec- tively managing unstructured and intricate docu- ment layouts. Later KIE systems formalize the problem as an NER task and start to employ pow- erful machine learning models. For example, the BiLSTM-CRF model employed by Huang et al. (2015); Lample et al. (2016); Ma and Hovy (2016); Chiu and Nichols (2016) decodes the chain of entity BIO tags from either textual or textual and visual features. Katti et al. (2018) proposes an image- based convolutional encoder-decoder framework that can encode the semantic contents. The graph- based LSTM utilized by Peng et al. (2017); Song et al. (2018) 8) allows varied number of incoming edges in a memory cell to learn cross-sentence en- tity and relation extraction. While those models are effective in their domains, they do not make use of all multimodal features available in the documents and thus could not solve semantic ambiguity and generalize. Therefore, recent research emphasizes more on the incorporation of multimodal features to generalize on documents with varied and compli- cated layouts. For example, PICK (Yu et al., 2021) models document input graph, where the text and visual segments are encoded nodes and spa- tial relations are encoded edges. However, due to the lack of pretraining on a large corpus, those methods are relatively limited terms of robust- ness and generalization ability.
To cope with the mentioned problems, we propose GenKIE, a robust multimodal genera- tive model for document KIE. GenKIE utilizes the encoder-decoder Transformer (Vaswani al.. 2017) as the backhone and formulates docu- ment KIE as aequence-to-ssqqunnee generation task. The encoder effectively incorporates mul- timodal features to handle semantic ambiguity and the decoder generates the desired output auto- regressively following the carefully designed tem- plate (i.e. prompt) and auto-correcting OCR errors. An example is shown in Figure demonstrate the prompting technique, the label company in the prompt guides the model to generate words that cor- respond to company names. Other labels such as address serve as additional label semantic signals and provide shared context about the task. Thanks to the generation capability, GenKIE does not need the laborious granular token-level labelling that is usually required by discriminative models.
Extensive experiments demonstrate that our pro- posed GenKIE model has not only achieved perfor- mance levels comparable to state-of-thu -otth--r--h--ttt (SOTA) models across three public KIB datasets but also exhibits enhanced robustness against OCR errors. Our contributions are summarized as follows:
We propose GenKIE,, a novel multimodal gen- erative model for the KTE task that can gener- ate entity information from the scanned docu- ments auto-regressively based on prompts.

---

2.2 Multimodal-basel KIE Methods
and decodes the key information with Bart (Lewis et al., 2020)-like decoder. However, the limitation of Swin Transformer to capture the local character patterns might lead to sub-optimal KIE results.
Multimodalit--baseee transformer encoder mod- cls, which are pretrained on large-scale linguistic datasets, show strong feature representation and achieve SOTA performance in downstream KIF tasks. L eX al., 2020) first proposes the pretraining framework to handle document infor- mation extraction. Textual and layout features are jointly utilized in pretraining and visual features embedded in finetuning. LayoutLMM22 (Xu al., 2021) further improves MayoutLM by incorporat- ing visual features in pretraining. LayoutLMMy3 (Huang ct al., 2022) introduces a new word-patch alignment objective in pretraining, which recon- structs the masked image tokens from surrounding text and image tokens. DocFormer (Appalaraju et al., 2021) designs a multimodal self-attention layer to facilitate multimodal feature interaction. BROS (Hong al., 2020) is also an encoder-based model that utilizes graph-bash based SPADE (Hwang al. 2021) classifier to predict both entity tags and entity relations. Layout features are specifically ex- ploited to solve the reading order serialization issue. Similar to BROS, LAMBERT (Garncarek et al., 2021) augments the input textual features with lay- out features to train layout-aware language model. SexucTText (Li al., 2021) and StrucTexxTvv2 (Yu al., 2023) exploit the structured information from the document image and use it to aid entity informa- tion extraction. However, all the above-mentionel models are classification bbsse,,,, which means fine- grained annotations are necessary and they lack the mechanism to auto-correct OCR errors.
Task Formulation
In this work, we address the KIE task which sup- ports many downstream applications such as en- tity labeling (Jaume et al., 2019) and entity extrac- tion (Huang et al., 2019). In particular, given col- lection of scanned document images Ip. the goal of KIE is extract a set of key entity fields (i.e. key-value pairs) for each image / E Ip. where T; is the predefined entity type, Vi is the entity value, and Nr the number of entity fields in image I. The entity types of each image example can be given or not, depend- ing on the downstream applications. For example, the output of the entity extraction task is the entity values of predefined entity types, while the output of the entity labelling task is the extracted entities and their entity types.
{< T, V | $ is Nr)
Methodology
The overall architecture of GenKIE is shown in Fig- ure 2. Unlike the encoder-based models (e.g. Lay- Mv2 (Xu et al., 2021)), GenKIE. is a genera- tive model that uses a multimodal encoder-decoder model, i.e. OFA (Wang et al., 2022a), as the back- bone. The encoder of GenKIE embeds multimodal features (c.g. text, layout and images) from the input and the decoder generates the textual output by following the prompts. Entity information is parsed from the decoder output. the following sections, we illustrate the process of multimodal feature embedding, and then go over the techniques of prompting and inferring.
2.3 Prompt-based Language Models
Recently the paradigm of "pretrain, prompt, and predict" has been prevailing in NLP due to its high adaptability and effectiveness in downstream tasks (Liu et al., 2023). Our proposed work follows this new paradigm by using OFA (Wang et al., 2022a), a multimodal generative vision language model, as the model backbone and leveraging task-specific prompts in finetuning. It is worth mentioning TILT (Powalski et al., 2021), which is also a generative KTE model. The major difference is that GenKIE emphasizes the prompt design and the model's OCR correction capability in practical scenarios, which are not explored in TILT. Donut (Kim et al.. 2022) is an OCR-free model and by design unaf- fected by OCR errors. It encodes the image fea- tures with a Swin Transformer (Liu et al., 2021)
4.1 Encoder
Following the common practice of the KIE task (Xu et al., 2020, 2021), we use an off-the-shelf OCR tool to extract the textual and layout features (i.e. transcripts and bounding boxes of the segments) from the input image. Then bone language encoder (i.e. byte-pair encoding (BPE) (Sennrich al., 2016)), layout encoder and visual encoder (i.e. ResNet (He et al.. 2016)) to obtain embeddings from these features.
back-
4.1.1 Textual Embedding
To process the textual feature, we apply the BPE tokenizer to tokenize the text segment into a sub-

---

2014
Output
Encoder
Decoder
VisualText Embedding
-1
x2
+8
16
pl
ps
20 Positin Embedding
Position Embedding
Prompts
Text/Layut Encoder
Visual Encoder
Template Pronpt
Company is (SEP) Address [SEP) Date [SEP)
Total is (SEP)
Output Company OUI MAITKE IING SENN HHD
ahone
OCR
ISPP) Add NO 2.4.4 JALAN BAYUJ
BANDA- BERI ALAM 01750 MABAJ, JOHOR (SEF)
Date 9230/2019 (BEP) Total 120,0 (BEP)
Input
Question Prompt
Quastion
Answer
Company OUC MARKETING SDN BHD
Address is
NO2 4 JSLAN JSLAN RAYILS
Date 0001-9011
Totel is
120.8
Figure 2: An illustration of GenKIE performing entity extraction from scanned SROIE receipt with the template and QA prompts. The input to the encoder consists of patched visual tokens and textual tokens embedded along with the positional features. Following the prompts, the decoder generates the desired output, which is processed into four entity key-value pairs.
word token sequence and then wrap around the sequence with the start indicator tag [BEG] and the end indicator tag [END]. Then a sequence of prompt tokens is appended after the [SEP] tag. which marks the end of the transcript tokens. Extra [PAD] tokens
work, we use a tuple of normalized coordinates to represent the layout feature of the tokens that bounding box. For instance, the layout fea- ture of i-th bounding box can be represented by by (z), 21 wN, vé ví h2), where (a v6) is the left top coordinate, (af yi) is the bottom right co- ordinate, wy is the width and h; the height of the bounding box, and all textual tokens in the bound- ing box share the same layout feature. Special to- kens such as [BEG], [END] and [SEP] default to have empty feature Demp = (0, O, 0,O, O, 0). If the prompt token Pj refers to the same textual to- ken 4,. then b; is shared with Pj for prompt layout embedding, otherwise P, defaults use beng- The final encoder layout embedding is given by:
appended to the end to unify the sequence length inside the batch. The token sequence S is formulated as:
s [BEG], BPE(T), [SEP], BPE(P)) [END], [PAD],
(1)
where T' represents the transcripts and P represents the prompt. To preserve positional information, we combine the token embedding with the trainable ID positional embedding in element-wise manner to obtain the final textual embedding. Specifically, the i-th textual token embedding T F4 represented as:
LE, CoxcAr(PosEMee2D.ccd. SasarfooEEneDD..a.o..sssP....... PosEMB2D.. PoSEMB2D, v/). (os)MB2D,,(4)) RR.
TE = (MB(S)) + PoSEMB D(i) 6 RR,i i 10, LJ,,
(2)
where CONCAT the concatenation function; POSEMB2D,, POSEMB2D,, POSEMB2D,, and PosEmb2D; are linear transformation to embed spatial features correspondingly following (Xu et al., 2021). The two-axis features concate- nated to form the 2D spatial layout embedding.
where d is the embedding dimension, L is the se- quence length, EMB R + d is the token em- bedding layer shared between encoder and decoder. POsEMR R d is the ID positional em- bedding layer that is not shared.
4.1.2 Layout Embedding
4.1.3 Visual Embedding
GenKIE uses layout embedding layer cap- ture the spatial context of text segments. We first normalize and discretize all bounding box coor- dinates so they fall between [0, 1024). In this
For visual embedding, we first resize the input im- age to 480 480 and then use visual encoder consisting of the first three blocks of ResNet (He et al., 2016) following the common practice of the

---

Prompt Type
Prompts are sequence of text inserted at the end Generation Target
Dataset
Prompt
Example Target
typel (SEP)
typel is valuel
company yongfatt enterprise SEP]
Template
SROIE
type2 is (SDP)
type2 is value2 SEP)
address is no 122 jalan dedap johor bahru ( (SCP)
(All Types)
SROIE
Templute
type is (SE2)
type value SEP]
company yongfatt enterprise SEP
SROIE
Question
type is?
value
yongfatt enterprise
FUNSD Template
value (SP))
value type [SEP)
coupon code registration form is header (SP))
Question
value is?
type
header
Template
value is [SEP]
value is type]SEP! [
Es Kopi Rupa is menu.nm [SEP]
CORD
Question
value is?
type
menu.nm
Table I: Prompting techniques for different datasets. For the template prompt, GenKIE fills in the missing entity information in the template and each entity type and value pair ended with the [SEP] indicator. Specifically for the SROIE dataset, the template prompt can include all entity types, for which GenKIE generates all entity values in one go. For the question prompt, GenKIE generates the answer, which is similar to the question-answerig task.
vision language models (Wang et al., 2022b,a) to extract the fix-sized contextualized feature map M HxWxx where II the height, W is the width and C' is the number of channels. The fea- ture map M further flattened into a sequence of patches Q E IR. xdums, where dang is the im- age token dimension and K = ps
prompts go through the same textual and layout embedding steps as the transcripts (see Figure 2). We introduce two types of prompts, template prompt and question prompt as presented Table 1. The performance of different prompts is dis- cussed in the ablation study (see $6.3.2). For the entity extraction task, the prompt is designed to be prefixed with the desired entity type and GenKIE continues the prompt with the entity value. For example, in Figure 2, GenKIE generates the value of the Company type by answering the question prompt "Company is?". For the entity labeling task, the prompt is prefixed with the entity value and GenKIE generates the entity type similarly to the classification-based model. For example, in Table on the CORD dataset, GenKIE generates the entity type of Es Kopi Rupa by filling in the template "Es Kopi Rupa is [SEP]".
HxW is the image
token sequence length given a fixed patch size P.
The patches are fed into linear projection layer to conform to the textual embedding dimension. Sim- ilarly textual embedding, we also add trainable ID positional embedding retain the positional in- formation since the visual encoder does not capture that. The visual embedding is formulated as:
VE W POSEMR D(i) Rd
(4)
where W is the trainable parameters of a linear projection layer that maps from the image token dimension to the model dimension. We concate- nate the image embedding with textual embedding to produce the final document multimodal feature embedding:
In essence, the prompt defines the decoder out- put schema and serves as the additional label or value semantic signal to enforce the model to gen- erate the expected output. Although prompt engi- neering requires manual efforts to construct appro- priate prompts, it is more effortless than granular token-level labelling for the entity extraction task. Unlike vectorized prompts in previous works (Li and Liang, 2021; Yang al., 2022), we design the prompts in natural sentences to leverage the power of the pretrained decoder. Moreover, the usage of natural sentences further reduces the overhead of composing the prompt. See Appendix for the method to construct the prompt.
E CV LE) (T++L)
where the concatenation is performed in the second to the last dimension.
4.2 Prompts
of the encoder input to formulate the task as gen- cration problem. The decoder is provided with the filled-in prompt as the sequence generation ob- jective. Inspired by the unimodal textual prompts in DEGREE (Hsu al., 2022), we design simple and efficient spatial awareness prompts which uti- lize both textual features and layout features. The
4.3 Decoder
The decoder input has the filled-in prompt that serves as the learning target. The same BPE to- kenizer and textual embedding in the encoder are

---

utillized. Note that only the textual modality is lever- aged take advantage of the pretrained decoder. During inference, if the prompt is template, we utilize prefix beam search that constrains the search space to start with the template prefix. For example, in Table 1, for the template prompt of the FUNSD dataset, we directly search af-
Dataset
Type
Keys
Images
FUNSD
Form 4
Train 149, Val 0. Test 50
SROIE
Receipt 4
Train 626, Val Test 347
CORD
Receipl 30
Train 800, Val 100, Test 100
Table 2: Statistics of our used datasets.
ter the prefix "coupon code registrat on Form is" for the entity type without generating the prefix from scratch. Prefix beam search not only makes the inference faster and more efficient but also guarantees a deterministic output that follows the template. Entity information can then be parsed from the template easily, In our experiments, we observe prefix beam search can improve the model performance all datasets. We will discuss more in the ablation study.
ski et al., 2021). All baselines except TILT are classification-base models.
Results and Analysis
This section provides experimental results and anal- yses of our model's s effectiveness ($6.1) !) and robust- ness against OCR errors ($6.2) on the KIE task. An ablation study is conducted to analyze the con- tributions of each component of our model ($6.3). Finally, provide case study in $6.4.
5 Experimental Setup
6.1 KIE Effectiveness
5.1 Settings
We evaluate the effectiveness of GenKIE on the entity extraction task (using the SROIE dataset) and the entity labelling task (using the FUNSD and CORD datasets). In this paper, we use token-level evaluation metrics across all datasets.
GenKIE takes a multimodal encoder-decoder- based pretrained model as its backbone. In par- ticular, the backbone model weights used in this work are initialized from the pretrained OFA base model (Wang et al., 2022a), which consists of 6-layer Transformer encoder and a 6-layer Trans- former decoder. The model dimension is 768. For the entity extraction task on the SROIE dataset, the maximum sequence length is 512. For the entity la- belling task on the CORD and FUNSD datasets, the maximum sequence length is 32 for the question prompt and 128 for the template prompt. The maxi- mum encoder sequence length is set to 1024 across all datasets. The model is trained for 50 epochs with batch size of 64 and an initial learning rate - 5. For beam search during inference, the number of beams is configured as and we limit the maximum generated sequence length to 512 for SROIE and 128 for CORD and FUNSD.
As shown in Table 3, GenKIE outperformed LayoutL by certain margin on the FUNSD dataset. However, GenKIE demonstrates compa- rable performance to other strong encoder-based baselines, such as LayoutLMv2 (Xu et al., 2021), DocFormer (Appalaraju et al., 2021), and L.MM- BERT (Garncarek et al., 2021). This result vali- dates the effectiveness of our method and suggests that modelling the KIE task in generative manner nearly as capable as classification-batt models. From Table 3, we can also observe that models with more modality features generally perform bet- ter than those with fewer across all datasets. This suggests that integrating multimodal features can effectively improve KIE performance.
6.2 Model Robustness
5.2 Datasets and Baselines
To verify the robustness of GenKIE against OCR errors, we run entity extraction experiments on the SROIE dataset with simulated OCR errors. We choose LayoutL (Xu et al., 2021) as our base- line model because of its similar embedding design.
We conduct experiments three real-world datasets, including SROIE (Huang al., 2019), CORD (Park al., 2019) and FUNSD (Jaume et al., 2019). Table shows statisties of these datasets Eleven baselines are used for comparison, includ- ing the SOTA models such as LayoutLMv3 (Huang et al., 2022), LayoutL_M2 (Xu et al., 2021), Doc- Former (Appalaraju et al., 2021) and TILT (Powal-
We manually add OCR errors the original ceipt transcripts by replacing the words with visu- ally similar ones* e.g. helle lo v.s. he 1 lo. Each
"The visually similar characters are collected from errors produced by the OCR tool.
"See Appendix A for more details about these datascts.

---

SROTE
CORD
FUNSD
Model
Modality
Params
R
F
R
(Devlin et al., 2019)
110M
90.99
90.99
88.33
91.07
89.68
54.69
67.10
60.26
RoBERTa (Liu 2019)
125M
91.07
91.07
91.07
66,48
66.48
66.48
UniL.Mv2 (Bao et al., 2020)
110M
94.59
94.59
94.59
89.87
91.98
90.92
65.61
72.54
68.90
BROS (Hong et al., 2020)
T+L
110M
96.03
95.48
95.58
95.14
95.36
81.16
85.02
83.05
Layoul (Xu et al., 2020)
T+l.
113M
94.38
94.38
94.38
94.37
95.08
94.72
76.77
81.95
79.27
LAMBERT (Gammareek et al., 2021)
T+L
125M
96.93
94.41
LayourLMv2 (Xu et al., 2021)
T+L4V
200M
96.25
96.25
96.25
94.53
95.39
94.95
80.29
85,39
82.76
LayourL.v33 (Huang et al., 2022)
T+L+V
133M
94.91
95.68
95.30
96.56
90.29
StrucText Li et al., 2021)
T+L4V
107M
95.84
98.52
96.88
85.68
80.97
83.09
TILT (Powalski et al., 2021)
T+L+V
230M
97.65
95.11
Dochomer (Agpalaraju 2021)
T+L4V
183M
96.52
96.14
96.33
80.76
86.09
83.34
GenKiE
T-LFV
180M
97.40
97.40
97.40
95.75
95.75
95.75
83.15
83.45
83.45
Table 3: Overall performance of the compared models our three datasets. Bold indicates the best performance per metric and underline indicates the second best. In the modality column, T represents the textual modality, L represents the layout modality and V represents the visual modality. All models except GenKIE and TILT are discriminative. For Lvy, the evaluation result on the SROIE dataset is obtained by using the source code provided by the authors. Other performance metrics of compared models are obtained from their original papers.
Modality
Precision
Recall
F1
100
T
96.24
96.24
96.24
T+V 97.20
96.82
97.01
90
T+L 96.89
97.39
97.14
T+L+V
97.40
97.40
97.40
80
Table 4: Model performance of different modalities for entity extraction the SROIE dataset. Bold indicates the best performance.
Madel
GenlE EE
LayautLMv2
noting that GenKIE can generate wrong answers although there no OCR errors in the input. We present some qualitative examples of KIE on the SROIE dataset in 6.4 and Appendix B.
Noise Level
Figure 3: Model robustness experiments under different noise levels. LayoutLMv2 entity extraction heavily impacted by the OCR errors. Under the 50% error level, the Fl score drops around 40%. While the performance of LayoutLMM2 entity labelling and GenKIE entity ex- traction drops only around 3%.
6.3 Ablation Study
6.3.1 Effectiveness of Multimodality Features
For the entity extraction task on the SROTE dataset, we ran experiments with the question prompt to analyze the feature importance of the layout and visual embedding. The textual modality in our model is vital and cannot be removed.
word in the transcript has v% of the chance to be replaced (n denotes the level of OCR errors) and if there visually similar characters, keep the original text.
As shown Table 4, the model trained with full modality achieves the highest Fl score. The model trained with unimodal textual modality has the lowest score, which verifies that both visual and layout embeddings are effective in this task.
We run experiments under different levels of OCR errors ranging from 5% to 50% with 5% step. As shown in Figure 3, when the error level in- creases, the Fl score of LayoutLMv2 drops signifi- cantly since the OCR errors
6.3.2 Effectiveness of Different Prompts
handled by de- OCR error level, GenKIE still achieves nearly 94% F1 score dropping only 3% from 97%, which proves that GenKIE has strong ro- bustness against OCR errors. However, it is worth
In Table compare the effectiveness of differ- ent prompts on all datasets. In particular, for entity extraction on the SROIE dataset, we have an addi- tional template prompt that includes all entity types in the prompt, e.g. the template prompt in Figure
sign. Under the 50%

---

Prefix
Dataset
Prompt
P
R
F
Search
Template
96.88
96,96
96.92
(All Types)
SROIE
Template
V
96.76
96,99
96.87
Template
X 96.86
96.63
96.75
QA
97.40
97.40
97.40
Template
V 83.45
83.45
83.45
1003 3777
FUNSD
Template
x
76.42
76.42
76.42
QA
75.17
75.17
75.17
(4) company
company
Template
95.75
95.75
95.75
Figure 4: Qualitative examples the entity extraction task SROIE. While LayoutLMv2 misled by the OCR result, GenKIE corrects these OCR mistakes.
CORD
Template
94,24
94.24
94.24
QA
92.44
92.44
92.44
Table 5: Effectiveness of different prompts and prefix beam search on SROIE, FUNSD, and CORD datasets. Bold indicates the best F1 performance over different schemes per each dataset.
76.42). This could be due to abundant value semantics in the entity labeling task of the FUNSD datasets, which adds difficulty for the model to gen- crate the complete prompt from scratch vanilla beam search. In the constrained search space of pre- fix beam search, the model only needs to generate the entity type.
2. The model needs to fill in all the required entity values in one generation. Note that formulating the dataset with the prompt of single entity type can result in possible duplication of data as the doc- ument can have multiple entity types. It requires careful processing to not incur unnecessary putational overhead during training. However, do not observe a large performance gap between a template prompt with all entity types and other prompts, c.g. 96.92 v.s. 97.40 in terms F1, which suggests that using a template prompt with all en- tity types is a simple while efficient mechanism to avoid duplication data processing at the cost of minor performance drop.
6.4
Case Study
The motivation behind GenKIE is to cope with the OCR errors for practical document KIE. To verify the model's robustness and effectiveness, we show some qualitative examples of the output of GenKIE on the SROIE dataset. As presented Figure 4, subfigures (a) to (c) have multiple OCR errors in the company and address entities, and our GenKIE is able to correct all of them, e.g. TED v.s. TEO, 8HD v.s. BHD, Pemias v.s. Permas, etc. See Appendix B for more examples, including the failure cases where GenKIE generates the wrong entities even when there no OCR error.
For entity labeling on the CORD and FUNSD datasets, the template prompt outperforms the ques- tion prompt while for entity extraction on the SROIE dataset, the question prompt outperforms the template prompt. This indicates in the entity extraction task, the model benefits more from the question question-answerte answering formulation since the answer space unconstrained from the template and the generation capability utilized more. In the entity labelling task, GenKIE essentially works similarly to classification- based models. The value seman- ties provided by the template prompt can effec- tively restrict the search space and guide the model to generate the desired entity type.
Conclusion
We propose GenKIE, novel prompt-based gen- crative model to address the KIE task. GenKIE is effective in generating the key information from the scanned document images by following care- fully designed prompts. The validated strong ro- bustness against the OCR errors makes the model applicable in real-world scenarios. Extensive ex- periments over two KIE tasks (i.c. entity labelling and entity extraction) on three public cross-domain datasets demonstrate the model's competitive per- formance compared with SOTA baselines. Our GenKIE incorporates multimodal features. which enables the integration with other vision language models, offering possibilities for future exploration and experimentation.
6.3.3 Effectiveness of Prefix Beam Search
As presented in Table 5, when the template prompt is used, prefix beam search outperforms vanilla beam search by small margin on the SROIE and CORD datasets, while the performance gap is no- tably large on the FUNSD dataset (e.g. 83.45 v.s.

---

Limitations
nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
One limitation with GeoKIE is that it might re- quire additional processing of the datasets. As most of the document KIE datasets nowadays might be tailored for the classification task only, it takes some time to formulate the datasets with different prompts, And it also takes time to experiment with those prompts to find the best one.
Teakgyu Hong, DongHyun Kim, Mingi Ji, Wonseok Hwang, Dachyun Nam, and Sungrae Park. 2020. Bros: pre-trained language model for understand- ing texts in document.
I-Hung Hsu, Kuan-Hao Huang, Elizabeth Boschee, Scout Miller, Prem Natarajan, Kai-Wei Chang, and Nanyun Peng. 2022. DEGREE: data-efficient generation-base event extraction model. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1890-1908.
Besides, although the multimodal feature embed- ding is effective in coping with semantic ambiguity, it is only utilized in the encoder in finetuning, Pre- training the model on large document datasets with multimodal features could potentially improve the model's 's performance.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutimv3; Pre-training for doe- ument ai with unified text and image masking. In Proceedings of the 30th ACM International Confer- ence on Multimedia.
References
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. 2021. Docformer: End-to-end transformer for document understanding. In Proceedings of the IEEE/CVF International Con- ference Computer Vision, pages 993-1003.
Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Di- mosthenis Karatzas, Shijian Lu, and C. V. Jawahar. 2019. lcdar2019 competition scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition. pages 1516-1520.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang. Jianfeng Gao, Song- hao Piao. Ming Zhou, et al. 2020. Unilmv2: Pseudo- masked language models for unified language model pre-training. In International Conference on Ma- chine Learning, pages 642-652
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirec- tional Istm-erf models for sequence tagging. arXiv preprini
Wonseok Hwang, Jinyeong Yim, Seunghyun Park, So- hee Yang, and Minjoon Seo. 2021. Spatial depen- dency parsing for 2d document understanding. Findings of the Association for Computational Lin- guistics: ACL-IJCNLP 2021, page 330-343.
Jason P.C. Chiu and Eric Nichols. 2016. Named Entity Recognition with Bidirectional LSTM-CNNs. Trans- actions of the Association for Computational Linguis- fics, 4:357-370.
Guillaume Jaume, Hazim Kemal Ekenel, and Jean- Philippe Thiran. 2019. Funsd: dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops, volume 2. pages 1-6.
Andreas Dengel and Bertin Klein. 2002. Smartfix: requirements-drivee system for document analysis and understanding. In Proceedings of the 5th Inter- national Workshop on Document Analysis Systems page
Jing Jiang. 2012. Information Extraction from Text, pages 1141. Springer US.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Anoop R Katti, Christian Reisswig, Cordula Guder, Se- bastian Brarda, Steffen Bickel, Johannes Höhne, and Jean Baptiste Faddoul. 2018. Chargrid: Towards un- derstanding 2D documents. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 4459 4469.
Lukasz Garnarrkk Rafal Powalski, Tomasz Stanistawek, Bartosz Topolski, Piotr Halama, Michal Turski, and Filip Gralinski. 2021. Lambert: Layout-aware language modeling for information extraction. Document Analysis and Recognition- - ICDAR 2021: 16th International Conference, page 532-547.
Geewook Kim. Teakgyu Hong. Moonbin Yim. JeongYeon Nam, Jinyoung Park, Jinycong Yim, Won- scok Hwang, Sangdoo Yun. Dongyoon Han, and Seunghyun Park. 2022. Ocr-free document under- standing transformer. In European Conference on Computer Vision pages 498-517.
Guillaume Lample. Miguel Ballesteros, Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog-

---

In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional I Linguistics: Human Language Technologies, pages 260-270.
Transactions of the Association for Computational Linguistics, 5:101-115.
Rafat Powalski, Lukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michal Pietruszka, and Gabriela Patka. 2021. Going full-tilt boogie on document understanding with text-image-layour transformer. In Document Analysis and Recognition ICDAR 2021, pages 732-747.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics, pages 7871-7880.
Daniel Schuster. Klemens Muthmann, Daniel Esser, Alexander Schill, Michael Berger, Christoph Wei- clingg, Kamil Aliyev, and Andreas Hofmeier. 2013. Intellix - end-user trained information extraction for document archiving. In 2013 12th International Conference on Document Analysis and Recognition. pages 101-105.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4582- 4597.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1715-1725
Yulin Li, Yuxi Qian, Yuechen Yu, Xiameng Qin, Chengquan Zhang. Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, and Errui Ding. 2021. Structext: Struc- tured text understanding with multi-modal transform- ers. In Proceedings of the 29th ACM International Conference Multimedia, page 1912-1920.
Linfeng Song, Yue Zhang, Zhiguo Wang. and Daniel Gildea. 2018. N-ary relation extraction using graph- state LSTM. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing. pages
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 1-(5):1335.
Tomasz Stanislawek, Filip Graliński, Anna Wróblewska, Dawid Lipinski, Agnieszka Kaliska, Paulina Rosal- ska. Bartosz Topolski, and Przemyslaw Biecek. 2021. Kleister: key information extraction datasets involy- ing long documents with complex layouts. In- ternational Conference on Document Analysis and Recognition, pages 364-579. Springer.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du. Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1 16692.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10012-10022.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022a. OFA: Unifying architectures, tasks, and modalities through a sim- ple sequence-to-sequeecce learning framework. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 23318 8-33340.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end se-
quence labeling via bi-directional L.SM-CNNN-CRRE In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2022b. SimVLM: Simple visual language model pretraining with weak supervision. In International Conference on Learn- ing Representations.
Behrang Mohit. 2014. Named Enttty Recognition, pages 221 221-245.66 Springer Berlin Heidelberg.
Seunghyun Park, Scung Shin, Bado Lec, Junyeop Lec, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. 2019. Cord: A consolidated receipt dataset for post-ocr parsing. In Workshop on Document Intelligence at NeurlPS 2019.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang. Yijuan Lu, Dinci Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021. Layoutl Mv2: Multi-modal pre-training for visually-rich document understanding, In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the I lik International
Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017. Cross-Sentence N-ary Relation Extraction with Graph LSTMs.

---

Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2579-2591
is intentionally removed from the end of the ad- dress line, leaving OCR recognize blank string, GenKIE still able to reconstruct the word thanks to the powerful generative capability. It's worth mentioning that subfigures (e) and (f) are failure cases, in which GenKIE generates wrong entity values even if the OCR result is correct. This is rea- sonable in that OCR errors could influence model training and lead the model to learn wrong features.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang. Furu Wei, and Ming Zhou. 2020. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD Interna- tional Conference on Knowledge Discovery &amp: Data Mining, page 1 1122-1200.
Hao Yang, Junyang Lin, An Yang, Peng Wang, Chang Zhou, and Hongxia Yang. 2022. Prompt tuning for generative multimodal pretrained models. arXiv preprint 2rXhv2220.02532.
C Prompt Construction
The prompt is intended to be precise and simple to not incur any semantic ambiguity and linguistic overhead. One strategy to construct the prompt for the entity extraction task is first identifying all the entity types in the document. Then for each entity type, c.g. A, we can either construct the template prompt such as "A is [SEP]" and "A: [SEP] )" or the question prompt such as "What is A?" and "A is ?". And we append the prompt to the end of the document transcript to form training instance. For the entity labeling task, we use the entity value to construct the prompt and the same strategy be applied.
Wenwen Yu, Ning Lu, Xianbiao Qi. Ping Gong, and Rong Xiao. 2021. Pick: Processing key information extraction from documents using improved graph leaming--onnvolutional networks. In 2020 25th Inter- national Conference on Puttern Recognition, pages 4363-4370.
Yuechen Yu, Yulin Li, Chengquan Zhang, Xiaoqiang Zhang. Zengyuan Guo, Xiameng Qin, Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang. 2023. Structextv2: Masked visual-textual prediction for document image pre-training. In The Eleventh Inter- national Conference on Learning Representations.
Datasets
D Zero-Shot and Few-Shot Entity Extraction
FUNSD consists of 199 forms annotated with 4 en- tity types, where each entity type can correspond to multiple values. Each image of the dataset contains many value-type pairs and the task to perform is entity labeling.
To further test our model's generalization abil- ity, conduct entity extraction experiments un- der zero-shot and few-shot settings on the SROIE dataset.
SROIE contains 626 receipts for training and 347 receipts for testing and each receipt has four entity types for information extraction. Unlike FUNSD, SROIE focuses on key entity extraction from the document, where there could be a lot of unrelated entities. The model performs entity extraction on this dataset and is evaluated on the mentioned four entity types.
D.1 Experimental Settings
In each experiment, we select one entity type as an unseen type and the other types as common types. To simulate the zero-shot setting, we remove all training instances with unseen types. For the few- shot setting, we only keep k training instances for the unseen type (denoted as k-shot). We evaluate the performance only for those unseen types in the test dataset with the Fl score.
CORD has 800 scanned receipts for the training set, 100 for the validation set, and 100 for the test set. There are total 4 main categories in this dataset, which are further classified into 30 subcat- egories, such as menu name under menu category, total price under total category etc. Similar to the FUNSD dataset, the task is entity labelling.
D.2 Experimental Results
Table shows the results of zero/few-shot experi- ments. Performance of GenKIE is relatively lim- ited compared to full-shot training. However, for entity types with common semantics such as date, few-shot training can significantly boost the perfor- mance, which justifies the strong generalizability of the model.
Case Study
We provide more qualitative examples on the SROIE dataset to validate the effectiveness of GenKIE. In subfigure (d), an entire word IPOH

---

INC
LCT SLMAA
Faym 4.M0
00:15
5TM
Fay 4.00
Eunrih
C0002
NETWONK
Giaft
J00
Guatt
Guaff
CEITO 5T8 1910
Goalt
SRO ACON TAMAX
compuny
03 company
03 Hcchees
(5) AddTNNO
(O) company
Y addresss
Figure 5: Qualitative examples of the entity extraction task on the SROIE dataset. We compare GenKIE with Lvv. In subfigures (a) to (d), GenKIE correctly fixes the OCR mistakes. In subfigures (e) to (f), GenKIE generates the wrong entity values even though there OCR error. Best viewed in colour.
Entity Type
0 shot
shot
shot
shot
full shot
company
4.0
3.7
34.18
42.43
96.97
address
3.6
12.38
55.60
97.22
date
3.2
14.53
77.21
78.21
97.45
total
5.0
4.9
45.92
66.33
97.83
Table 6: Results of zero/few-shor entity extraction SROIE.


Reviews:
[{"review_id": "qora8Ts7WJ", "reviewer": "Reviewer_Bc8K", "summary": "", "questions": "Question A: It would be useful to have a reference of the fraction of labels affected by OCR corruption in the datasets.\n\nQuestion B: Not clear how prompts can have visual features unless they are contained in the input document, which seems unlikely if the prompt starts with an entity type rather than value.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The motivation of standard discriminative extraction being prone to OCR noise is valid, and using a generative prediction framework is a sensible idea, if not particularly novel.\n\nThe presented generative system is competitive with several discriminative and few generative systems on 3 IE datasets. It especially excels in the robustness study where authors vary the level of OCR noise.", "reasons_to_reject": "The idea of using encoder-decoder for IE is far from novel, which the authors partially acknowledge in the related work section.\n\nThey cite the limitations of a similar encoder-decoder Donut in related work but do not include it in the evaluation.\n\nAlso, not clear what innovations were performed on top of OFA other than adding textual and layout embeddings.\n\nThe authors go to a great length to explain the details that seem fairly common among multimodal transformers (e.g., concatenation of position embeddings with token embeddings, using ResNet feature patches), rather than focusing on specific modifications.\n\nVanilla OFA performance not included in the evaluation (or in the appendix), and it remains unclear whether or not the introduced textual features are effectively leveraged by the model.\n\nFinetuning dataset sizes appear too small to expect significant improvement without dedicated textual pre-training.", "questions_to_authors": "Question A: It would be useful to have a reference of the fraction of labels affected by OCR corruption in the datasets.\n\nQuestion B: Not clear how prompts can have visual features unless they are contained in the input document, which seems unlikely if the prompt starts with an entity type rather than value.", "paper_topic_and_main_contributions": "The paper raises the issue of the standard token-classification-based approach to VDU being prone to OCR errors and proposes to use a generative sequence-to-sequence framework, which does not require costly token-level annotations and has the potential to correct OCR errors.\n\nThey augment an image-to-text model OFA with textual and layout features and finetune it on SROIE, CORD and FUNSD datasets, demonstrating competitive results against a panel of baselines.\n\nIn a follow up study they show their model's robustness to increasing OCR noise compared to LayoutLM and include ablations studies with different types of prompts.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Language/grammar:\n\n018 entice the generation\n\n050 extracting interested entities\n\n271 positional embedding element-wisely\n\nMissing citations for BROS:\n179 (?)"}, "extra_text": {}}, {"review_id": "0qVit65FsB", "reviewer": "Reviewer_r4Eh", "summary": "", "questions": "- Question A: (line 446) It is unclear what types of OCR noises are added in the ablation in Sec. 6.2. It would be good to clarify them in the appendix.\n- Question B: (figure 4 and line 551) In the examples, do you show the original OCR text or perturbed version in Sec 6.2? \n- Question C: (line 383) In the decoder, apart from the prefix decoding, do you use any constrained decoding methods (e.g., limiting the vocabulary to those appeared in the input)? \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Overall this is a nice approach that can be helpful in practical tasks with potentially high OCR errors. In addition, the decoder-based training removes the need for token level annotations and reduces the cost for annotation. \n2. While the idea of using an encoder-decoder network for KIE tasks is not completely new (as the author clarified in the paper (line 207)), it is unclear of the used prompts and the generation task in the previous work TILT. It is good that this paper provides a detailed explanation of the methodology and prompts and detailed analysis. \n", "reasons_to_reject": "This work can be improved in the following dimensions: \n1. While it is understandable, the size of the test set for all the datasets is generally small (e.g., FUNSD has only 50 examples) and the results might be highly volatile. \n2. The design of the robustness experiment (line 458) can be improved. The argument from the authors is that the GenKIE approach can be robust to (realistic) OCR noises. However, the authors added additional OCR noise to the SROIE dataset (which already contains OCR noise). As such, this setting could be overly harsh to token-classification based models like LayoutLMv2: the experiment is based on the assumption that the current OCR models can have worse accuracy which is not necessarily true. An ideal experiment should be comparing two scenarios: (1) using the imperfect OCR text generated (from different OCR engines) vs. (2) using a manually corrected text representation in the input. By doing so it can reveal to what extent this approach can be helpful in practice. \n", "questions_to_authors": "- Question A: (line 446) It is unclear what types of OCR noises are added in the ablation in Sec. 6.2. It would be good to clarify them in the appendix.\n- Question B: (figure 4 and line 551) In the examples, do you show the original OCR text or perturbed version in Sec 6.2? \n- Question C: (line 383) In the decoder, apart from the prefix decoding, do you use any constrained decoding methods (e.g., limiting the vocabulary to those appeared in the input)? \n", "paper_topic_and_main_contributions": "This paper describes a novel modeling approach called GenKIE for key information extraction tasks from scanned documents. Different from traditional sequence tagging approaches, the proposed approach adopts an encoder-decoder transformer to directly generate the desired data text. Experimental results on three datasets show that this new approach can achieve similarly good performance as prior work, and can be more robust to OCR noises in the input compared to other methods. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 60: There is a missing reference for BROS."}, "extra_text": {}}, {"review_id": "eOg3T99K5S", "reviewer": "Reviewer_Y6Ra", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\nThe multimodal approach to representing documents and using generative AI to avoid OCR errors is appealing.\nThere is very less work in this area and this paper helps move the field forward.", "reasons_to_reject": "None. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "Authors propose a generative approach to key information detection in document images.\nInstead of the standard OCR-followed-by-NER approach, they build a multimodal representation of the document by fusing image and text representations and then using generative approach to output the desired text. The generative approach helps in avoiding issues with\nOCR errors. Results are presented on open datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MkwDYBAjpG", "reviewer": "Reviewer_FmZH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed approach is more capable in theory.", "reasons_to_reject": "While the generative approach is proposed to solve some OCR error issues, but the overall end-to-end accuracy is lower than previous methods.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes to use a generative approach to perform key-value extraction of document images, e.g. decoding the key and value text directly. Compared to previous discriminative works that produces BIO tagging, this approach can fix OCR errors by design.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors consider a sequence to sequence approach for key information extraction, taking multi-model data for encoding and yielding the output structures directly, thereby avoiding OCR errors. This is a dedicated study for solving a practical problem. While there are existing sequence to sequence methods, the authors address several important issues in prompt design, generation tasking and OCR errors etc.
Decision: Accept-Findings