WUKONG-READER: Multi-modal Pre-training for Fine-grained Visual Document Understanding
Year: 2023

Abstract:
Unsupervised pre-training on millions of digital-born or scanned documents has shown promising advances in visual document understanding (VDU). While various visionlanguage pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far. A document textline usually contains words that are spatially and semantically correlated, which can be easily obtained from OCR engines. In this paper, we propose WUKONG-READER, trained with new pre-training objectives to leverage the structural knowledge nested in document textlines. We introduce textline-region contrastive learning to achieve fine-grained alignment between the visual regions and texts of document textlines. Furthermore, masked region modeling and textline-grid matching are also designed to enhance the visual and layout representations of textlines. Experiments show that WUKONGREADER brings superior performance on various VDU tasks in both English and Chinese. The fine-grained alignment over textlines also empowers WUKONG-READER with promising localization ability.

heading: 1 Introduction
text: Visual document understanding (VDU) handles various types of digital-born or scanned documents like forms, tables, reports, or research papers, and is becoming increasingly important for real-world industrial practices [7]. Multi-modal pre-training on millions of documents is a popular solution for visual document understanding [12, 33, 35, 34, 14, 27]. Unlike the conventional vision-language pre-training over natural images and their paired short and abstractive descriptions [29, 22, 21], the document texts are usually long and highly correlated with the images, since they can be easily
∗Equal Contribution. †Corresponding authors.
obtained from accurate Optical Character Recognition (OCR) engines from the scanned images. Therefore, it is crucial to strengthen the connection between vision and language for VDU with more fine-grained alignment across the two modalities.
Towards that end, existing efforts seek to align the visual and textual knowledge of documents at different levels. A commonly used pre-training objective for documents is masked language modeling [8] over document text tokens [35, 34, 14, 33, 12, 27], often accompanied by the layout information encoded via the positional embedding. Besides, various visual and vision-language multimodal pre-training objectives are also proposed, leveraging the patch-level features [34, 14], objectlevel features from object detectors [23, 11], or the whole image feature through a global text-image matching loss [34].
However, as an intrinsic granularity for VDU, document textlines have been mostly neglected in past efforts. Intuitively, a textline contains a set of words that are spatially and semantically related. For instance of information extraction, the desired text span (e.g., the names on letters and addresses on receipts in Figure 1) often appears in a single textline. Therefore, the document textline serves as an appealing fine-grained granularity for VDU
13386
tasks. While StructualLM [19] similarly considers textlines as cell layout information, they only use the textual features of these textlines in language modeling. Instead, in this work, we seek to enhance the multi-modal representation of a document by aligning the visual region and text span corresponding to the same textline.
In this work, we propose WUKONG-READER, a pre-trained document model with a hybrid dualand single-stream multimodal architecture. To learn fine-grained document representation, we propose the Textline-Region Contrastive Learning to align the visual and textual features of document textlines from the dual-stream encoders. The objective thus connects the spatial and semantic information among document textlines for various VDU tasks. Additionally, we also introduce two other objectives to further improve the textline representation. We design the Masked Region Modeling to recover the masked textline regions, so as to enhance the visual features of textline. We also propose the Textline Grid Matching to strengthen the layout information of textlines, which localizes each word of textlines to the pre-defined image grids. Similar to previous works [35, 34, 14], the classic masked language modeling objective is also applied over document texts.
Experimental results show that our WUKONGREADER notably improves various document understanding tasks across both English and Chinese. For instance, WUKONG-READERlarge with 470M parameters achieves the weighted F1 score of 93.62 on FUNSD [16] and 98.15 on SROIE [15], leading the new state-of-the-art records on information extraction tasks. We also demonstrate that the textline-based pre-training objectives empower the model with meaningful textline features with promising localization ability.

heading: 2 Related Work
text: Visual document understanding (VDU) has been widely studied in recent years [12, 19, 27, 34, 35]. VDU tasks are abundant in textual and visual information, as intensive texts and their layout information can be extracted from documents via Optical Character Recognition (OCR) or other document parsers. Muti-modal pre-training has been a popular solution for VDU. Usually, a pre-trained text encoder (e.g., BERT [8]; RoBERTa [24]) is applied to learn contextualized representations of the textual input. Meanwhile, a pre-trained visual encoder
such as CNN-based [34] and transformer-based [14, 28] models are applied to process visual features. Various self-supervised pre-training objectives over millions of documents have shown promising effects for VDU. Reconstructive objectives such as masked language modelling (MLM) [8], and masked image modelling, (MIM) [9], are often used to perform the self-supervised document pretraining [20, 34].
Since the textual knowledge is parsed from the document image, existing efforts explore various document granularities to align the vision and language modalities. They can be generally divided into four categories: 1) Word-level: LayoutLM [35] jointly models the inner-relationship between texts and layout 2D positions from documents, via pre-trained language models [8, 24]. However, the visual features are not used in the pretraining architecture. TILT [28] additionally adds a contextualized image embedding to the word embedding. 2) Grid/Patch-level: LayoutLMv2 [34], DocFormer [2] and ERNIE-Layout [27] extract image grid features with CNN backbones, and LayoutLMv3 uses image patches to encode visual features inspired by ViT [9]. To achieve the cross-modal alignment, they adopt the textimage alignment (i.e., TIA) and matching (i.e., TIM) objectives during pre-training. 3) Objectlevel: SelfDoc [23] and UniDoc [11] extract object features via document object detectors, and concatenate them with word features. SelfDoc [23] uses two cross-modality attention functions to identify the inner-relationships from one modality to another. UniDoc [11] designs the similaritypreserving knowledge distillation to encourage alignment between words and visual features. 4) Cell-level: StructualLM [19] uses the textual features of cell layout information, which is similar to document textlines. However, it only considers the textual feature without the visual information.
Different from existing works, we target at the textline-level features of both textual and visual modalities. We propose a hybrid dual- and singlestream model architecture for multi-modal pretraining. Similar architectures are previously explored in general multi-modal pretraining [22, 21], however, they mostly focus on learning the global visual and textual features over natural images. Instead, we aim to align the fine-grained knowledge nested in document textlines. We believe such an important granularity of documents can benefit
both language and visual representation learning in VDU tasks.

heading: 3 Methodology
text: We propose WUKONG-READER, a new pre-trained multi-modal model for visual document understanding. Our model jointly encodes the visual image and textual tokens via two mono-modal encoders, followed by a multi-modal encoder to fuse the two modalities. To leverage the structural information nested in document textlines, We propose several novel pre-training objectives for finegrained representation learning of documents.

heading: 3.1 Model Architecture
text: The overall architecture of the proposed WUKONGREADER is shown in Figure 2. WUKONG-READER encodes the document image and text through separate encoders and then fuse the two modalities via the multi-modal encoder. Besides, we also deploy an RoIhead and an image decoder for fine-grained learning over document textlines.
Image Encoder. We use the Mask-RCNN model trained on PubLayNet1 to learn the visual representations for WUKONG-READER. Specifically, we use the visual backbone of Mask-RCNN as the image encoder. The visual features from the image encoder are adaptively pooled into 49 visual tokens. The RoIHead of Mask-RCNN then extracts the regional features of document textlines for contrastive learning with texts. Meanwhile, an image decoder is also deployed to recover the visual features over textline regions.
Text Encoder. Given a document image, we use an off-the-shelf OCR tool to extract the textual information from the image, which includes both the words and their corresponding bounding boxes. Following [35, 34], we normalize the bounding boxes within [0, 1000] and use 2D positional embedding layers to encode the layout information. We initialize the text encoder with the first six layers of the RoBERTa model, and employ the spatialaware self-attention mechanism following [34] in the Transformer layers. We calculate the input embedding as the summation of the token embedding
1We adopt the configuration of “MaskRCNN ResNeXt101 32x8d FPN 3X” as provided in https://github.com/ hpanwar08/detectron2.
from RoBERTa tokenizer, 1D positional embedding, 2D positional embedding and the segment embedding following [34]. The input embedding is then fed to the text encoder to get textual features.
Multimodal Encoder. We concatenate the tokenlevel features from both vision and text, and feed them to the multi-modal encoder to jointly fuse the two modalities. We initialize the multi-modal encoder with the rest layers of the RoBERTa model. Before concatenation, we also add 1D and 2D positional embeddings to visual features following [34].
During inference, the model architecture also enjoys model efficiency. Instead of concatenating the visual and textual features together for self attention [34, 14], it would be computationally cheaper to first obtain the single-modal representations with respective encoders before fusion. Similar ideas are also discussed in ALBEF [22] and BLIP [21]. Moreover, for downstream tasks such as information extraction or document classification, the ROI head and image decoder can be safely discarded to further reduce the model size.

heading: 3.2 Pre-training Objectives
text: As the fundamental pre-training objective in modeling languages, we use the Masked Language Modeling (MLM) to recover the masked word tokens in the document text. We follow the standard masking strategy in BERT [8] and mask out 15% word tokens. Besides, to prevent information leakage, we also cover the corresponding image regions and set their bounding boxes to zeros, following [34].
Despite the powerful effect of MLM, it fails to explicitly leverage the visual information. We thus propose to mine the fine-grained image-text alignment through multiple new pre-training objectives.

heading: 3.2.1 Textline-Region Contrastive Learning
text: As is shown Figure 1, a textline of a document returned by OCR usually contains a set of words that are semantically related. We are thus motivated to exploit structural knowledge within it by textlineregion contrastive learning (TRC). Specifically, to obtain the textual representation of a textline, we average the features of tokens within that textline. Besides the textual feature, we also employ a multilayer perception based RoIHead on top of the image encoder to extract the visual feature corresponding to the textline region in the document image.
Contrastive representation learning has been widely used for vision-language cross-modal pre-
training [29, 37]. To enhance the alignment of a document image and its textual content, we also utilize contrastive learning to align the textline-region and texts. For ease of presentation, we suppose there is a batch of N document image-text pairs, and each document has L textlines. For the nth document, denote ρn and τn as the visual and textual feature of document textlines, respectively. Note that we pad ρn and τn with 0 to length L for documents with fewer than L textlines. For each document image, its paired text is used as its positive, and the texts from other documents are used as its negatives. The contrastive learning from image to text can be formulated as
L(ρm, τ 1:N ) = − 1
N log exp(s(ρm, τm))∑N n=1 exp(s(ρm, τn)) ,
where s(ρm, τn) represents the similarity of the m-th image to the n-th text computed in the granularity of textlines. By symmetry, the contrastive objective from text to image is
L(τm,ρ1:N ) = − 1
N log exp(s(τm,ρm))∑N n=1 exp(s(τm,ρn)) .
The TRC objective thus sums the two terms as:
LTRC= 1
2
N∑
m=1
( L(ρm, τ 1:N )+L(τm,ρ1:N ) ) . (1)
The cross-modal interaction is reflected in how the similarity between the image and text is computed. Existing contrastive learning methods simply calculate the similarity based on the global feature of the image or text [34, 14, 27]. To establish fine-grained alignment over textlines, the key lies in the following similarity metric. Inspired by [37, 10] we adopt the average textline maximum similarity which is computed as
s(ρm, τn) = 1
L
L∑
l=1
max 1≤k≤L
( ρ⊤m,lτn,k ) , (2)
s(τm,ρn) = 1
L
L∑
l=1
max 1≤k≤L
( ρ⊤m,kτn,l ) , (3)
where ρm,l represent the l-th textline of the m-th visual feature, and τn,k similarly denotes the kth textline of the n-th textual feature, respectively. The defined similarity shows that for each image region of textlines, we find their most similar text segments. Similarly, for each textline text, we also
find its closest image region of textlines. With the objective in Equation (1), such design intrinsically encourages the fine-grained alignment between the visual and textual features of textlines.

heading: 3.2.2 Masked Region Modeling
text: To enhance the visual representation of document textlines, we further propose the Masked Region Modeling (MRM) to recover the masked pixels of textline regions during pre-training.
Specifically, for the n-th document image, we randomly mask 15% textlines of the document for recovery. A document textline is usually dominated by white background pixels instead of foreground characters. To avoid trivial solutions and balance the foreground and background pixels in a textline, we mask all black strokes as well as 15% of background pixels within each textline. Our pre-training objective is to predict these masked pixels based on their surroundings. On top of the image encoder, we use three deconvolution layers as the image decoder to recover the textline visual features ρ̃maskn . As the pre-training objective of MRM, we adopt the ℓ1 loss [23] between the reconstructed ρ̃maskn and the original ρn:
LMRM = N∑
n=1
ℓ1(ρn, ρ̃ mask n ). (4)
Note that if a masked textline contains masked tokens introduced in the MLM task, we do not calculate the reconstruction loss for this token.

heading: 3.2.3 Textline Grid Matching
text: Aside from enhancing the visual representations of textlines, layout information of textlines also plays an important role for visual document understanding. We thus introduce the Textline Grid Matching (TGM) to explicitly model the layout of each word in textlines.
Specifically, we first split each document image into G pre-defined grids. Then we randomly sample 15% textlines that are not used in MLM and MRM, and predict which grid each output token in the selected textline belongs to. For the n-th document, suppose we sampled L′ textlines. We first transform the output from the multi-modal encoder to obtain a set of grid logits yl,1:Tl , where Tl is the number of words in the l-th textline. To avoid leakage of position information, we set the 2D bounding boxes of tokens in the selected textlines as [0, 0, 0, 0]. We then classify the grid logits into the
G classes over the image, by minimizing the crossentropy loss ℓce as
LTGMn = L′∑
l=1
Tl∑
t=1
ℓce(yl,t, gl,t),
where gl,t is the corresponding ground-truth label of yl,t. The Texline Grid Matching loss for a minibatch is the summation over all the documents in this batch:
Ltgm = 1
N
N∑
n=1
LTGMn . (5)
Compared with the previous TIA loss in LayoutLMv2 [34] which simply classifies whether a token is masked, TGM enhances the layout information via explicit grid localization from both nearby unmasked textual tokens and visual regions.
The total pre-training loss is the combination of the four pre-training objectives introduced above:
Ltotal = LMLM + λ1LTRC + λ2LMRM + λ3LTGM, where λ1, λ2 and λ3 are the scaling parameters that control the weights of different loss terms. For simplicity, we choose λ1 = 0.2, λ2 = λ3 = 1 for all our experiments. It is possible that better performance can be achieved with a more careful tuning of these scaling parameters.

heading: 4 Experiments
text: In this section, we empirically verify the proposed WUKONG-READER across different VDU tasks in both English and Chinese. We first introduce the experimental setup and the main results on English VDU tasks in Section 4.1 and Section 4.2, respectively. Section 4.3 provides further discussion, e.g., ablations and localization abilities. Finally, we also investigate the ability of Wukong-Reader in Chinese in Section 4.4. The implementation of WUKONG-READER is based on MindSpore [1].

heading: 4.1 Experimental Setup
text: Model Configuration. We study the proposed model in two sizes: WUKONG-READERbase and WUKONG-READERlarge. For both sizes, we use the pre-trained MaskRCNN model to initialize the image encoder, including the ResNet-101 visual backbone and the multi-layer-perception based RoIHead. RoBERTa-base and RoBERTa-large2 are
2RoBERTa-base and RoBERTa-large are downloaded from https://huggingface.co/roberta-base/ tree/main and https://huggingface.co/ roberta-large/tree/main, respectively.
used to initialize the rest parts of the base and large models, respectively. We fix the textual encoder with 6 transformer layers, and use the rest Transformer layers of the RoBERTa model for the multimodal encoder. Following [34, 14], the image is cropped to 224×224 resolution and then adaptively pooled into 49 visual tokens by the image encoder. We fix the sequence length of the textual encoder as 512, and hence 561 for the multi-modal encoder. For textline-region contrastive learning, we truncate the first 64 textlines for each document. We evaluate WUKONG-READER on various document understanding tasks: information extraction and document classification in Section 4.2, layout analysis in Appendix B.1, and document visualquestion answering in Appendix B.2.
Compared Methods. We compare WUKONGREADER against the following methods with different granularities: (i) Word-level features: BERT [8] and RoBERTa [24] trained with the conventional masked-language modeling over words. LayoutLM [35] and TILT [28] obtains words’ bounding boxes from OCR and add them to the paired text embeddings. (ii) Grid/patch-level features: LayoutLMv2 [34] and DocFormer [2] ex-
tract image grid features with a CNN backbone, and LayoutLMv3 uses ViT [9] to encode image patches; (iii) Object-level features: SelfDoc [23] and UniDoc [11] concatenate text embeddings with region features from object detectors; and (iv) Textline-level features: StructuralLM [19] leverages the cell-level text and layout information.
Pre-training. Following previous studies [35, 34], we adopt the IIT-CDIP Test Collection dataset [18] for pre-training, which contains 11M document images from various industrial domains. We extract the texts and bounding boxes using our internal OCR tool. We use 64 Ascend 910 accelerators for pre-training, and the batch size of 24 per device. We use the Adam optimizer [17]. The learning rate is linearly warmed up to 1e-4 within the first 10% iterations, and then linearly decayed to 0. The weight decay is set as 1e-2. To save running memory we also enable gradient checkpointing [5] and FP16 training. We conduct pre-training for 10 epochs, which takes around 3 days and 5 days on 64 accelerators respectively.

heading: 4.2 Main Results

heading: 4.2.1 Information Extraction.
text: Datasets and Evaluation Metric. For information extraction, we evaluate over three datasets: FUNSD [16], CORD [26], and SROIE [15]. Following [35, 34, 14], we build a token classification layer on top of the multi-modal encoder, and predict the BIO tags for each entity field for FUNSD, CORD and SROIE. The weighted F1 score is used as the evaluation metric. Following StructuralLM [19] and LayoutLMv3 [14], we use the cell bounding box of each token in substitution of word bounding boxes. Similar to LayoutLMv2 [34], we use entity-level F1 score on SROIE, and correct OCR mismatch as the official OCR annotations are inconsistent with the test set provided by the official evaluation site. More details of these datasets can be found in Appendix A.
Results. According to Table 1, our model generally outperforms existing baselines on both scales. Specifically, we achieve 91.52 and 93.62 weighted F1 score on FUNSD for WUKONG-READERbase and WUKONG-READERlarge, respectively. Both results are 1.23 to 1.56 points higher than LayoutLMv3, the previous SOTA models on document understanding. On CORD, our models also achieve comparable performances to state-of-the-art methods like LayoutLMv3. For SROIE, we again lead the performance with 96.88 and 98.15 weighted F1 scores on the base and large model, superior to LayoutLMv2 by 0.63 and 0.34 points, respectively.

heading: 4.2.2 Document Classification.
text: Datasets and Evaluation Metric. For document classification, we use the RVL-CDIP dataset [13], which contains around 400K industrial documents in 16 classes. Following [34], we use the preencoder and post-encoder visual features, together with the [CLS] token of the multi-modal encoder for document classification. By default, we perform fine-tuning for 10 epochs over 8 Ascend 910 accelerators, with the batch size of 24 per accelerator. The classification accuracy is used for evaluation. We set the learning rate to 5e-5 with the same scheduler to pre-training, and the weight decay is 1e-2.
Results. From the last column in Table 1, the proposed WUKONG-READERbase and WUKONGREADERlarge achieve the competitive 94.91% and 95.26% accuracies among the baselines, and have
room for further improvement.

heading: 4.3 Discussions
text: Ablation Study of Training Objectives. We provide a comprehensive study on the effect of the different pre-training objectives on WUKONGREADERlarge over each downstream dataset. To better understand how these proposed objectives affect visual document understanding, we compare with the following settings: (i) the MLM objective; and (ii) the MLM and MRM objectives; and (iii) the MLM, MRM and TRC objectives; and (iv) the MLM, MRM, TRC and TGM objectives.
From Table 2, it can be found that training with only MLM objective leads to a significant performance drop. When MRM is used, the performance of each task is consistently improved, e.g., 2.27 and 3.36 F1 scores on FUNSD and CORD, respectively. Moreover, the TRC objective enhances the fine-grained visual and textual representation learning, and further improves the F1 score of FUNSD by 0.84. Finally, the TGM objective can further boost the performance of sequence labeling tasks, improving the F1 score by 0.81 on FUNSD.
Further Analysis of MRM. We visualize the training curves of both total loss and MLM loss in Figure 3(a) and Figure 3(b). It can be found that with only the MLM objective, the training fails as a result of NaN errors at early training steps, as indicated by the red ×. Thus we have to lower the learning rate to 1e-5 to finish the pre-training. However, when armed with MRM loss, the training stabilize and the overall process can be easily finished with a larger learning rate of 1e-4. We hypothesize that the enhanced visual features can help stabilize the pre-training. In addition, the MRM objective significantly improves the task performance. We notice that even only using self-reconstruction losses such as MLM and MRM, the pre-trained model can still achieve a relatively good perfor-
mance. It shows the self-reconstruction objective on each separate modality serves to facilitate the implicit cross-modal interaction.
Visualization of TRC. We also study the WUKONG-READER’s capability of capturing finegrained cross-modal localization of textlines. We use the WUKONG-READERlarge model, and visualize the textline-region alignment in Figure 4, where the green and red boxes denote the correctly and incorrectly aligned pairs. Following [37], we compute the alignment scores of textual and visual textline representations based on Equation (2). From Figure 4, WUKONG-READER automatically learns to align the textline with its corresponding visual regions, with above 80+% accuracies. The learned alignment between two modalities implicitly explains the powerful effect of WUKONGREADER in various downstream tasks. This ability of WUKONG-READER provides a promising multimodal solution towards document localization tasks, instead of using naive text matching based on OCR results.
Model Efficiency With superior performance on information extraction and classification in Section 4.2, WUKONG-READER has similar parameters size (i.e., 211M and 442M) with other base-
lines, according to Table 1. As mentioned in Section 3.1, the model architecture is also computationally cheaper. For instance, it takes 375.4G FLOPs for the WUKONG-READERlarge model, while LayoutLMv2 and LayoutLMv3 require 379.0G and 431.7G FLOPs, respectively. We also compare the inferece latency: WUKONG-READER takes 340ms and 794ms for the base and large models, both of which are faster than LayoutLMv3, i.e., 346ms and 964ms for the base and large sizes respectively.

heading: 4.4 WukongReader in Chinese
text: We also evaluate WUKONG-READER on Chinese VDU tasks. We collect a large-scale Chinese document collection with 8 million documents, and follow the same pre-training setting as the English the IIT-CDIP Test Collection dataset [18]. The details of data collection are left in Appendix C. We intialize the backbone with XLM-Roberta [6] for the text and multimodal encoders3. We also follow XLM-Roberta to use SentencePiece with a unigram
3XLM-Roberta-base and XLM-RoBERTa-large are downloaded from https://huggingface.co/ xlm-roberta-base and https://huggingface. co/xlm-roberta-large, respectively.
language modela as the tokenizer. We fine-tune WUKONG-READER on two Chinese VDU tasks: 1) XFUND [36] for information extraction and relation extraction; 2) EPHOIE [31] for information extraction. For XFUND, we follow the language-specific fine-tuning setting in [36], i.e., the fine-tuning and testing are both operated on the Chinese subset of XFUND. We report F1 scores for evaluation. More details regarding to these two datasets are be found in Appendix A.
Results. Table 3 shows that our WUKONGREADERbase and WUKONG-READERlarge still achieve superior performance on most metrics in Chinese document understanding tasks. In particular, our model takes a new state-of-the-art F1 score of 85.82% on the relation extraction of XFUND and 99.63% on the EPHOIE dataset. Note that LayoutXLM [36] and LayoutLMv3 [14] use 30 million and 50 million documents for pre-training, which are far more than our 8 million data. Therefore, this verifies that our fine-grained pre-training objectives are more data-efficient and effective.

heading: 5 Conclusion
text: In this paper, we propose WUKONG-READER, a multi-modal pre-trained model for fine-grained visual document understanding. Unlike existing solutions that ignore the intrinsic textual segment information, our WUKONG-READER aims to leverage the semantics in textline regions of documents, by aligning with the visual and textual contents over document textlines via textline-region contrastive learning. Meanwhile, we also propose masked region modeling and textline grid matching to further enhance the visual and layout information of document textlines. We evaluate WUKONG-READER on various visual document understanding tasks such as information extraction and document classification, and the proposed model demonstrates superior performance against previous counterparts.
Limitations
The potential limitations of this work are that Wukong-Reader has fixed sequence length that may prevent it from modeling long and multi-page documents. Therefore it would be promising to handle varying-length inputs for Wukong-Reader by, for instance, equipping the model with relative positional embeddings of the model backbone. Additionally, the pre-training objectives used in this
work may not be applicable to all VDU tasks. For instance, it can be hardly applied in abstractive question answering or document summarization.
Ethics Statement
Wukong-Reader inherits the publically released RoBERTa model [24], where the checkpoint may contain some harmful information learned from the pre-trained corpus. Meanwhile, Wukong-Reader is further pre-trained on the IIT-CDIP Test Collection dataset [18] or the collected large-scale Chinese document corpus, where there can be also improper expressions. Although we have developed rules to manually filter out harmful expressions from the OCR-recognized texts during pre-processing, it is not guaranteed that all harmful information can be removed.

heading: Acknowledgement
text: We gratefully acknowledge the support of MindSpore for this research, as well as the insightful suggestions from the anonymous reviewers.
References [1] Mindspore. https://www.mindspore.cn.
[2] Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. Docformer: End-to-end transformer for document understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 993–1003, 2021.
[3] Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, et al. Unilmv2: Pseudomasked language models for unified language model pre-training. In International Conference on Machine Learning, pages 642–652. PMLR, 2020.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 213–229, Cham, 2020. Springer International Publishing.
[5] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016.
[6] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised
cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online, July 2020. Association for Computational Linguistics.
[7] Lei Cui, Yiheng Xu, Tengchao Lv, and Furu Wei. Document ai: Benchmarks, models and applications. arXiv preprint arXiv:2111.08609, 2021.
[8] J. Devlin, M. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[10] Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Minzhe Niu, Hang Xu, Xiaodan Liang, Wei Zhang, Xin Jiang, and Chunjing Xu. Wukong: 100 million large-scale chinese cross-modal pre-training dataset and a foundation framework. arXiv preprint arXiv:2202.06767, 2022.
[11] Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. Unidoc: Unified pretraining framework for document understanding. Advances in Neural Information Processing Systems, 34:39–50, 2021.
[12] Zhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, and Liqing Zhang. Xylayoutlm: Towards layout-aware multimodal networks for visually-rich document understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4583– 4592, 2022.
[13] Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In 2015 13th International Conference on Document Analysis and Recognition, pages 991–995, 2015.
[14] Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia, page 4083–4091, 2022.
[15] Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. Icdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition, pages 1516–1520. IEEE, 2019.
[16] Guillaume Jaume, Hazim Kemal Ekenel, and JeanPhilippe Thiran. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops, volume 2, pages 1–6. IEEE, 2019.
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[18] David Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, David Grossman, and Jefferson Heard. Building a test collection for complex document information processing. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 665–666, 2006.
[19] Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. StructuralLM: Structural pre-training for form understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistic, pages 6309– 6318, 2021.
[20] Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei. Dit: Self-supervised pretraining for document image transformer. In Proceedings of the 30th ACM International Conference on Multimedia, page 3530–3539, New York, NY, USA, 2022. Association for Computing Machinery.
[21] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pretraining for unified vision-language understanding and generation. arXiv preprint arXiv:2201.12086, 2022.
[22] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Systems, volume 34, pages 9694–9705, 2021.
[23] Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. Selfdoc: Self-supervised document representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5652–5660, 2021.
[24] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
[25] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021.
[26] Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. Cord: a consolidated receipt dataset for post-ocr parsing. In Workshop on Document Intelligence at NeurIPS 2019, 2019.
[27] Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang, Teng Hu, Weichong Yin, Yongfeng Chen, Yin Zhang, et al. Ernielayout: Layout knowledge enhanced pre-training for visually-rich document understanding. arXiv preprint arXiv:2210.06155, 2022.
[28] Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, and Gabriela Pałka. Going full-tilt boogie on document understanding with text-image-layout transformer. In International Conference on Document Analysis and Recognition, pages 732–747. Springer, 2021.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
[30] Jiapeng Wang, Lianwen Jin, and Kai Ding. LiLT: A simple yet effective language-independent layout transformer for structured document understanding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7747–7757. Association for Computational Linguistics, May 2022.
[31] Jiapeng Wang, Chongyu Liu, Lianwen Jin, Guozhi Tang, Jiaxin Zhang, Shuaitao Zhang, Qianying Wang, Yaqiang Wu, and Mingxiang Cai. Towards robust visual information extraction in real world: new dataset and novel solution. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2738–2745, 2021.
[32] Jiapeng Wang, Tianwei Wang, Guozhi Tang, Lianwen Jin, Weihong Ma, Kai Ding, and Yichao Huang. Tag, copy or predict: A unified weakly-supervised learning framework for visual information extraction using sequences. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 1082– 1090. International Joint Conferences on Artificial Intelligence Organization, 8 2021. Main Track.
[33] Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, and Furu Wei. LayoutReader: Pre-training of text and layout for reading order detection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4735–4744, 2021.
[34] Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. LayoutLMv2: Multi-modal pre-training for visuallyrich document understanding. In Proceedings of the
59th Annual Meeting of the Association for Computational Linguistics, pages 2579–2591, 2021.
[35] Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1192–1200, 2020.
[36] Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, and Furu Wei. Layoutxlm: Multimodal pre-training for multilingual visually-rich document understanding. arXiv preprint arXiv:2104.08836, 2021.
[37] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. In International Conference on Learning Representations, 2022.
[38] Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for document layout analysis. In 2019 International Conference on Document Analysis and Recognition, pages 1015– 1022. IEEE, 2019.

heading: A Downstream Tasks
text: We comprehensively evaluate WUKONG-READER on various VDU tasks in both English and Chinese. We summarize the downstream dataset used for evaluation as follows.
FUNSD [16] consists of noisy scanned documents and aims at understanding the structure of textual content of forms. It contains 199 fully labelled real scanned images, including 149 training samples and 50 test documents. We follow [34] to use the entity-level F1 to evaluate the model performance.
CORD [26] is a consolidated dataset for receipt parsing. CORD collected over 11,000 Indonesian receipt images from shops and restaurants. The dataset comprises 800, 100, and 100 receipt samples for training, validation, and testing. We adopt entity-level F1 and transcript of CORD for training and evaluation.
SROIE [15] contains 1000 scanned receipt images for text recognition and key information extraction. SROIE annotated 626 and 347 receipts for training and test, respectively. The dataset labelled four entities: company, date, address, and total. We correlate the entity annotation files with OCR results to generate ground-truth BIO labels for training and testing. During inference, we extract entities according to BIO labeling results and employ the entity-level F1 for evaluation. We use the official OCR annotations, however which contain OCR mismatch and are inconsistent with test set provided by the official evaluation site. Therefore, LayoutLMv2 [34] and other top methods on SROIE leaderboard4 claim to exclude OCR mismatch and fix total entities. We thus follow the same evaluation protocol as these methods to correct OCR mismatch via post-processing on entities.
RVL-CDIP [13] contains around 400K industrial document images in 16 classes, such as forms, advertisements, and letters, among which 360K and 40K are selected for training and testing. We extract text and layout information using Huaweideveloped text recognition algorithms. We use the overall classification accuracy as the evaluation metric. We use the official OCR annotations, however which are inconsistent with test set provided
4https://rrc.cvc.uab.es/?ch=13&com= evaluation&task=3
by the official evaluation site. We thus follow LayoutLMv2 [34] to post-process extracted entities and correct OCR mismatch.
PubLayNet [38] is a collection of research paper documents, with 355,703 training images and and 11, 245 validation images, respectively. The annotation of the dataset follows the object detection task of MS COCO, where each object is assigned with a bounding box and one of the five categories: figure, list, text, table and title.
DocVQA [25] contains 50,000 manually designed questions over 12,767 industrial document images. These scanned documents include various categories: figure/diagram, form, table/list, layout, free text, image/photo, handwritten characters, yes or no and others. We use the Microsoft OCR tool to extract the text and their bounding boxes. We also re-organize the OCR recognized text based on reading order of human, i.e., we heuristically cluster the word bounding box based on their intervals. This can be beneficial for documents with irregular layouts. For instance, reading from left to right in double column documents may fail to produce natural text.
XFUND [36] is a multilingual form understanding benchmark dataset including 7 languages (Chinese, Japanese, Spanish, French, Italian, German, Portuguese). This dataset provides humanannotated key-value pairs from form documents, and thus the goal is to perform key-value extraction with two sub-tasks: semantic entity recognition and relation extraction. The experiment in this paper only uses the Chinese source, which has 10288 and 3629 entities for training and test set, respectively. We follow the same methods as used in LayoutXLM [36] to perform these two sub-tasks.
EPHOIE [31] is a visual information extraction dataset on Chinese examination paper heads. It contains 1,494 images (1183 for training and 311 for testing) with human annotations for 15,771 Chinese text instances. Similar to the semantic entity recognition in XFUND, EPHOIE is also a token-level entity labelling task with ten pre-defined categories, and thus we use the same fine-tuning method for SER and EPHOIE.

heading: B More Experiments
text: B.1 Layout Analysis
Datasets and Evaluation Metric. We use the PublayNet dataset [38] for layout analysis. Following standard practice of object detection, we use the mean average precision (MAP) and intersection over union (IOU) [0.50:0.95] of bounding boxes to evaluate the model performance.
In contrast to previous methods [14, 11] that solely employ the vision encoder to detect document elements, we propose to reuse the multimodal features for layout analysis task to explore the effectiveness of our multi-modal encoder. Specifically, we design a feature selection decoder similar to [4] on top of the multi-modal encoder, which enables us to detect document layouts using both vision and text. We apply the ADAM optimizer with a total batch size of 32 over 8 AI processors. Both the base learning rate and weight decay are set to 4e-4, and a linear learning rate scheduler is used. We train the model for 10 epochs on the training set and evaluate the performance on the validation set.
Results. We demonstrate the results on Publaynet in Table 4. We first feed only the visual backbone of Wukong-Readerbase to the transformer decoder following[4], which achieves 95.3 mAP scores and leads the previous vision-based methods [14, 11, 20]. Additionally, we can further improve the mAP score to 96.0 when employing the multi-modal fea-
tures by Wukong-Readerbase, outperforming the rest baselines with a clear margin. This again verifies the effectiveness of the learned multi-modal representations from WUKONG-READER.
B.2 Document Question Answering
Datasets and Evaluation Metric. For document question answering, we use the DocVQA dataset [25], which contains 50,000 questions over 12,000 pages of various industrial documents. We use the official website for evaluation5, which compares the extracted answer span with the ground-truth and reports the averaged normalized Levesitein distance (ANLS).
Results. The results on DocVQA are listed in Table 5. For LayoutLMv2-base [34], we report the best reproduced result marked as ∗. As suggested by existing methods [34], leveraging the additional techniques of post-processing, data augmentation and model ensemble contributes a lot to this performance, while we leave this exploration to the future work. Overall, our WUKONG-READERbase and WUKONG-READERlarge achieve 74.1 and 78.9 ANLS score, respectively. This is comparable to
5https://rrc.cvc.uab.es/?ch=17&com= introduction
the competitive LayoutLMv2 without using additional techniques. For instance, LayoutLMv2 is initialized from UniLMv2 [3] that naturally owns a more powerful question answering ability than RoBERTa. Unfortunately, we are unable to access UniLMv2 model since it is not publicly released yet and thus our model was initialized from RoBERTa. We also visualize the ANLS score of each class in DocVQA returned by our WUKONG-READERlarge in Figure 5. Our model can perform reasonably well on “Form” and “Layout” with around 80.0 ANLS scores, yet there is still room for improvement for categories such as “Figure” and “Image”.

heading: C Preparing the Chinese Pre-training Document Corpus
text: We also collect a 8 million Chinese document corpus to validate WUKONG-READER in Chinese. The collection comes from various resources: the Chinese documents from Common Crawl6; the static Chinese HTML dumps of Wikipedia7 and publicly available Chinese digital books, contracts and IPO (Initial Public Offering) documents via the official Chinese websites.8 To obtain the textual and layout information of the collected documents, we first obtain the character-level bounding boxes via OCR. Then we calculate the bounding box of each token by merging the bounding boxes of all characters it contains. Thus the output format is consistent with those in the IIT-CDIP Test Collection dataset [18].
6https://commoncrawl.org/ 7https://dumps.wikimedia.org/ 8http://www.sse.com.cn/ & http://www.szse.cn/
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: The second to last section.

heading: 3 A2. Did you discuss any potential risks of your work?
text: The last section.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Introduction: Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We keep the same random seed to fairly compare with other baselines.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: As suggested by existing methods [34], leveraging the additional techniques of post-processing, data augmentation and model ensemble contributes a lot to this performance, while we leave this exploration to the future work. Overall, our WUKONG-READERbase and WUKONG-READERlarge achieve 74. 1 and 78. 9 ANLS score, respectively. This is comparable to
5https://rrc. cvc. uab. es/?ch=17&com= introduction
the competitive LayoutLMv2 without using additional techniques. For instance, LayoutLMv2 is initialized from UniLMv2 [3] that naturally owns a more powerful question answering ability than RoBERTa. Unfortunately, we are unable to access UniLMv2 model since it is not publicly released yet and thus our model was initialized from RoBERTa. We also visualize the ANLS score of each class in DocVQA returned by our WUKONG-READERlarge in Figure 5. Our model can perform reasonably well on “Form” and “Layout” with around 80. 0 ANLS scores, yet there is still room for improvement for categories such as “Figure” and “Image”