StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-training
Authors: Yuechen Yu
Yulin Li
Chengquan Zhang
Xiaoqiang Zhang
Zengyuan Guo
Xiameng Qin
Kun Yao
Junyu Han
Errui Ding
Jingdong Wang
Venue: ICLR 2023 poster (2023)

Abstract: In this paper, we present StrucTexTv2, an effective document image pre-training framework, by performing masked visual-textual prediction. It consists of two self-supervised pre-training tasks: masked image modeling and masked language modeling, based on text region-level image masking. The proposed method randomly masks some image regions according to the bounding box coordinates of text words. The objectives of our pre-training tasks are reconstructing the pixels of masked image regions and the corresponding masked tokens simultaneously. Hence the pre-trained encoder can capture more textual semantics in comparison to the masked image modeling that usually predicts the masked image patches. Compared to the masked multi-modal modeling methods for document image understanding that rely on both the image and text modalities, StrucTexTv2 models image-only input and potentially deals with more application scenarios free from OCR pre-processing. Extensive experiments on mainstream benchmarks of document image understanding demonstrate the effectiveness of StrucTexTv2. It achieves competitive or even new state-of-the-art performance in various downstream tasks such as image classification, layout analysis, table structure recognition, document OCR, and information extraction under the end-to-end scenario.

Published as a conference paper at ICLR 2023
STRUCTEXTV2: MASKED VISUAL-TEXTUS PREDIC- TION FOR DOCUMENT IMAGE PRE-TRAINING
Yuechen Yu', Yulin Li", Chengquan Zhang', Xiaoqiang Zhang, Zengyuan Guo, Xiameng Qin, Kun Yao, Junyu Han, Errui Ding, Jingdong Wang
Department of Computer Vision Technology (VIS), Baidu Inc.
tree/ httpee/mmiintttueeeeeeeee https:/ pttps://giae http://ggtttttbbtt com wwwpo///aaaaledddleeeeeeeaeeeeee
ABSTRACT
In this paper, we present Veruc,e an effective document image pre-training framework, by performing masked visual-textual prediction. It consists of two self-supervised pre- tasks: masked image modeling and masked language modeling, based on text region-level image masking. The proposed method ran- domly masks some image regions according to the bounding box coordinates text words. The objectives of pre-training tasks are reconstructing the pixels masked image regions and the corresponding masked tokens simultancouully Hence the pre-trained encoder can capture more textual semantics in comparison to the masked image modeling that usually predicts the masked image patches. Compared to the masked multi-modal modeling methods for document image derstanding that rely on both the image and text modalities, StrucTexTv2 mod- els image-only input and potentially deals with more application scenarios free from OCR pre- pre-processing Extensive experiments on mainstream benchmarks of document image understanding demonstrate the effectiveness of StrucTexTv2. It achieves competitive stream tasks such as image classification, layout analysis, table structure recogni- tion, document OCR, and information extraction under the end-to-end scenario.
even new state-of-the-art performance in various down-
INTRODUCTION
In the Document Artificial Intelligence, how understand visually-rich document images and ex- tract structured information from them has gradually become popular research topic. Its main associated tasks include document image classification Harley al. (2015), layout analysis Zhong et al. (2019), form understanding Jaume et al. (2019), document OCR (also called text spotting) Li et al. (2017); Liao et al. (2021), and end-to-end information extraction (usually composed OCR and entity labelling phrase) Wang al. (2021), etc. To solve these tasks well, it is necessary fully exploit both visual and textual cues. Mcanwhile, large-scale supe-vupe pre-traue vre-training Li et al. (2021a); Appalaraju al. (2021); Xu et al. (2020; 2021); Huang et al. (2022); Gu et al. (2021) is recently rising technology to enhance multi-modal knowledge learning of document images.
There mainstream self-supervised pre- pre-tiaining frameworks for document image understand- ing. As illustrated in Fig. 1: (a) The first category is the masked multi-modal modeling such as the proposed pre-training tasks: MLM Devlin al. (2019), MVLM Xu al. (2021), MM-MLM Ap- palaraju et al. (2021) and MSM Gu et al. (2021), whose inputs mainly consists of OCR-extracted texts and image embeddings. The methods collect semantic information from text and image. de- pending heavily on front-end OCR engines with certain computing costs. Additionally, the two components of OCR engine and document understanding module are separately optimized, which is hard to ensure performance of the whole system. (b) The second category is the masked image modeling (MIM) that inherits the concept of vision-based self-supervised learning such as BEiT Bao et al. (2022), SimMIM Xie et al. (2022), MAE He et al. (2022), CAE Chen et al. (2022), and DiT Li et al. (2022), etc. MIM is powerful image-only pre-training technique to learn the visual contex- tualized representations of document images. Because of the insufficient consideration of textual
Equal contribution. Correspondence to: Chengquan Zhang</hhangehengguaeee wbaidu.com22

---

Published as a conference paper at ICLR 2023
OCR Results
Encoder
Information Extraction
(a)
MLM, MVLM, etc.
Image Classification
Image Embeddings
Document VQA
Encoder
Image Embeddings
Image Classification
MIM. MAE, etc.
Layout Amalysis
Image Clussification
Enender
Layout Analysis
Image Embbedingg
(c)
MIM-MLM
Information Extraction
E2E Information Extraction
Table Structure Recognition
Document image
Input modality
Pre-training tasks
Downstrean tasks
Figure 1: Comparisons with the main-stream pre- vre-tranning models of document image understanding. (a) It shows the masked multi-modal modeling methods which input both OCR results and image embeddings. (b) The framework that inputs image-only embeddings is suitable for vision-dominated tasks like document image elassification and layout analysis. (c) StrucTexxTv2 learns visual-textual representations using only the information from images in the pre-training step and then optimizes various downstream tasks of document image understanding end-to-end.
contents, MIM often applies some vision-dominated tasks Li al. (2022) like image classifica- tion, layout analysis and table detection. (c) To take full advantage of multi-modality knowledge on the basis of MIM, we propose the third pre-training framework by learning visual-textual repre- sentations with image-only inputs, to optimize the performance of document image understanding tasks in an end-to-ena manner.
Due to the great disparities between vision and language, the existing document understanding meth- ods either consider single modality introduce an OCR engine to capture textual content in ad- vance. Researchers used text tokens as the input in language modeling, or selected fixed-size image patches as the granularity of vision pre-training tasks. However, the textual content is visually- situated in document and extracted from the image. Thus, we propose the text region-level image masking scheme corresponding to document content to bridge vision modeling to language model- ing with the shared representations.
This paper proposes StrucTexTv2. a novel multi-modal knowledge learning framework for docu- ment image understanding by performing text region-level image masking with dual parallel self- supervised tasks of image reconstruction and language modeling (as shown in Fig. 2). First off, we adopt an off-the-shelf OCR toolkit perform word-level text detection and text recognition pre-training dataset (ITTCDIIP Test Collection Lewis et al. (2006)). Next, we randomly mask some text word regions given the input images and fed them into the encoder. Finally, the pre-training ob- jectives of StrucTexTv? learn to reconstruct image pixels and text content of the masked words. In support of the proposed pre-training tasks, we introduce a new backbone network for Vv In particular, CNN-based network with the Rol-Align He et al. (2017) operation produces vi- sual features for the masked regions. Inspired by ViBERTGrid Lin al. (2021), the backbone uses FPN Lin et al. (2017) to integrate features of CNN. The following transformer model enables capturing semantical and contextualized representations from the visual features. We evaluate and verify pre-trained model in five tasks including document image classification, layout analy- sis, table structure recognition, document OCR, and end-to-end information extraction, all which have achieved significant gains. The experimental results have also confirmed that the framework of StrucTeex Tv2 can construct fundamental pre- pre-ted models for document image understanding.
the
The major contributions of our work be summarized following:
A novel supersupe pre-training framework by performing text region-level document image masking, named StrucTexTv2, is proposed to learn iisual-textual representations in end-to-end manner.
2

---

Published as a conference paper at ICLR 2023
Superior performance in five downstream tasks demonstrates the effectiveness of the Struc- Tvx pre- trained model in document image understanding.
2 RELATED WORK
Sulf-super Learning Thanks the development of self-supervised tasks and Transformer architectures, in the past few years, the computer vision (CV) and natural language processing (NLP) have achieved breakthroughs in knowledge learning from large-scale unlabeled data. In the domain of NLP, Masked Language Modeling (MLM) task has been widely used in pre-trained models Devlin et al. (2019); Radford et al. (2018) due to its efficiency and effectiveness. The MLM task randomly masks a set of text tokens of input and reconstructs them according to the context around the masked tokens. In the CV domain, taking a similar idea, Masked Image Modeling (MIM) has also been successfully verified. There several variants of MIM, for example, BEiT Bao et al. (2022) and CAE Chen et al. (2022) randomly mask a percentage of image patches and predict discrete image patch tokens learned by dVAE Ramesh et al. (2021). MAE He et al. (2022) and SimMIM Xie et al. (2022) take a simpler behavior, predicting RGB values of raw pixels by direct regression.
Document pre-trained models As stated above, the cutting-edge document pre-training models can be roughly separated into two categories: masked multi-modal modeling and masked image model- ing approaches. The representative works of the former are LayoutLM series Xu et al. (2020; 2021); )); Huang et al. (2022), Docformer Appalaraju et al. (2021), UDoc Gu et al. (2021) and StrucTexx Li al. (2021c), they usually depend on a mature OCR engine to extract textual content from docu- ment images, and fed them to the encoder combined with image embeddings. These OCR-based methods promise excellent performance, but they are cumbersome to solve some vision-dominated tasks. The latter methods, such as DiT Li et al. (2022) almost following the idea of BEiT Bao et al. (2022), directly use a general CV pre-training framework to learn on large-scale document image data. Due to the lack of guidance for text information, the model is likely to be deficient in semantic understanding. Recently, wrompt-learning Liu et al. (2023) has been studied for adapting pre-trained vision- models. Those coompt-basets methods Kim et al. (2022); Davis et al. (2022) directly generate textual output from documents and achieve competitive performance on downstream tasks.
In this paper, the proposed SerucTexxTv2 is new solution that integrates the advantages of CV and NLP pre-training methods in an end-to-end manner. Benefiting from image-only input of the encoder, our framework can avoid the interference of false OCR results compared with the OCR- based pre-trained models. For our pre-training, although the supervision labels partially come from OCR results, only the high-confidence words from OCR results are randomly selected. The impacts of the OCR quality on our pre-trained model is alleviated to a certain extent.
3 APPROACH
3.1 MODEL ARCHITECTURE
As illustrated in Fig. 2, there are two main components of StrucTexxTv2. a encoder network using FPN integrate visual features and semantic features, and the pre-training framework containing two objectives: Mask Language Modeling and Mask Image Modeling.
The proposed encoder consists of a visual extractor (CNN) and a semantic module (Transformer). Given an input document image, StrucTexxTv2 extracts visual-textual representations through this backbone network. Specifically, the features of the last four down-sampled stages on CNN are ex- tracted from the visual extractor. In the semantic module, following ViT Dosovitskiy et al. (2021) to handle 2D feature maps, the features of the last stage in CNN are flattened in patch-level and are linearly projected to obtain 1D sequence of patch token embeddings, which also serves as the effective input for the Transformer. A relative position embedding representing the token index is added to the token embeddings. Then, the standard Transformer receives the input token embed- dings and outputs the semantic enhanced features. We reshape the output features back to context feature maps in the 2D visual space and up-sample the feature maps with the factors of 8. We adopt FPN strategy Lin et al. (2017) to merge visual features of different resolutions from CNN and then concatenate context feature maps with them, deriving feature map with 1/4 size of the input image.
3

---

Published as a conference paper at ICLR 2023
Finally, fusion network which consists of two successive 1x1 convolutional layers is introduced to take a further multi-modal fusion.
MIM Branch
Experiments
CNN
Fine-tuning
Trandorner
Tasks
ROI Alignmenttt
"Eperiments"
Fine-tuning" "Tasks"
FPN Fasion
Document image with masked text regions
MLM Branch
The overview of StrueTexTv2. Its encoder network consists of a visual extractor (CNN) and a semantic module (Transformer). Given document image, the encoder extracts the visual feature of the whole image by CNN and obtains the semantic enhanced feature through a Trans- former. Subsequently, lightweight fusion network is utilized to generate the final representation of the image. With the help of ROI Alignment, the multi-modal feature of each masked text region is processed by the MIM branch and the MLM branch reconstruct the pixels and text, respectively.
Figure 2:
3.2 PRE-TRAINING
To enhance the document image understanding of Stru we perform pre-training large- scale document dataset IIT-CDIP Test Collection 1.0 dataset Lewis et al. (2006). Unlike MAE He al. (2022), BEiT Bao al. (2022), and DIT Li al. (2022), we do not use patch-level masking strategy for pre-training. Instead, we use a novel text region-level masking strategy and employ two sele-super pre-training tasks to learn visual-textual representations. The subsequent experiment results suggest that the fine-graines text region-level masking strategy is more suitable than the coarse-grained patch-level masking strategy in document understanding.
3.2.1 TASK #1: MASKED LANGUAGE MODELING
To encourage the model to learn contextual representation, similar to BERT Devlin et al. (2019), mask a portion of the text region of the input document with RGB values [255, 255, 255] random. We find that 30% of masking ratio is the best choice in our experiments. According to the contextual information of the unmasked regions in a document, the encoder-decoder architecture learns to predict the indexes of text tokens (in a pre-defined vocabulary) of the masked text regions with cross-entropy loss. To avoid the interference of sub-words produced by WordPiece Song et al. (2021), we only choose the first sub-word in each word for keeping the number of words in the document images. Notice that the Masked Language Modeling task in StrucTexTv2 does not require the text input, however, which is essential for the NLP domain.
Decoder for Task #1. For Mask Language Modeling, we employ an extremely 2-layer MLP as decoder to project the encoding feature. The multi-model feature of each text region is extracted from the fused feature maps Juse by ROI-Align Ren et al. (2017), which is computed as follows,
pmum
MTP MT.P fusez b,)),
where b; is the bounding box of the ith text region, and the pmum is optimized by cross-entropy loss with 30,522 token categories.
3.2.2 TASK #2: MASKED IMAGE MODELING
In MAE He et al. (2022), BETT Bao et al. (2022), and SimMIM Xie et al. (2022), patch-level Masked Image Modeling has shown strong potential in representation learning. However, in the document understanding domain, leteh-level feature learning is too coarse to represent the details text or word region. Therefore, introduce text region-level visual representation learning task called
4

---

Published as a conference paper at ICLR 2023
Masked Image Modeling to enhance document representation and understanding. Instead of clas- sifying the classes defined by tokenization like Lvv and BEiT. we regress the raw pixel values of the masked text region with mean square error loss following SimMIM and MAE. Specif- ically, we mask the rectangle text regions and predict the RGB values of missing pixels, leading to significant improvement for performance in representation learning.
Decoder for Task #2. We develop a Fully Convolutional Network (FCN) with Transpose Convo- lution carry out the document image reconstruction of masked text regions. Specifically, we apply the global average pooling to aggregate each text region's 's feature and generate the embedding Embstyee that mainly represents the visual "style" for each masked text region. To strengthen its text information, we encode the MLM prediction pmumm content by using an embedding layer, denoting the "content" knowledge. At last, we concatenate Embsstle and Emb and feed it to FCN, generating the final restored image prediction P.aim, The procedure of Mask Image Modeling can be formulated as follows,
Emb = GAP(ROT-Align fuse:: bc)),
(2)
content = EmbeeddinLLayer(PP (P.ml/)
(3)
Pmim FCN EEmbstyle Emb
where GAP is the global average pooling operator. In MIM, we follow MAE and predict the missing pixels masked text regions. For example, we resize the spatial resolutions each masked text region to fixed values. The Prim is optimized by MSE loss in the pre-training phrase.
and each text-region's 's regression target is 12,288=64x644 x4 pixels of RGB
3.3 DOWNSTREAM TASKS
The VtrucTexxT22 pre-training scheme contributes to visual-textual representation with input of image-only. The multi-modal representation available to model fine-tuning and profits numerous downstream tasks.
3.3.1 TASK #1: DOCUMENT IMAGE CLASSIFICATION
Document image classification aims to predict the category of each document image, which is one of the fundamental tasks office automation. For this task, we downsample the output feature maps of backbone network by four 3x3 convolutional layers with stride 2. Next, the image representation fed into the final linear layer with softmax to predict the class label.
3.3.2 TASK #2: DOCUMENT LAYOUT ANALYSIS
Document layout analysis aims to identify the layout components of document images by object detection. Following DiT, we leverage Cascade R-CNN Cai & Vasconcelos (2018) as the detection framework to perform layout element detection and replace its backbone with SerucTexTv2. Thanks to the multi-scale context design of backbone networks, four resolution- moolutionnmodiiyinng features (P2~~P5) of FPN fusion layers backbone networks are sent into the iterative structure heads of the detector.
3.3.3 TASK #3 TABLE STRUCTURE RECOGNITION
Table structure recognition aims to recognize the internal structure of table which critical for document understanding. Specifically, We employ Cascade R-CNN for cell detection our table structure recognition framework while replacing the feature encoder with backbone networks. Since some table images are collected by cameras and many cells are deformed, we modify the final output of Cascade R-CNN to the coordinate regression of four vertexes of cells.
3.3.4 TASK #4: DOCUMENT OCR
We tend to read the text in an end-to-end manner based on StrucTexTv2. Our OCR method consists of both the word-level text detection and recognition modules. They share the features of backbone networks and are connected by ROI-Align operations. The text detection module adopts the standard DB Liao al. (2023) algorithm, which predicts the binarization mask for word-level bounding boxes. Similar to NRTR Sheng et al. (2019), the text recognition module is composed of multi-layer Transformer decoders to predict character sequences for each word.
5

---

Published as a conference paper at ICLR 2023
3.3.5 TASK #5: END-TO-ENT END INFORMATION EXTRACTION
The aim of the task to extract entity-level content of key fields from given documents without predefined OCR information. We evaluate the SerucTexTv2 model based on the architecture of document OCR and devise a new branch for semantic entity extraction. Concretely, another DB detection is developed to identify the entity bounding boxes. An additional MLP block is performed with the ROI features to classify entity label. These bounding boxes are utilized for word grouping to merge the text content from Task #4. At length the key information is obtained by grouping words according to the reading order.
4 EXPERIMENTS
4.1 DATASETS
Pre-training Data By following DiT Li et al. (2022), we pretrain StrucTexTv2 on the IIT-CDIP Test Collection 1.0 dataset Lewis et al. (2006), whose million multi-page documents are split into single pages, totally 42 million document images.
RVL-CDIP Harley et al. (2015) contains 400,000 grayscale document images in 16 classes, with 25,000 images per class. We adopt RVL-CDIP as the benchmark to carry out experiments docu- ment classification task. Average classification accuracy is used evaluate model performance.
PubLayNet Zhong al. (2019) consists of more than 360,000 paper images built by automatically parsing PubMed XML files. The five typical document layout elements (text, title, list, figure, and table) are annotated with bounding boxes. Mean average precision (mAP) @ intersection over union (IOU) is used as the evaluation metric of document layout analysis.
WTW Long et al. (2021) covers unconstrained table in natural scene, requiring table structure rec- ognizer to have both discriminative and generative capabilities. It has a total of 14581 images in a wide range of real business scenarios and the corresponding full annotation (including cell coordi- nates and row/column information) of tables.
FUNSD Jaume al. (2019) is form understanding dataset that contains 199 forms, which refers to extract four predefined semantic entities (questions, answers, headers, and other) and their link- ings presented in the form. We focus on two tasks of document OCR and end-to-end information extraction on FUNSD. For evaluation, we compute the normalized Levenshtein similarity (1-NED) between the predictions and the ground truth.
4.2 IMPLEMENTATION DETAILS
The proposed encoder network of StrucTexxTv2 is composed mainly of the CNN and Transformer. To balance efficiency and effectiveness, StrucTexxTv2sm.. consists of ResNet-50 and 12-layer Transformers (128 hidden size and attention heads) and introduces only 28M parameters. A larger version of StrucTexxT22 and 24-layer Transformers (768 hidden size and 8 attention heads), which total parameters are 238M. We use the networks trained with ImageNet Deng et (2009) as the initialization of CNNs. The Transformers are initialized from the language models Sun et al. (2020). StrucTexTv2sma. and StrucTexxT22.uee take 17 hours and 52 hours to train one epoch of the IIT-CDIP data, respectively. The whole pre-training phase takes nearly a week with 32 Nvidia Tesla 80G A100 GPUs.
Pre-training on IIT-CDIP
is set up as ResNeXt-101 Xie al. (2017)
Fine-tuning on RVL-CDIP We evaluate Vtr for document image classification. We fine- tune the model on RVL-CDIP for 20 epochs with cross-entropy loss. The learning rate is set for 3e-4 and the batch size 28. The input images are resized to 960x960 and maintain its aspect ratio. We use label smoothing=0.1 in the loss function. Besides, the data augmentation methods such as CutMix and MixUp with 0.3 probability are applied on the training phase.
Fine-tuning on PubLayNet We evaluate on the validation set of PubLayNet for document layout analysis. We fine-tune the Cascade R-CNN and initialize the backbone with our pre- pre-trained model. The detector is trained 8 epochs with Momentum optimizer and batch size of 8. The learning rate set to le-2, while it decays to 1e-3 on 3 epoch and decays le-4 on 6 epoch. We use random resized cropping to augment the training images while the short edges does not exceed 800 pixels.

---

Published as a conference paper at ICLR 2023
Performance comparisons on the RVL-CDIP dataset. We report classification accuracy on the test set. T and I denote the text and image modality of input. The proposed StrucTeexTv2 achieves a comparable accuracy the state-of-the-ar models with image-only input.
Table I:
Methods
Modalities
Accuracy
#Param.
BERT Base Declinel al. (2019)
T
89.81 %
110M
BERT Large Devlinet al. (2019)
T
89.92%
340M
StructuralLM Large Liet al. (2021a)
T
96.08%
355M
SelfDoc Li et al. (2021b)
V+T
92.81% %
LayoutLLMLaree Xuct al. (2020)
V+I
94.43%
390M
UDoc Gu et al. (2021)
V+T
95.05%
272M
TILT Large Powalskiet al. (2021)
V+T
95.52%
780M
LayoutLMv2t.arge Xuet al. (2021)
V+T
95.64%
426M
LayoutL Huanget al. (2022)
V+T
95.93%
368M
DocFormer Base Appalarajuel al. (2021)
V+T
96.17%
183M
BEiT Base Baoet al. (2022)
V
91.09%
87M
DiT Base Liet al. (2022)
V
92.11% %
87M
Large Liet al. (2022)
V 92.69%
304M
StrucTexTv2.ee
V
93.40%
28M
StrucTexTvv1..
V
94.62%
238M
Fine-tuning on WTW We conduct experiments on WTW for table structure recognition. We also employ Cascade R-CNN to detect the cells of table whose backbone is replaced by pre-trained StrucTeexTv2 We fine-tune our model end-to-end using ADAM Kingma Ba (2015) optimizer for 20 epochs with batch size of 16 and learning rate of le-4. The input images resized to 640 x 640 after random scaling and the long size being resized to 640 pixels.
Fine-tuning FUNSD On account of the full annotations, both the document OCR and end-to-end information extraction tasks are measured on FUNSD. We set the text recognition network as a 6- layers Transformer and fine-tune the whole model for 1200 epochs with batch size of 32. We follow a cosine learning rate policy and set the initial learning rate to 5e-4. Extra position embeddings are appended to roi-features and we pass it each layer decoders. The training losses except DB detector are the cross-entropy function. Additionally, the same loss is estimated for each decoder layer in the text recognition module for better convergence training.
4.3 COMPARISONS WITH THE SFATE-OF-THE--
To investigate the effect of visual-textual representations, we benchmark StrucTexTv2 with several state-of-the-art techniques on different downstream tasks. Since only small datasets (149 training documents on FUNSD) and to avoid exceeding GPU memory (a tremendous number of table cells on WTW), we only evaluate StrucTexI on the FUNSD and WTW datasets.
RVL-CDIP As the Tab, 1 shows, OCR-based approaches such as DocFormer Appalaraju et al. (2021) with multi-modal modeling takes superior performance on RVL-CDIP. Compared with image-only methods, it relies on stand-alone OCR systems which are low efficiency for practi- cal applications. Our StrucTexTv2smatl achieves an accuracy of 93.40% and outperforms previous image-only methods with much smaller model size. Besides, StrucTexxT22..... drives accuracy of 94.62%, which brings 1.93% improvement over DiTiarge Li al. (2022). Intuitively, the results show that StrucTexTv2 is effective for semantic understanding document image classification.
PubLayNet The experiment results on PubLayNet are presented Tab. 2. It is observed that Struc- TexTv2 achieves new state-of-the-art performance of 95.4% and 95.5% mAP scores for both small and large settings. Teuuxexlll2mll beats even LayoutL.Mv3... Huang et al. (2022) (the result of LayoutLMv3 is not released in the paper) which contains multi-mode modal inputs by 0.3%. We sug- gest that our dual-modal pre-tree training tasks can learn rich visual-textual representations of document images and performs excellently in confusing situations. Notably, TvruTTexTV2,..o. gets 0.1 1% mAP

---

Published as a conference paper at ICLR 2023
Table 2: Performance comparisons on the PubLayNet validation set. The mAP @ IOU (0.50:0.95] is used as the metric.
Table Performance comparisons on the WTW dataset. The FI-Score is used to measure the accuracy of cell co- ordinate when IOU=0.9.
Methods
mAP
Methods
F1 F1-Score
BEiT Base
93.1% %
Split-Heuristic Tensmeyer et al. (2019)
3.4%
UDoc
93.9%
Paster-RCNN Ren al. (2017)
64.4%
Large
94.9%
TridenNet Li et al. (2019)
65.0%
LayoutLMv3 Base
95.1%
CenterNet Duan al. (2019)
73.1%
StrucTexTvy
95.4%
Cycle- Tycle-CenterNer Long et al. (2021)
78.3%
StrucTexTv2..com %
Struc TexTy2 Small
78.9%
gain over that on StrucTexTT2Small We attribute this subtle improvement to the over-fitting to certain extent with increasing model size.
WTW Tab.3 shows quantitative results of table structure recognition on the WTW dataset. Struc- TexTv2 achieves the physical 78 78.9% Fi-score among all published methods. We reconstruct the ble structure based on the detection results of table cells. The superior performance of StrucTexxTv2 largely due to the proposed pre-training framework.
FUNSD We evaluate Struc'Tex TV2mml on both the document OCR and end-to-end information ex- traction tasks. As shown in Tab. 4, Vex achieves outstanding performance, 84.1% I-NED for document OCR and 55.0% 1-NED for information extraction. Significantly, the whole network is end-to-end trainable. Compared to StrucTexT Li et al. (2021c) and LayoutLMv3 with need of separately stage-wise training strategies, our model alleviates the error propagation in documental system with key information parsing.
Table 4: Performance comparisons FUNSD. We present the Normalized Edit Distance (1-NED) for the word-level document OCR and the entity-level information extraction. The * denotes a multi-stage process in which the methods are applied using our OCR results and entity boxes for word grouping in information extraction.
The results of Document OCR
The results of Information Extraction
Methods
1-NED
Methods
1-NED
DB+NRTR Sheng et al. (2019)
73.8%
StrucTexT Base
46.8%
Google Vision Google (2019)
76.4%
LayoutLMv3 Base
53.1%
StrucTexTv2ssmal
84.1% %
StrucTexxv2s..m
55.0%
4.4 ABLATION STUDY
To further examine the different contributions of StrucTexTv2, we conduct several ablation exper- iments, such as document layout analysis on PubLayNet, document image classification on RVL- CDIP, and end-to-end information extraction on FUNSD. All models in ablation study are pre- trained for epoch with only IM documents sampled from the IIT-CDIP dataset.
Table 5: The ablation study pre- pre-training tasks and different encoding structures.
Model
#Param. MIM MLM RVL-CDIP PubLayNet
SwinT base +FPN
176M
86.6%
92.2%
ViT TPPN +FPN
116M
88.6%
93.2%
StrucTex Tiexxvv22 T Large
238M
94.1 %
95.6 %
StrucTexxTve TV2 S Small
28M
91.8%
94.1%
StrucTexTv2.. small
28M
92.0%
94.5%
Ter2
28M
92.5%
94.9%
8

---

Published as a conference paper at ICLR 2023
Encoding Structures In this study, evaluate the impact of encoding structures by replacing the backbone of StrucTexTv2 with ViT Dosovitskiy et al. (2021) and Tranfrasffrmmrr Liu et al. (2021). As shown in Tab. 5, the proposed network of TerucTexxvv2eer 92.5% accuracy and 94.9% mAP on RVL-CDIP and Publ ayNet, respectively. The performance of two benchmarks dropped by 3.9% accuracy and 1.7% mAP with the BaT Replaced with the SwinTraansormerrees the degradation is more obvious. In addition, StrucTexTv2 improves performance by 1.6% on RVL-CDIP and 0.7% on PubLayNet.
even achieves the better results of
Pre-training Tasks In this study, we identify the contributions of different pre-training tasks. As shown in the bottom of Tab. the MIM-onll pre-trained model achieves an accuracy of 91.8% on RVL-CDIP and an mAP of 94.1% on PubLayNet. The MLM-only pre-trained model achieves 92.0% and 94.5% for the two datasets. The MLM and MIM can jointly exploit the multi-modal feature representations in Struc'TexTv2. By combining both the proposed pre-training tasks, the aceuracy is improved to 92.5% on RVL-CDIP and the mAP achieves 94.9% on PubLayNet.
Table 6: The ablation study the influence of masking ratios (MR.) with StrueTexTV2 on RVL- CDIP and PubLayNet.
Table 7: Consumption analysis RVL-CDIP. We implement LayoutLMv3 with open-source OCR engines to provide text. denotes the cost of OCR process. All the models are inferred NVIDIA Tesla 80G A100.
MR. RVL-CDIP PubLayNet
Methods
OCR
GPU(MB)
Time(ms)
0.15
92.1%
94.7%
LayoutLMv3 Base
Tesseract 2,184+n/a
22+1,105f
0.30
92.5%
%
LayoutLMy33 Base
PaddleOCR
2,184+3,450*
22+252t
0.45
91.7%
94.8%
StrucTexxTv2, Syaalll
None
2,276
56
0.60
92.4%
94.8%
Struc'eexTvv Large
None
4,058
284
Masking Ratios We investigate the effect of training with different masking ratios. As shown in Tab.6, by replacing the masking ratio with 0.15, 0.30, 0.45 and 0.60, the accuracy of RVL-CDIP is 92. 1%, 92.5%, 91.7% and 92.4%, respectively. We also report the results on PubLayNet, the mAP of PubLayNet is 94.7%, 94.9%, 94.8%, and 94.8%, respectively. It suggests that the best masking ratio of our pre-training tasks is 0.30. At the same time, It also suggests that the performance of downstream tasks is less sensitive to the selection of masking ratio.
Consumption Analysis As shown in Tab. 7, Struc'TexTv2smal consumes 56ms and 2,276MB GPU memory infer one image RVL-CDIP, while LayoutLMv3gas spends more GPU memory or time with different OCR engines. It is observed that the OCR process the two-stage method accounts for the vast majority of computation overhead. Thus, OCR-free framework can achieve a better trade-off between performance and efficiency.
Masking Strategies The impact of adjusting text region-level masking patch-level masking is evaluated in Tab.8. The performance drops a 4.2% accuracy score on RVL-CDIP and a 1.0% mAP on PubLayNet, which demonstrates the effectiveness of the proposed Text-Region masking strategy.
Table 8: Comparison between performance of different masking strategies on RVL-CDIP and Pub- LayNet. The model is only pre-trained with the MIM task.
Model
Masking strategy RVL-CDIP PubLayNet
Struc TeoceexTTVe TeuucexTTl2mlll
patch-level region-level
87.4%
93.1% 94.1 %
StrucTexTv S
91.8%
5 CONCLUSION
This work successfully explores a novel pre-training framework named StrucTexTv2 to learn visual- textual representations for document image understanding with image-only input. By performing text region-basse based image masking, and then predicting both corresponding visual and textual con- tent, the proposed encoder benefit from large-scale document images efficiently. Extensive experiments on five document understanding tasks demonstrate the superiority of StrucTexTy2 over ofte-offttheart methods, especially an improvement in both efficiency and effectiveness.

---

Published as a conference paper at ICLR 2023
REFERENCES
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. Doc- former: End-to-end transformer for document understanding, In ICCV, pp. 973-983, 2021.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: BERT pre-training of image transformers. In ICLR, 2022.
Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: delving into high quality object detection. In CVPR, pp.
2018.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representa- tion learning. CoRR, abs/2202.03022 2022.
Brian L. Davis, Bryan S. Morse, Brian L. Price, Chris Tensmeyer, Curtis Wigington, and Vlad Morariu. End-to-end document recognition and understanding with dessurt. In ECCV, volume 13804 of Lecture Notes in Computer Science, pp. 280-296, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248 248-255, 2009.
Jacob Devlin, Ming-Way Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pp. 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In ICCV, pp. 6568 6577, 2019.
Google. Google cloud vision. https:/ clwdd////////d/ooo/////ooo google www.le..ooovii....oeoooo onn 2019.
Jiuxiang Gu, Jason Kuen, Vlad I. Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. Unified pretraining framework document understanding. NeurIPS, pp. 30-59990 2021.
Adam W. Harley, Alex Ufkes, and Konstantinos G. Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In ICDAR, pp. 991-995, 2015.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, pp- 2980-2988, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. CVPR, pp. 15979-15988, 2022.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutmv3: Pre-training for document Al with unified text and image masking. ACM Multimedia, 2022.
Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. FUNSD: A dataset for form understanding in noisy scanned documents. In ICDAR, pp. 1 1-6, 2019.
Geewook Kim, Teakgyu Hong. Moonbin Yim, Jeong Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document un- derstanding transformer. In ECCV, volume 13688, pp. 498-517, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), ICLR, 2015.
David D. Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, David A. Grossman, and Jefferson Heard. Building test collection for complex document information processing. In Efthimis N. Efthimiadis, Susan T. Dumais, David Hawking, and Kalervo Järvelin (eds.), ACM SIGIR, pp- 665-666, 2006.
10

---

Published as a conference paper at ICLR 2023
Chenliang Li, Bin Bi, Ming Yan, Wei Wang. Songfang Huang, Fei Huang. and Luo Si. Structurallm: Structural pre-training for form understanding. In ACL/IJCNLP, pp. 6309-6318 2021a.
Hui Li, Peng Wang, and Chunhua Shen. Towards end-to-end text spotting with convolutional recur- rent neural networks. In ICCV, pp. 2248-2556, 2017.
Junlong Li, Yiheng Xu, Tengchao Lv. Lei Cui, Cha Zhang, and Furu Wei. Dit: Self-supervised pre-training for document image transformer. ACM Multimedia, pp. 3530-3539, 2022.
Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Varun Manju- natha, and Hongfu Liu. Selfdoc: Self-supervised document representation learning. In CVPR, pp- 5652-5660, 2021b.
Yanghao Li, Yuntao Chen, Naiyan Wang, and Zhao-Xiang Zhang. Scale-aware trident networks for object detection. In ICCV, pp. 6053-6062, 2019.
Yulin Li. Yuxi Qian, Yuechen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, and Errui Ding. Structext: Structured text understanding with multi-modal transformers. In ACM Multimedia, pp. 1912-1920, 2021c.
Minghui Liao, Pengyuan Lyu, Minghang He, Cong Yao, Wenhao Wu, and Xiang Bai. Mask textspot- ter: An end-to-end trainable neural network for spotting text with arbitrary shapes. TPAMI, 43 (2):532-548, 2021.
Minghui Liao, Zhisheng Zou, Zhaoyi Wan, Cong Yao, and Xiang Bai. Real-time scene text detection with differentiable binarization and adaptive scale fusion. TPAMI, 45(1):919 9311 2023.
Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Be- longie. Feature pyramid networks for object detection. In CVPR, pp. 936 936-944, 2017.
Weihong Lin, Qifang Gao, Lei Sun, Zhuoyao Zhong, Kai Hu, Qin Ren, and Qiang Huo. Vibertgrid: A jointly trained multi-modal 2d document representation for key information extraction from documents. In ICDAR, volume 12821, pp. 548 548-563, 2021.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. ACM Comput. Surv.
2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp. 9992- 10002, 2021.
Rujiao Long, Wen Wang, Nan Xue, Feiyu Gao, Zhibo Yang, Yongpan Wang, and Gui-Song Xia. Parsing table structures in the wild. ICCV, pp. 924 932, 2021.
Rafal Powalski, Lukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michal Pietruszka, and Gabriela Palka. Going full-tilt boogie document understanding with ext-image-layout trans- former. In ICDAR, volume 12822, pp. 732-747, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, volume 139, pp. 8821-8831, 2021.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. TPAMI, 39(6):1 2017.
Fenfen Sheng, Zhineng Chen, and Bo Xu. NRTR: A no-recurrence scene text recognition. In ICDAR, pp. 781-786, 2019.
model for
Xinying Song, Alex Salcianu, Yang Song. Dave Dopson, and Denny Zhou. Fast wordpiece to- kenization. In Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), EMNLP,, pp. 2089-2103, 2021.
11

---

Published as a conference paper at ICLR 2023
Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE 2.0: A continual pre-training framework for language understanding. In AAAI, pp. 8968-89 -978, 2020.
Chris Tensmeyer. Vlad I. Morariu, Brian L. Price, Scott Cohen, and Tony R. Martinez. Deep splitting and merging for table structure decomposition. In ICDAR, pp. 114-121, 2019.
Jiapeng Wang, Chongyu Liu, Lianwen Jin, Guozhi Tang, Jiaxin Zhang, Shuaitao Zhang, Qianying Wang, Yaqiang Wu, and Mingxiang Cai. Towards robust visual information extraction in real world: New dataset and novel solution. In AAAI, pp. 2738-2745, 2021.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, pp. 5987-5995, 2017.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. In CVPR, pp. 9643-9653, 2022.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei A. F. Florêncio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. pp. 2579-2591, 2021.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre- training of text and layout for document image understanding. In ACM SIGKDD, pp. 1192-1200, 2020.
Xu Zhong, Jianbin Tang, and Antonio Jimeno- -imeno-Yepes. Publaynet: Largest dataset ever for document layout analysis. In ICDAR, pp. 1015-1022, 2019.
12


Reviews:
[{"review_id": "nAAt-dBLfw", "reviewer": "Reviewer_oo5U", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes two new pre-training tasks for document image understanding problems such as classification, detection, recognition, and information extraction. The first task, Masked Language Modeling (MLM), to predict the word token for an ROI-Aligned feature map. The second task, Masked Image Modeling, is to reconstruct the pixels corresponding to the word bounding box of concern. Both tasks randomly mask word regions based on the bounding boxes and take only the masked image as input. It enables it to be used for many down streams tasks and reduce the dependency to OCR systems that often required by other relevant methods such as LayoutLMs. The experimental results confirm the effectiveness of the approach.", "summary_of_the_review": "It presents a concise but effective method to perform pre-training for document image understanding models. There are some concern and the impact could be limited in the domain, but given the practical importance of the domain and the effectiveness of the approach, I'm happy to support acceptance.", "strength_and_weaknesses": "Strengths:\n\nIt only takes an image is input, but still provide great results. It effectively gets rid of the need of text input while keeping it in the output. It could be used for almost any tasks for document image understanding.\n\nIt less relies on external OCR engines. It could even be used to improve OCR. This is a major difference to other methods.\n\nWeaknesses:\n\nIt uses words as units. Both tasks might not be very robust when needing to deal with very short and long words.\n\nFrom Table 5, it looks like the MaskDoc backbone itself is much superior to other popular backbones. It is not clear if the algorithm is only effective to MaskDoc or the backbone is just very strong.\n\nThe last ablation study was conducted with ViT due to some reason. I was not sure why it was not MaskDoc.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hAda_ZZcugn", "reviewer": "Reviewer_vxyY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces Mask-Doc, a new self-supervised pre-training framework for automatic Document Image Understanding that learns from a single input modality (image). Two pretraining tasks (*Masked Image Modeling* and *Masked Language Modeling*) are learned in an end-to-end manner to extract visual-textual representations.\n\nFirst, an off-the-shelf OCR is used to mask 30% of text zones in a document image. \nThe masked image is passed through a feature extractor composed of CNNs and Transformer. Features are merged and ROIAlign is applied to extract the features of each masked zone. \nFinally, two sub-networks are trained to:\n1) reconstruct the pixel values of the masked zone ;\n2) reconstruct the textual content corresponding to the masked zone.\n\nThis architecture is pre-trained on the IIT-CDIP Test Collection 1.0 dataset (large-scale document dataset) and fine-tuned on 5 downstream tasks related to document image understanding: Document Classification (on RVL-CDIP), Document Layout Analysis (on PubLayNet), Table Structure Recognition (on WTW), Optical Character Recognition (on FUNSD), Information Extration (on FUNSD). MaskDoc obtains strong results on every task and SOTA results on four of them.\n", "summary_of_the_review": "The proposal to carry out a combined pre-training on both images and text is interesting and well evaluated. It would be interesting to demonstrate in which type of application the model actually outperforms its competitors. More technical details should be provided on some aspects.", "strength_and_weaknesses": "##### Strong points\n* The experiments are convincing. The model obtains strong/SOTA results for 5 tasks related to Document Image Understanding. \n* The pretraining framework strategy is an interesting contribution that is relevant to ICLR, and could be useful for other researchers. \n* The ablation study is very interesting, especially the study of the masking ratio, masking strategies and pretraining tasks. \n\n#### Weak points\n\n* My main concern is that I am not convinced that using visual-textual representations (MIM+MLM) is really that helpful. At least for Document Layout Analysis, pretraining on both tasks imporve performance by less than 1% (in Table 5). It would be **very** interesting to complete this ablation study on other tasks (especially on tasks that are more text-dependent than DLA). I feel like this is the heart of the article: is visual-textual pretraining actually better than visual-only or textual-only pretraining ? This question is not fully answered in the paper.\n* It's not clear to me how the text is obtained/tokenized for each text zone. I understand that an OCR is used, and random words with high confidence scores are selected to be masked. However, how is the sub-word tokenization performed ? And why the authors only keep the first sub-word during pretraining (\"we only choose the first sub-word for keeping the number of words in the document images\") ? I feel like this step is crucial for reproducibility, and should be explained more clearly. \n* The authors claim that using an OCR is 1) costly 2) sub-optimal as image/text features are not jointly optimized. These claims are not really supported in the paper. For 1) it would be intersting to compare MaskDoc to LayoutLM-v3 or DocFormer in terms of computing time/resources. For 2), it makes sense, but the classification results (Table 1) show that LayoutLM-v3 and DocFormer outperform MaskDoc. It would be interesting to comment these results in Section 4.3.\n* The ablation study comparing different masking strategies is not sufficiently detailed, especially since the authors insist that masking text zones is better than masking patches. \n* The use of CutMix and MixUp data augmentation on RVL-CDIP is not detailed. Do the other models use this data augmentation strategy as well ? What is the impact on performance ?\n* There is no information regarding the computing requirements, memory consumption and training/inference time for MaskDoc. \n\n#### Minor comments:\n* Some typos/english mistakes should be fixed:\n    * abstract \"comprises ~~of~~ two self-supervised\" (or \"consists of\")\n    * 3.1 a backbone network \"consist**s**\n    * Fig2. \"to reconstruct~~s~~\"\n    * 3.2.1 \"does not need **to** convert\"\n    * 3.3.1 \"a**n** embedding layer\" \n    * 3.3 \"to strength**en**\" \n    * 4.2 \"The ~~rest~~ other parameters\"\n    * 4.4 \"pretrain**ed** models\" \n    * ...\n* Appearing twice in the references:\n    * Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In ICDAR, pp. 991–995, 2015b\n    * Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll ́ar, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022b. \n* 8 articles are cited as arxiv preprints - please update the citation for articles that have been published. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZWiSF4hu3HU", "reviewer": "Reviewer_XqWz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a new multimodal framework for pre-training a document image representation that can be used for several downstream document understanding tasks. The main contribution of the proposed framework lies on using only the image as input, while some OCR detections are used only for supervising the language branch in pre-training. This reduces the dependence on OCR of current methods. Specific pre-training tasks are defined for this setting. Experimental results that the proposed framework can obtain results that are comparable or slightly better than current SoA with a much smaller number of parameters.\n", "summary_of_the_review": "I consider that the paper is an interesting contribution to the domain of pre-training multimodal document representations. The approach is novel and results are good enough. As commented before, there are a few issues that could be better discussed and analysed in the paper, mainly the role of the OCR and the application to more language-oriented tasks, but overall I think that the paper has more positive contributions than weaknesses.", "strength_and_weaknesses": "Strengths\n- A novel multimodal pre-training framework for document representation is proposed where only image is used as input. OCR is only used as supervision for the masked language modelling pre-training task, but it is not necessary for inference for downstream tasks. This reduces the dependance on OCR. \n- A novel pre-training architecture and pre-training tasks are defined to apply masked language modelling and image reconstruction from masked OCR text regions, with the image reconstruction branch integrating language information coming from the masked language modeling branch.\n- The small version of the model has a much smaller number of parameters thant current SoA models while achieving competitive performance (in some cases slightly below the SoA and in other cases slightly above)\n- The paper is clear and well written\n\nWeaknesses\n- Although the paper claims for avoiding dependance on OCR, OCR is still required for the pre-training (even if only a subset of all OCR tokens in the document are used). In this sense, I miss more details on the OCR and how OCR tokens are selected.  1) Which is the OCR system that has been used in the experiments? Can the framework have some type of dependance on the quality of the OCR used to obtain the tokens? Some analysis on the impact of the quality of the OCR in the final results could be insightful. 2) In the last sentence of section 2, it is stated that \"only some high-confidence word-level OCR results need to be randomly selected\", but there are no further details on how the selection of OCR tokens is done during pre-training. Are OCR tokens completely randomly selected? Or selection is based somehow on the confidence of the OCR detections? If this was the case, how?\n- The large version of the model does not seem to bring about signifficant performance improvement in relation to the large increase in the number of parameters of the model.\n- The model has shown pretty good performance on tasks which largely rely on visual information (document classification, layout analysis, table structure recognition) or on semantically simple linguistic information (information extraction, table recognition). I have some doubts on the ability of the model to adapt and perform well on tasks which require a deeper semantic understanding of language, such as DocVQA. Any insight on this?\n- I missed some details on the computational resources and time required to train the model. This is an important aspect in these type of models trained on such a large set of data. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "M64eJssuY_8", "reviewer": "Reviewer_HUEm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new visual-text learning algorithm for document understanding. In particular, the authors argue that the advantage of proposed algorithm is free of using OCR in the prediction. ", "summary_of_the_review": "The novelty is needed to be justified as the motivation is not clear and the discussion of missing related works.  Also, it would be nice to showcase on more challenging NER tasks. ", "strength_and_weaknesses": "* The major question I have is the motivation of the work.  It is not clear to me why we would like to discard OCR as it is quite matured and well-developed, also it can leverage external data to train the OCR model.  The argument of the cost of running OCR does not convince me either.  The QPS of the modern OCR system is quite high (e.g. on major cloud platforms). Therefore, it seems not well-motivated to me.  Regardless of using OCR in the final algorithm,  the possibility can be on improving the image component by absorbing what OCR is doing, then showing how it can improve the final performance together with text components with OCR. If the authors provide some experiments on this, it would be more promising. \n* Missing important related work, “OCR-free Document Understanding Transformer”, which is published in ECCV 2022. How does the proposed work compare with this earlier work?  A separate note: although this work also shares the same motivation of being OCR free, it doesn’t really justify why we want an algorithm that discards OCR as it is quite standard in the representative works and it is actually cheap in practice. \n* It is surprising that the small model can achieve almost on-par performance with large models in Table 1. However, it looks a bit too good to be true for complex language understanding tasks. So does it mean the classification is actually a relatively simple task? If so, I would recommend the authors to try on more challenging NER tasks, as LayoutLMs and FormNet (Lee et al., 2022) have studied to demonstrate how much the proposed algorithm really learn the task semantics. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster