UniDoc: Unified Pretraining Framework for Document Understanding
Authors: Jiuxiang Gu
Jason Kuen
Vlad I Morariu
Handong Zhao
Rajiv Jain
Nikolaos Barmpalios
Ani Nenkova
Tong Sun
Venue: NeurIPS 2021 Poster (2021)
Keywords: BERT
Self-Supervised Learning
Document Understanding

Abstract: Document intelligence automates the extraction of information from documents and supports many business applications. Recent self-supervised learning methods on large-scale unlabeled document datasets have opened up promising directions towards reducing annotation efforts by training models with self-supervised objectives. However, most of the existing document pretraining methods are still language-dominated. We present UDoc, a new unified pretraining framework for document understanding. UDoc is designed to support most document understanding tasks, extending the Transformer to take multimodal embeddings as input. Each input element is composed of words and visual features from a semantic region of the input document image. An important feature of UDoc is that it learns a generic representation by making use of three self-supervised losses, encouraging the representation to model sentences, learn similarities, and align modalities. Extensive empirical analysis demonstrates that the pretraining procedure learns better joint representations and leads to improvements in downstream tasks.

Unified Pretraining Framework for Document Understanding
Jiuxiang Gu', Jason Kuen', Vlad I. Morariu', Handong Zhao', Nikolaos Barmpalios', Rajiv Jain', Ani Nenkova', Tong Sun
Adobe Research, 2 Adobe Document Cloud {jigu, monari,, hazhao, barmpali, rajinai, nenkova, tsun Gadobe.ecc
Abstract
Document intelligence automates the extraction of information from documents and supports many business applications. Recent self- self-supervised learning methods on large-scale unlabeled document datasets have opened up promising directions towards reducing annotation efforts by training models with self suferiuperiiee ob- jectives. However, most of the existing document pretraining methods are still language-dominated. We present UDoc, new unified pretraining framework for document understanding. UDoe designed to support most document understand- ing tasks, extending the Transformer to take multimodal embeddings as input. Each input element composed words and visual features from semantic region the input document image. An important feature of UDoc is that it learns a generic representation by making use of three self-supervised losses, encouraging the rep- resentation to model sentences, learn similarities, and align modalities. Extensive empirical analysis demonstrates that the pretraining procedure learns better joint representations and leads to improvements in downstream tasks.
Introduction
Document intelligence is a broad research area that includes techniques for information extraction and understanding. Unlike plain-text documents in natural language processing (NLP) [1, 2]. a physical document can be composed of multiple elements: tables, figures, charts, etc. In addition, document usually includes rich visual information, and can be one of various types of documents (scientific paper, form, resume, etc.), with various combinations of multiple elements and layouts. Complex content and layout, noisy data, font and style variations make automatic document understanding very challenging. For example, to understand text-rich documents such as letters, a system needs to focus almost exclusively on text content, paying attention to a long sequential context, while processing semi seni-structured documents such forms requires the system analyze spatially distributed short words, paying particular attention to the spatial arrangement of the words. Following the success of BERT [3] NLP tasks, there has been growing interest in developing pretraining methods for document understanding [4, 5, 6]. Pretrained models have achieved state-of-the--a (SoTA) performance across diverse document understanding tasks [7, 8].
Huge training datasets help pretraining models to learn a good representation for downstream tasks. However, we observe three major problems with the current pretraining setup: (1) documents are composed of semantic regions. Most of the recent document pretraining works follow BERT and split documents into words. However, unlike the sequence-to- -oseuuunce learning in NLP, documents have hicrarchical structure (words form sentences, sentences form semantic region, and semantic regions form document), Also, the importance of words and sentences highly context-dependent, i.e., the same word or sentence may have different importance a different context. Moreover, current transformer-based document pretraining models suffer from input length constraints. Also, input
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

length becomes a problem for text-rich documents or multi page documents. (2) documents are more than words. The semantic structure of the document is not only determined by the text within it but also the visual features such as table, font size and style, and figure, etc. Moreover, the visual appearance of the text within a block are often overlooked. Most of recent BERT-based pre- training works only take the words as input without considering multimodal content and alignment of multimodal information within semantic regions. (3) documents have spatial layout. Visual and layout information is critical for document understanding. Recent works encode spatial information via 2D position encoding and model spatial relationships with self-attention, which computes attention weights for long inputs [4, 5]. However, for semi-structuee documents, such as forms and receipts, words are more related to their local surroundings. This corresponds strongly with human intuition newspapers, the receptive fields are modulated by our reading order
when we look at magazines and attention. Based on the above observations, we ask the following question: Can unifted document pretraining benefit all of these different kinds of documents?
We propose a unified pretraining framework for document understanding, shown in Fig. Our model integrates image information in the pretraining stage by taking advantage of the transformer architecture to learn cross-modal interactions between visual and textual information. To handle textual information, we encode sentences using a hierarchical transformer encoder. The first level of the hierarchical encoder models the formation of the sentences from words. The second level models the formation of the document from sentences. With the help of the hierarchical structure, UDoc learns how words form sentences and how sentences form documents. Mcanwhile, it reduces model computation complexity exponentially and increases the number of input words. This also mimics human reading behaviors since the pentence/paragraph is a reasonable unit for people to read and understand peopaa rarely check the interactions between arbitrary words across different regions in order to understand an article. Convolution has been very successful in the extraction of local features that encode visual and spatial information [9], so we use convolution layers as more efficient complement to self-attention for addressing local intra-region dependencies in a document image. Meanwhile, self-attention uses all input tokens to generate attention weights for capturing global dependencies. Thus, we combine convolution with self- attention to form a mixed attention mechanism that combines the advantages of the two operations.
We depart from previous vision-language pretraining [10, 11] by extracting both the textual and visual features for each semantic region. We propose novel gated cross-attentional transformer that enables information exchange between modalities. A visually-rich region (figure, chart, etc) may have stronger visual information than textual information. Instead of treating outputs from both modalities identically, we design gating mechanism that can dynamically control the influence of textual and visual features. This approach enables cross-modal connections and allows for variable highlight the relevant information in visual and textual modality and enables cross-modal connections. During pretraining, the easebased visual backbone and multi-layer gated cross-attentin encoder are jointly trained in both pretraining and fine-tuning phases.
summarized as follows: (1) We introduce UDoc, powerful pretraining framework for document understanding. UDoc is capable of learning contextual textual and visual in- formation and cross-modal correlations within single framework, which leads to better performance. (2) We present Masked Sentence Modeling for language modeling, Visual Contrastive Learning for vision modeling, and Vision-Language Alignment for pretraining. (3) We present extensive experiments and analyses to validate the effectiveness of the proposed UDoc. Extensive experiments and analysis provide useful insights the effectiveness of the pretraining tasks and show outstanding
Our contributions
performance on various downstream tasks.
2 Related Work
Self-supervised learning has shown great success in producing generic representations that learn from large-scale unlabeled corpora [3]. Like the development of pretraining in computer vision [12] and NLP [3], there has been a surging interest in self eeffsuueee learning for Vision-Language (VL)
tasks [10, 13, 14, 11]. Transformers [3] are the key technology that enables learning contextualized representations from large-scale unlabeled training data. The unique characteristics of document images (spatial layout and multiple elements) distinguish document image pretraining from pretrain- ing works in NI.P and VI. domains. the NI.P domain, the inputs are pure texts without spatial layouts (bounding boxes). In the VL domain, the inputs are the visual objects and captions. While for

---

(CLS
Smiite
Iog
Sautbe
MSV
Retl Pine
Lecadions
Promaling Judh Video Alguoest
Words
Ral Post
fuatAlige
Smaliee
Cross
Premaiinn Vinael Communtyv Learning
RaI
MASK
Feature
Attention
SEF
Unlabeled Document Images
Feature Embedding
Pretraining Tasks
Figure I: Overview of the proposed approach, UDoc. UDoc first uses a CNN-based visual backbone to learn visual representations, The model then extracts the Region of Interest (Rol) features with OCR bounding boxes and generates a multimodal embedding by combining the textual embedding and position encoding. The transformer- based encoder takes of masked multimodal embeddings as imput and is pretrained with three pretraining tasks. All the network parameters except those the textual encoder are jointly trained during both pretraining and fine-tuning phases.
document images, the input elements are spatially distributed, and the visual and textual information co-occur within the semantic regions.
Several recent works have explored pretraining document images [4, 5, 15]. LayoutLM [4] extends BERT to learn contextualized word representations for document images through multi-task learning. takes a sequence of OCR words as input during pretraining and incorporates the 2D position embedding as input for each token. However, LayoutLM only considers textual information during pretraining without modeling the alignment between visual and textual information-visute information is only incorporated into the model during the fine-tuning stage. The most recent version, LayoutLMM2 [5], improves on this by incorporating the image encoder into pretraining and jointly training the image encoder along with the BERT model. LayoutLMv2 splits the document image into several parts and concatenates the visual embeddings and text embeddings into a single sequence. Apart from masked language learning (MLM), LayoutLMVV also considers image-text alignment and image-text matching during pretraining. The most related work to ours is SelfDoe [6], which proposes multimodal document pretraining framework. It first extracts the document object proposals from pre- pre-trained Faster R-CNN [16] and then applies OCR for each proposal to get the words. takes the pre-extracted Rol features and sentence embeddings input, and models the perform learning over the textual and visual information using the cross-modality encoder.
There is a noticeable difference between our proposed method, UDoc, and other concurrent works in document image pretraining. UDoc multimodal end-to-end pretraining framework for document images. Unlike the fixed document object detector in [6]. the parameters of the image encoder with Rol align, which derive the visual features for semantic regions, are also updated in UDoc. In contrast to [5], our visual features come from the semantic regions instead of splitting the image into fixed regions. Like the object-level semantic elements in natural images, for document images, we represent the typical document layout elements such as paragraph, title, figure, and table as semantic regions. Morcover, to learn the contextualized visual representations, UDoc masks visual information in the latent space and learns contextualized representations by solving a contrastive learning task defined over a quantization the latent visual embeddings.
3 Method
3.1 Model Architecture
Fig. 1 illustrates our approach, UDoc, which consists of four components: feature extraction, feature embedding, multi-layer gated cross-attention encoder, and pretraining tasks. Given document image and the locations of document elements (sentence or Rol), UDoc takes image regions and words that correspond each document elements as inputs, and extracts their respective embeddings through visual feature extractor and a sentence encoder. These embeddings are then fed into transformer-ban encoder to learn the cross-modal contextualized embeddings that integrate both visual features and textual features.
3

---

In the feature extraction step, we first employ an off-the-shelf OCR tool [17] to extract text from a doc- ument image I, where the words are grouped into sentences S = [ .s....... .......5.. SN) } whose corresponding For each sentence bounding box pt. we use ConvNettbased
bounding boxes are P {pi,- ........ :....,.'::: .NN) backbone fimEin and Rol Align [18] [RolAlin to extract the pooled Rol features v.. To obtain a feature embedding, we extract the sentence embedding
for each sentence 8; via pretrained sentence encoder fSentEnc. Each region's 's Rol feature v, is discretized into a finite set visual representations o E V2 via product quantization [19]. The multi-layer Gated Cross-Artention encoder takes the position information, masked visual features V and masked textual features $ as inputs, and then it generates the contextualized multimodal representations (Hl and H's. I E [1, L)) and outputs the predicted features (V and Š), where L is the number of stacked transformer blocks.
More formally, the pretraining procedure can be decomposed into the following steps:
OCR
P S
/ininct Snatatie
V, (VS
H H
V S
Preetrining
(I)
fsentio:
where Mask denotes the masking function that randomly masks Rol features and sentence embeddings with the respective probabilities Pirnsk and Pitma Preetrining is composed of three pretraining tasks: Masked Sentence Modeling (MSM), Visual Contrastive Learning (VCL), and Vision-Language Alignment (VLA). Next, we provide details mentioned in Eq. 1.
Feature Extraction and Embedding. Formally, a document image I E RW*H consists of N regions, where each region's bounding box is characterized by a 6-d vector, as p; (aur MT 288 yen W }. where w and are of the width and height the region, W and are the W. H W : width and height of I. while (æLr: Mar) and (ØRB, YRB) denote the coordinates of the top-left and bottom-right corners respectively. The 6-d vector is mapped into a high- nigh-dimensional representation via a linear mapping function.
The visual embedding is the sum of the mapped Rol feature and position embedding. Likewise, textual embedding is the sum of sentence embedding and position embedding. We also have different types of segments to distinguish different modalities. The input sequence to the transformer- based encoder starts with a special start element ([CLS) and full visual features), then it is followed by multimodal elements, and ends with a special ending element ([SEP]+full visual features). For the special elements ([CLS] and [SEP]), the corresponding full visual features are features extracted from the whole input image, by applying fimtine to an Rol covering the whole input image.
Quantization Module. Unlike the fixed image encoder in [6]. we jointly learn the image encoder in an end-to-end fashion alongside the multimodal model. A visual representation can be learned by predicting the visual features of the masked regions, but it is challenging to predict such features exactly, since they are unconstrained and of continuous representation. To constrain the representation space of the visual features and facilitate the end-to-end learning of image encoder (see Task #2 in Sec. 3.2), we follow [20, 21] and use vector quantization to discretize the visual features V {uI...,, uN into finite set of representations Vi = {uf.... ..... U Specifically, we define latent embedding spaces e E IR CxE where C is the number of codebooks, and E is the number of entries for each codebook. For each U, we first map it to logits of E RCE'' and calculate the probability for the j-th codebook entry in i-th group as Pe.e = exp((vÉ, + ge)/T)/ Σ exp((x( &. + gx)/T), where T is a non-negative temperature, g are i.i.d samples drawn from Gumbel(0,, 1) distribution. During the forward pass, we choose one entry vector from each codebook by é, argmax
and generate the quantized representation of by a concatenation of (ē),.... ( ēa} which then followed by a linear transformation. During the backward pass, the gradients are computed through Gumbel-Softmat estimator [22].
Gated Cross-Attention.
To model the interactions among multimodal inputs, introduce multimodal transformer with gated cross-attention to model the cross-modality relationships. Let H be output features the l-th layer for one modality m. and let be another modality (m, S)). We obtain the features at (1 + 1)-th layer as:
H +++ SIN (JiN + Fé Au (H', (H,)) + fir + &ernss Au (H (H()))
(2)

---

where fin denotes layer normalization [23]. The feed-forward sub-layer fFr in Eq. 2 is further composed of two fully-connected sub-layers, both wrapped in residual adds and fin-
The core part of Eq. 2 is the cross-attention Au()s-Au(-). Given the intermediate representations H m and H the cross-attention output for modality m is computed as:
fCross Aul H H [Cross-Atue (H :
Cross-Aut (H H ((HH)
Cross-Att (H H softmax
where (H,)), (H.),, and ft (H.) are the query, key, and value calculated by linear mapping layers for the i-th head. the model dimension, the number of heads, and U the weight
matrix that combines the outputs of the heads.
Considering the substantial diversity of document images and the different information needs of differing document types, we use gating mechanism [24] to dynamically weight the outputs of the visual and textual branches. Specifically, we feed the concatenated the visual and textual features to non-linear network frate m H "1), which generates the modality-seecific attention weights o', and an, and returns the weights separately to their respective modality-specific branches to perform element-wis products. We multiply the features for modality with its modality-specific attention weight, and compute the updated feature as: H4+1 H + am)) same that for modality n.
3.2 Training Tasks and Objectives
The full pretraining objective UDoc (right block in Fig. 1) is defined as: Leretraniigg LMMM LVCL + LVLA. In the rest of this section, describe each task in detail.
Task #1: Masked Sentence Modeling, This task is similar to the MLM task utilized in BERT. The key difference is that we mask sentences instead of tokens. During pretraining, each sentence and Rol of the input document is randomly and independently masked. For the masked sentence, its token replaced with a special sentence [MASK]. The model is trained to predict masked sentence feature, based on the unmasked words and the visual features. The goal is to predict the masked sentence embeddings based on the contextual information from the surrounding sentences and image regions, by minimizing the smooth L1 loss [16]:
(MSM(O) Σ smoothr, (s, fuDoo (8: V))
where is the trainable parameters and JuDee()) outputs the unmasked textual feature, 8, is the surrounding features for the i-th input, V are the image features with random masking.
Task #2: Visual Contrastive Learning. We learn visual feature representations by solving a visual contrastive learning task which requires estimating the true quantized latent Rol representation. Given a prediction û; E V for the masked Rol ē, V. the model needs to estimate the positive quantized representation oi in set of quantized candidate representations VQ. Good representations are learned by maximizing the agreement between output representation and quantized representation
of the same Rols as follows:
Σ (lgg exp(sim 0. 0, 0/9)KK) V
Lvcr. (0)
CF ΣΣ
Pa.e log Pe.e
(6)
E.g exp(sim( 0. )/)/
where sim(..) :) computes the cosine similarity between two vectors, À is a hyperparameter, and x is a temperature scalar. The second term encourages the model to use the codebook entries more equally.
Task #3: Vision-Language Alignment. To enforce the alignment among different modalities, we explicitly encourage alignment between words and image regions via similarity preseeriig knowledge distillation [25]. Note that, unlike the text-image alignment in AyyouI_Mv2 [5] which splits the image into four regions and predicts whether the given word covered or not on the image side, align the image and text belonging the same region. The goal is to minimize the differences
5

---

between the pairwise similarities of sentence embeddings and the pairwise similarities of image region features:
(O) x N |./ooo (S ST) NNom (H₂ H M|p
where S is the unmasked input sentence embeddings, Hό the mapped visual representations of the final layer, || ||e is the Frobenius norm, and fNomm performs L2 normalization.
4 Experiment
4.1 Pretraining UDoc
Pretraining corpus. We build our pretraining corpus based on IIT-CDIP Test Collection 1.0 [26], which contains more than 1 11M scanned document images. To differentiate pretraining from finetuning, we filter out the document images of RVL--DIIP [8] from IIT-CDIP since it is a subset of IIT-CDIP, and sample 1M document images as our pretraining corpus.
Table shows the dataset statistics. IIT-CDIP only provides the OCR texts in XML format. We extract words and their locations by applying EasyOCR [17] on document images. As shown in Fig. 3 (a), EasyOCR provides two kinds of paragraph and paragraph. The paragraph mode groups the non-paragraph results into text regions. We think document image pretraining should be treated differently than sequence-based pretraining in NLP, since the words in the document (2D) are arranged according to spatial layouts, while the words in NLP corpora are sequential (ID). Considering the special characteristics of documents (complex layout, multi-pages) and the limited input length of BERT models, it is not intuitive to formulate the input at the word level. Hence, we adopt the paragraph-leve outputs as the basic input elements since textual regions provide semantically more meaningful information than independent words.
Table I: Comparison of the datasets used for pre- training and finetuning process. 'Box', 'Label', and "Text' indicate the availability of location, label and text annotations for document entities. "Tag' denotes the document class label availability.
output modes:
Dataset
Type
Size
Box
Label Text
Tag
TTT-CDIP [26]
Misc
11M
X
d
X
RVL-CDIP [8]
Misc
400K
X
x
x
CORD [7]
Receipt
1K
/
d
X
FUNSD [27]
Form
0.2K
X
PubLayNet [28]
Article
347K
x
x
There are some advantages to our design: (1) the region-level design hierarchically encodes document elements and this facilitates the mod- cling of latent relationships at the region level which has higher-level semantics than the word level. (2) the hierarchical encoding also over- comes the input size limitation of word-level BERT-based models [4, 5]. Fig. 2 shows the dis- tribution of words per region on RVL-CDIP. It can be seen that even though we consider region- level input, for some semi- cemi-structured documents, single-word words dominate the inputs; this somehow forces UDoc to pay attention to word-level in- puts. Unlike MLM that predicts the masked word, UDoc predicts the textual embedding of the masked input with MSM.
Figure 2: Distribution of words per region on RVL.- CDIP according to the categories.
Pretraining setting. We initialize the sentence encoder fsemEun with BERT-NLI-STSb-bar [29] pretrained for NLI [30] and STS-B [31]. The ResNet-50 backbone in the image encoder is pretrained on the PubLayNet training set [28]. All the parameters (except fSemtine and fimiin)) are randomly initialized. During pretraining, we freeze the parameters of fsemtine and jointly train the visual encoder and multi- modal UDoc model in an end-to-end fashion. Such an end-to-end training allows the ConvNet and Transformer to realize their full potentials in spatial and sequence modeling for pretraining. UDoc contains 12 layers of gated cross-attention transformer blocks. We set the hidden size to 768 and the number of heads to 12, the maximum number of regions N to 64, and the maximum input sequence length for fSentine to 512. The pretraining is conducted on 8 NVIDIA Tesla

---

Amstle Ancre'
(4) III-CIIP
(b) FUNSD
CORD
(d) PubLayNet
Figure 3: Document image samples. The boxes in red/green are OCR bounding boxes obtained with/without paragraph mode, while the boxes in blue are officially-provide bounding boxes.
V100 32GB GPUs with a batch size of 64. It is trained with Adam optimizer [32], with an initial learning rate of 10-5, weight decay of 10-1, and learning rate warmup in the first 20% iterations.
To learn a useful multimodal representation, random masking is applied to both textual and visual inputs. For MSM, we set the mask probability Puss for input sentences to 15%. 80% among the masked sentences are replaced by special sentence [CLS, MASK, SEPJ, while 10% sentences are replaced by random sentences sampled from other documents, and 10% remains unchanged. For VCL, the À is set to 0.1, K is set to 0.1, the mask probability PMask is set to 7.5% and the masked Rol features are filled with zeros. The temperature is annealed from 2.0 to 0.5 by factor of of 0.999995 at every iteration. We select the pretraining checkpoint with the lowest Prramining for finetuning stage.
4.2 Finetuning Tasks
Form Understanding. Form understanding requires the model to predict the label for each semantic entity. We use FUNSD [27] as the evaluation dataset. It contains 149/50 training/testing images. Fig. 3 (b) shows a sample from FUNSD. Each semantic entity comprises a list of words, label, and bounding box. The officially-rrovided OCR texts and bounding boxes used during training and testing. We take the semantic entities as input and feed the concatenated visual and textual output representations classifier. We apply cross-entropy loss for finetuning. The model is finetuned for 100 epochs with a learning rate of 10-5 and batch size of 16. All the parameters except fsemtnn trained. One of question, answer, header other is predicted for each semantic entity. We use entity-level Fl score as the evaluation metric.
Receipt Understanding. Receipt understanding requires the model to recognize a list of text lines with bounding boxes. The performance this task is evaluated on CORD [7] dataset. It contains 626/247 receipts for training/testing. The receipts are labeled with 30 types of entities under categories: company, date, address, and total. Like FUNSD, we feed the concatenated visual and textual output representations to the classifier. The model is finetuned for 200 epochs with a batch size of 16 and a learning rate of 10 The evaluation metric is entity-level F1 score.
Document Classification. Document classification involves predicting the category for each doc- ument image. We use RVL-CDIP [8] as the target dataset. It consists of 320K/40K/40K train- ing/validatiion/tssinng images under 16 categories. The OCR words and bounding boxes are extracted by EacyOCR. To fine-tune UDoc on RVL-CDIP, we compute the overall representation as an element- wise product between the visual and textual representations averaged from all sentences/regions, and learn a classifier on top of the overall representation with cross-entropy loss. We fine-tune the model for 30 epochs with batch size of 64 and a learning rate of 10 Classification accuracy over 16 categories is used to measure model performance.
Document Object Detection. Document object detection involves decomposing a document im- age into semantic units. We evaluate the effectiveness of our pretrained visual backbone on Pub- LayNet [28]. As shown Fig. 3 (d), the documents in PubLayNet are scientific articles. PubLayNet consists of 336K/1 IK training/validation images with six category labels (text, title, list. figure, and
7

---

table). We train Faster-RCNN (F-RCNN) using Detectron2 [33] and initialize the visual backbone with the pretrained ResNet-50 from UDoc. The model trained for 180k iterations with a base learning rate of 0.01 and a batch size of 8. Mean average precision (MAP) @ intersection over union (IOU) [0.50:0.95] of bounding boxes is used to measure the performance.
4.3 Results and Discussion
The importance of multimodal learning. To study the effect of multimodal learning, we experi- ment in three different settings (1) Vision only (V): this setting omits the textual components of UDoc and adopts multilayer self-attention transformer to learn the visual representation. (2) Language only (L); this setting omits the visual encoder and keeps only the textual components. (3) Vision-Language (V+L): this setting considers both vision and language information. We first train three settings with- out pretraining. Table 2 shows consistent improvement across tasks for V+L over the single-stream baselines (V or L). This demonstrates that our UDoc model is able to learn important visual-linguise relationships that benefit downstream tasks even without pretraining.
In Table 2. we find that visual information dominates the performance of document classification, while language information contributes lot to form understanding and receipt understanding. The results also indicate that different document tasks rely on different information. For document entity recognition tasks, language information is more important than visual features. As can be seen in Fig. 3 (b) and (c), entity recognition is more word-oriented. On the other hand, document classification more focused on global-level understanding. As a result, visual and layout information contribute a lot to the final prediction of the document classification model. This matches well with the innate abilities of humans distinguish between document types without fully understanding the words. We also observe that gated cross-attention (V+L) achieves better performance than the non-gated version (V+L*), as its gating mechanism can learn to adaptively determine how much each modality contributes to the output features.
Effect of pretraining tasks. We analyze the effectiveness of different pretraining settings through ablation studies over FUNSD, CORD, and RVL-CDIP, which are representative document bench- marks. Table 2 ablates the key design choices in pretraining UDoc. For experimental efficiency, UDoc models evaluated here are trained with 5 epochs on 300k training corpus. Overall, the pretraining of UDoc consistently improves the performance over all three downstream tasks. The improvement gains vary among different tasks.
Table 2: Experimental results and comparison on FUNSD, CORD, and RVL-CDIP test sets.
Pretraining
FUNSD
CORD RVL-CDIP
Enable #Data Modality Max #Words #Param.
Tasks
Epoch
F1
F1
Accuracy
85M
77.49
57.08
91.35
153M
78.46
71.52
86.82
X
V+L
255M
80.60
95.98
92.76
V+L
267M
83.34
96.59
92.93
300K
V+L
64 x 512
270M
MSM + MVM
5
84.37
97.44
93.10
300K
V+L
61 x 512
272M
MSM + VCL
S
86.87
98.70
93.59
300K
V+L.
64 512
272M
MSM VCL + VLA
87.38
98.75
93.92
300K
V+L
64 512
274M + VCL + VLA + REL
87.20
98.13
93.64
We first establish two baselines: MSM+MVM in Table 2 indicates the combination of masked sentence learning and masked visual feature prediction. Similar to MSM, for MVM, we freeze the visual back- bone and perform masked visual feature prediction via Rol-feature regression. MSM+ VCL jointly trains the visual backbone end-to-end with contrastive learning. As shown in Table 2, MSM+MVM achieves better results than the model without pretraining. Furthermore, when combining VCL to- gether with MSM, consistent performance gains are observed across all the benchmarks. Among the three finetuning tasks, the improvements on FUNSD and CORD are bigger than on RVL-CDIP. We think the local context modeling capability of the ConvNet-basee image encoder brings more benefits to entity recognition, since entities are heavily linked and correlated to their local surroundings. When MSM, VCL, and VLA are jointly trained, we observe further performance gains across all the benchmarks. For VCL, instead of sampling the negatives from the same input document, we also try including the negative samples from other document images of the same batch. However, we find that sampling negatives from the entire batch of document images hurt the performance. This likely because the negatives from other document images easy to distinguish from each other. We
8

---

10C
(V) liBlrr (Ve.Thaapedd
CONTINGATION
TUT
del
(a) Samples from RVL-CDIP
(b) Performance Comparsion 16 Classes
Figure 4: For (a) we show the samples from RVL-CDIP. The boxes in orange color are grouped OCR bounding boxes. For (b) we plot the accuracies on 16 classes achieved by different models that are represented by different colors in the bar chart.
also consider the image-text matching task (Rel) [5], and combine Rel with MSM+VCL+VLA, It hurts the performance of all three downstream tasks. We conjecture that the image-text matching task introduces mismatched pairs of image and OCR texts as negative examples that potentially hamper the training of other tasks.
What if Masked Language Modeling is included? To study the feasibility of that, we consider MLM during pretraining. Since the number of words may be very large, we select the tokens by randomly applying a sliding window (window size 128) across all sequenced OCR words. Each word is formulated as a single- word sentence ([CLS] [Token] [SEP]). We randomly mask 15% of those sampled words ([CLS) [MASK] [SEP]) and concatenate them along with the region- inputs. During pretraining, we add the word prediction head on top of UDoc and predict the masked words. We conduct finetuning experiments on entity recognition tasks (FUNSD and CORD), and find that such direct combination hurts the performance: FUNSD: 87.38 (UDoc) vs. 83.76L (UDoc+MLLM) CORD: 98.75 (UDoc) vs. 98.63 (UDoc+MLM). There are consistent performance drops from adding MLM. One possible reason is that the Rol features extracted by token bounding boxes might not be discriminative enough due to the tiny word-level bounding boxes.
Table 3: Comparison with state-of-the-ar ahe methods. The symbol implies using Google OCR engine.
Pretraining
FUNSD
CORD RVL-CDIP
Method
Source
#Data Scale
Max #Words Modality #Param.
F-
FI
Accuracy
BERT [S]
Word
512
L
110M
60.26
89.68
89.81
BERT LARGE [5]
Word
512
L
340M
65.63
90.25
89.92
LayoutLMM RASE [5]
ITT-CDIP
11M
Word
512
113M
78.66
94.72
94.42
LayoutLM LARGE [5]
IT-CDIP
11M
Word
512
L.
343M
78.95
94.93
94.43
LayoutLMM2 [5]
IT--CDIP
11M
Word
512
V+L
200M
82.76
94.95
95.25
LayoutLMM2,AARGG [5]
IIT-CDIP
11M
Word
512
V+L
426M
84.20
96.01
95.64
SelfDoc [6]
RVL-CDIP
320K
Region
50x512
V+L
83.36
92.81
SeffDec+VGG-16 [6]
RVL-CDIP
320K
Region
50x512
V+L.
93.81
TILT-Basc [34]
RVL-CDIP+
1.1M
Word
512
V+L
230M
95.11
95.25
TILT-Large [34]
RVL-CDIP+
1.1M
Word
512
V+L
780M
96.33
95.52
UDoc
ITT-CDIP
1M
Region
64x512
V+L
272M
87.96
98.85
93.96
UDoc*
ITT-CDIP
IM
Region
64x512
V+L
272M
87.93
98.94
95,05=
Performance Comparison with SOTA. We further pretrain UDoc on 1M document images with 5 epochs and report the finetuning results in Table UDoc outperforms previous models on FUNSD and CORD, by a significantly large margin, demonstrating that our proposed approach is highly effective, partially due to the end-to-end training of the image encoder that improves the semantic alignments between images and texts. Note that UDoc is pretrained on a subset of IIT-CDIP (IM document images), which is considerably less than the 11M document images used in LayoutLM [4] and LayoutLMv2 [5]. TILT [34] builds 1.1M pretraining corpus by combining RVL-CDIP, UCSF Industry Documents Library, and Common Crawl. UDoe also achieves promising results on document classification. Note that both LayoutL. v2 and TILT use Microsoft OCR, which is commercial service with a stronger OCR performance than EasyOCR, which is used in our experiments. We find that OCR plays a key role in document classification performance. As shown in Fig. 4, UDoc

---

performs the best on the "email' category but worst on the 'form' category. We also report the results with different OCR engines: 93.42 (Tesseract [35]) vs. 93.96 (EasyOCR [17]) vs. 94.10 (Google OCR [36]). UDoc with EasyOCR achieves a better performance than with Tesseract since EasyOCR is powered by an advanced neural network, while Tesseract is based on less sophisticated techniques. Since different tasks require task-specific input embeddings to perform well, instead of finctuning the sentence encoder during pretraining, we explore unfreezing the sentence encoder during the finetuning stage (named UDoc*) and report the results in Table 3. Unsurprisingly, we see performance improvements on several downstream applications. Egg. RVL-CDIP: 93, 33.96 (UDoc) vs. 95.05t (UDoc"). However, this also makes the training more challenging in terms of computational resources and training time.
Effect of visual backbone. Addition- ally, we apply the trained visual back- bone to document object detection on PubLayNet. The performance of
Table 4: MAP @ IOU [0.50:0.95] of the document detection models on PublaayNet dev set.
Method
Text Title List Table Figure
mAP
F-RONN (ResNet-101) [28]
91.0 82.6 88.3 95.4
93.7
90.0
the F-RCNN on the validation set is
M-RCNN (ResNet- [28]
91.6 84.0 88.6 96.0
94.9
90.7
depicted in Table 4. To better com- pare, we establish two F-RCNN mod-
F-RCNN (ResNet-50)
92.2 84,4 89.5 96.3
94.5
91.4
F-RCNN (UDoe, Ncs 93.9 88.5 93.7 97.3
96.4
93.9
els with: (1) backbone initialized with
ResNet-50 pretrained on ImageNet; (2) backbone initialized from UDoc's pretrained visual backbone. can be seen that our pretrained backbone outperforms ImageNet-pretraingg backbones. By leverag- ing UDoc, we can train different variants of the visual backbone and apply them to document-specific downstream applications, without relying on incompatible pretrained backbones from other domains (e.g., natural image). Moreover, the visual backbone of UDoc does not require any custom layers, and thus any ConvNet architecture can be used in place of ResNet.
5 Conclusion, Limitations, and Future Works
We develop UDoc, a unified pretraining framework for document understanding. Our model intro- duces novel joint training framework that effectively exploits the visual and textual information during pretraining and finetuning. We evaluate the UDoc comprchensively on three downstream tasks: form understanding, receipt understanding, and document image classification. Extensive empirical analysis demonstrates that the pretraining procedure can take advantage of multimodal inputs. Also, it can effectively aggregate and align visual and textual information of document images with the proxy tasks. This work has a broader impact on document applications. By finetuning the pretrained UDoc on task-specific data, document processing systems can provide better results and reduce the expensive data annotations costs. In terms of negative social impact, the document images used for pretraining may contain sensitive information and therefore the models trained on such data may inappropriately leak some private information. To address the privacy Icakage, it is worthwhile to explore the combination of privacy-preserving learning and self-supervised learning.
There are interesting short- and long-term research directions for UDoc: (1) we freeze the sentence encoder during pretraining and fine-tuning phases due to computational constraints. A better doc- ument representation can be learned by jointly training the sentence encoder, visual backbone and cross-attention encoder in a completely end-to-end fashion. (2) Although impressive performance has been achieved in document entity recognition tasks such as form and receipt understanding, the classification accuracy on semi-structured documents such forms is still inferior that of rich-text documents. It is possible to devise a better method to model the spatial relationship among words. (3) An interesting direction is to extend UDoc multipagc/mulilinguall document pretraining, Additionally, there exist many text-based labeled document datasets in the NLP domain, such as document summarization. Can we transfer the knowledge learned from the text-based document domain to the image- based document domain? How to unify the pretraining of the pure-text document (1D) and image- based document (2D) in a single framework is also worth to try. Lastly, the use of different OCR tools is one of the major sources of inconsistency among the existing document pretraining works. It is worthwhile and essential to build standardized pretraining document image datasets with preprovided OCR results. In addition to scanned documents, using digital PDF as part of the pretraining data is direction worth exploring since it provides rich metadata which could be beneficial for multimodal learning.
10

---

References
[1] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong Hc, Alex Smola, and Eduard Hovy. Hicrarchical attention networks for document classification. In NAACL, 2016.
[2] Jade Goldstein, Vibhu O Mittal, Jaime G Carbonell, and Mark Kantrowitz. Multi-document summarization by sentence extraction. In NAACL-ANLP Workshop, 2000.
[3] Zihang Dai, Zhilin Yang. Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In ACL, 2019.
[4] Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou, Layoutlm: Pre-training of text and layout for document image understanding. In SIGKDD, 2020.
[5] Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang. Wanxiang Che, et al. Layouthm2:: Multi-modal pre-training for visually- rich document understanding. In ACL, 2021.
[6] Peizhao Li, Jiuxiang Gu. Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Varun Manju- natha, and Hongfu Liu. Selfdoc: Self-supervised document representation learning. In CVPR, 2021.
[7] Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jacheung Surh, Minjoon Seo, and Hwalsuk Lee. Cord: consolidated receipt dataset for post-ocr parsing. In NeurIPS Workshop, 2019.
[8] Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In ICDAR, 2015.
[9] Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang, Jianfci Cai, et al. Recent advances in convolutional neural networks. Pattern Recognition, 2018.
[10] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task task-agnostic visiolinguistic representations for vision-and-langune language tasks. In NeurIPS, 2019.
[11] Jiuxiang Gu, Jason Kuen, Shafiq Joty, Jianfei Cai, Vlad Morariu, Handong Zhao, and Tong Sun. Self-supervised relationship probing. In NewrlPS, 2020.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[13] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li. Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. In ICLR, 2019.
[14] Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. Unicoder-vl: A universal encoder for vision and language by moosl pre-training. In AAAI, 2020.
[15] Akshay Gadi Patil, Omri Ben-Eliezer, Or Perel, and Hadar Averbuch-Elor. Read: Recursive autoencoders for document layout generation. In CVPR Workshop, 2020.
[16] Ross Girshick. Fast r-cnn. In CVPR, 2015.
[17] Easyoer. https://githhie com
http:/thhhb.cmm/aadddaallaaa http:/JaadedAIIIaaa EaayyO0 2020.
[18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[19] Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. PAMI, 2010.
[20] Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS. 2017.
[21] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vee 2.0: framework for self-supervised learning of speech representations. In NeurIPS, 2020.

---

[22] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with softmax. In ICLR, 2017.
[23] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton, Layer normalization, arXiv preprint vrXiv:1607.06450, 2016.
[24] Jie Hu, Li Shen, and Gang Sun. Squeezz-and-exxitation networks. In CVPR, 2018.
[25] Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In ICCV, 2019.
[26] D. Lewis, G. Agam, S. Argamon, O. Frieder, D. Grossman, and Heard. Building a test collection for complex document information processing. In SIGIR, 2006.
[27] G. Jaume, H. Kemal Ekenel, and J. Thiran. Funsd: A dataset for form understanding in noisy scanned documents. In ICDAR Workshop, 2019.
[28] Xu Zhong. Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for document layout analysis. In ICDAR, 2019.
[29] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. EMNLP, 2019.
[30] Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL, 2018.
[31] Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpin and Lucia Specia. Semeval-2017 task I: Semantic textual similarity-mlltiinguaal and cross-lingual focused evaluation. In ACL, 2017.
[32] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2014.
[33] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://gee thub. http:////w....eoeeeeeeee/aaeaaaa
2019.
[34] Rafat Powalski, Lukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michat Pietruszka. and Gabriela Palka. Going full-tilt boogie on document understanding with text-image-layout transformer. In ICDAR, 2021.
[35] Anthony Kay. Tesseract: An open-source optical character recognition engine. Linux J., 2007(159):2, July 2007.
[36] Google cloud vision ocr. https://ccoot wwwgle..o./viiiooie....oooooooea wot.///iiio/////////ooo.. 2021.
12


Reviews:
[{"review_id": "a_omtb_za2A", "reviewer": "Reviewer_Zk5e", "summary": "This paper proposes a multi-modal pretraining method for document understanding. It involves a new hierarchical model architecture and three pretraining tasks. It compares with previous SOTA methods on three tasks and achieves stronger performance on two of them. \n\nThe major contribution is the newly proposed multi-modal pretraining tasks and the architecture is a combinations of models from both domains.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: Apart from some novelties of the pretraining objectives, the method reuses some of the existing architectures in a hierarchical way. However, it achieves significant improvements of two of the proposed tasks. \n\n**Quality and Significance**: Overall, this is a good application method and might be useful for many practical applications. \n\n**Clarity**: The writing needs to be improved. Many parts of the method are described verbally instead of formal definitions, making the paper hard to read. Some of the terms also need to be defined before using, such as RoI features, semantic regions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "To the best of my knowledge, the authors have identified and discussed the potential social impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "VOCspinvJ-M", "reviewer": "Reviewer_xpLb", "summary": "This paper proposes a self-supervised pre-training framework for document understanding called UniDoc. UniDoc takes multimodal data (image feature and text feature from document) as input and uses gated cross attention for learning cross-modal correlation. The superiority of UniDoc has been empirically shown on several downstream tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work proposes a pre-training method for document understanding by designing a model architecture and training objectives that can effectively utilize visual features as well as textual features.\nUniDoc achieves remarkable performance improvement in downstream tasks such as entity recognition, document detection, and reception understanding. Extensive experiments and ablation studies show that each component is beneficial.\n\nWhile used components are effective, each of them is from other works. In other words, UniDoc is a combination of well-known techniques, having a lack of significant novelty.\n\nThere is no performance comparison with word-level pre-training. Since the entity recognition task is word-oriented as mentioned in the paper, it would be better to perform pre-training on word-level rather than sentence-level.\nUniDoc does not improve the performance of document classification tasks over previous methods. Although the authors argue that it attributes to the difference of OCR engines, they should explain the magnitude of this difference. They say the difference between EasyOCR and Tesseract, but should report performance metric and qualitative analysis on the RVL-CDIP dataset to compare real Microsoft OCR and EasyOCR.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think interesting future research directions are described in the last section of the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7 hours"}, "extra_text": {}}, {"review_id": "ShrRiRUbNJV", "reviewer": "Reviewer_9qdW", "summary": "This paper presents UniDoc as a multimodal pre-trained model for document understanding. UniDoc consists of 3 layers: a feature extraction layer, which extracts sentence embeddings and region embeddings from the original document; a fusion layer, which fuses textual and visual information based on a gated cross-attention mechanism; a task layer, which uses 3 tasks as pre-training objectives to optimize model parameters. Evaluations are conducted on 4 downstream tasks, including Form Understanding, Receipt Understanding, Document Classification and Document Object Detection. Comparing to several SOTA baselines, UniDoc performs better on Form Understanding and Receipt Understanding, but worse on Document Classification. The paper also includes some ablation studies to investigate the effects of V/L/V+L, pre-training tasks and visual backbones, and some interesting future directions.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the papers: (1) A multimodal pre-trained model for document understanding, where both visual regions and texts extracted from visual regions are used as inputs; (2) 3 specific pre-training tasks designed document understanding are proposed, which show strong gains on some of the downstream tasks.\n\nThe weaknesses of the paper: (1) The effects of some factors are not fully discussed (listed in the question part); (2) The result on RVL-CDIP is not SOTA. The paper said that the potential reason could be using a weaker OCR, but not prove it directly. \n\nIn general, I think this is a good paper for document understanding. I give 6 this time but would like to change the score if the questions are addressed in the rebuttal period.\n\nQuestions: (1) What is the effect of the gated mechanism used in cross-attention? What the results look like if the gate is removed? (2) Can we replace the gated cross attention layer with a normal Transformer (a single-stream model)? (3) Is it possible to include Masked Language Models as well? If so, will the results become better with this typical pre-training task? (4) Are discrete representations indispensable? For example in Figure 1, can we use the RoI features from other documents in the same batch as negative examples for contrastive learning?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As the paper said: \"...the document images used for pretraining may contain sensitive information and therefore the models trained on such data may inappropriately leak some private information...\".", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "3jJ42vaGeLr", "reviewer": "Reviewer_5oFD", "summary": "This paper proposes a new unified framework for pre-training image/text encoders for capturing both visual and textual features in documents. The main Transformer model is pre-trained with multimodal embeddings as input features (visual & textual features) through three pre-training tasks: Masked Sentence Modeling, Visual Contrastive Learning, and Vision-Language Alignment. The pre-trained model can be fine-tuned on downstream tasks on which the proposed UniDoc model outperforms baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Originality]\n\nI feel that the methods/model architectures are largely putting together existing pieces of work from vision and language fields. In Sec 3.1, the proposed feature extraction, quantization, and gated cross-attention modules are all rather straightforward variations of existing work (i.e., the essential mechanisms are not new). Although they do make sense and are reasonable choices for a multimodal framework, I see little novelty here. In Sec 3.2, the three pre-training tasks also resemble well-known techniques, like masked language modeling in NLP and contrastive learning in vision. The alignment of vision and language is also not particularly surprising/insightful and provides little performance gain according to Table 2.\n\n[Quality]\n\nI do not find anything wrong with the proposed methods and models, since they are largely borrowed from well-known existing work. However, the overall technical contribution does not seem exciting/insightful to me. One thing that appears weird to me is the frozen sentence encoders initialized with the weights trained on NLI and STS datasets--if they have been fine-tuned on those tasks, they may capture too corpus-specific features to generalize to other domains (like the new pre-training documents). I don't think the computational constraints are valid excuses if the paper is proposing a new pre-training framework, which should be generic and not task-specific.\n\n[Clarity]\n\nThe paper is overall well-structured and the methods are explained clearly.\n\n[Significance]\n\nThe effectiveness of the proposed model is evaluated on four downstream tasks, by comparing with a set of baselines. The model outperforms the baselines with large margins. However, I am not sure how significant the results are to the NeurIPS community, because none of the baselines compared, as far as I am aware of, are from ML conferences. I would like to see comparisons with stronger baselines (e.g., there are many better PLMs than BERT from the language field), and ideally how the framework performs on pure language applications (e.g. the GLUE & SQuAD benchmarks) since a lot of applications may not have images and texts simultaneously. If the UniDoc model cannot be effectively used for pure language applications, I am not convinced that this can be called a \"unified\" framework.\n\n[Summary]\n\nPros:\n* The methods are clearly introduced with reasonable motivation.\n* The connection with existing work is described in detail.\n* The model has better empirical performance than a set of baselines on some specific applications.\n\nCons:\n* The methods are not novel/insightful as they are largely borrowing from existing work.\n* There are concerns about the generalization ability of the model since it uses dataset-specific text encoders.\n* The significance of the empirical contribution made is not clear to me (the baselines are not from the ML community and are not strong enough; it's not clear whether the model works for pure language applications as a \"unified\" framework)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors have made reasonable responses regarding the limitations and potential negative social impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new framework for pre-training for document understanding tasks like document and entity classification. The framework combines visual information from the input document image with textual information from OCR output using several self-supervised objectives. Results demonstrate improvements over state-of-the-art document encoding baselines on several downstream tasks. 

The majority of reviewers are in favor of acceptance. Generally, reviewers praised the paper's approach as well-motivated and clearly described, and results as compelling. Several reviewers question the novelty of the ML contribution of this paper. I discount this criticism to some extent given that this is clearly a novel *application* of existing pre-training components, arranged into a new framework, demonstrating gains on important downstream tasks -- this seems squarely fair game for NeurIPS.   

Other more minor criticisms include: 

(1) Concerns about outdated baselines, e.g. why not use more recent MLMs as baselines like RoBERTa and ELECTRA? Rebuttal argues that these are not comparable because they focus on text representation alone. I don't completely buy this since BERT was used as a baseline. 

(2) Some concerns about clarity (more formal mathematical descriptions would help) and ablations (there are many moving parts here, which model components are indispensable?). 

(3) One reviewer was concerned that there were no comparisons with word-level pre-training -- author response points out that region-level often is word-level, but also include additional experiments with word-level MLM for comparison showing worse performance. Thus, I believe this concern was mostly addressed in rebuttal. 

(4) Two reviewers were concerned that there was no improvement on document classification over the top baseline. This has been resolved in author response with new experiments demonstrating the effect of underlying OCR system.

Overall, I agree with the majority of reviewers and recommend acceptance. The issues mentioned by reviewers (and summarized above) absolutely need to be addressed in revision -- e.g. further ablations, improvement to mathematical clarity, comparisons with more recent MLMs, and the inclusion of all additional experiments provided in rebuttal. However, in this case, I do not believe the revisions are so substantial as to warrant another round of review.
Decision: Accept (Poster)