Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a self-supervised pre-training framework for document understanding called UniDoc. UniDoc takes multimodal data (image feature and text feature from document) as input and uses gated cross attention for learning cross-modal correlation. The superiority of UniDoc has been empirically shown on several downstream tasks.
main_review: This work proposes a pre-training method for document understanding by designing a model architecture and training objectives that can effectively utilize visual features as well as textual features.
UniDoc achieves remarkable performance improvement in downstream tasks such as entity recognition, document detection, and reception understanding. Extensive experiments and ablation studies show that each component is beneficial.

While used components are effective, each of them is from other works. In other words, UniDoc is a combination of well-known techniques, having a lack of significant novelty.

There is no performance comparison with word-level pre-training. Since the entity recognition task is word-oriented as mentioned in the paper, it would be better to perform pre-training on word-level rather than sentence-level.
UniDoc does not improve the performance of document classification tasks over previous methods. Although the authors argue that it attributes to the difference of OCR engines, they should explain the magnitude of this difference. They say the difference between EasyOCR and Tesseract, but should report performance metric and qualitative analysis on the RVL-CDIP dataset to compare real Microsoft OCR and EasyOCR.
limitations_and_societal_impact: I think interesting future research directions are described in the last section of the paper.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a new unified framework for pre-training image/text encoders for capturing both visual and textual features in documents. The main Transformer model is pre-trained with multimodal embeddings as input features (visual & textual features) through three pre-training tasks: Masked Sentence Modeling, Visual Contrastive Learning, and Vision-Language Alignment. The pre-trained model can be fine-tuned on downstream tasks on which the proposed UniDoc model outperforms baselines.
main_review: [Originality]

I feel that the methods/model architectures are largely putting together existing pieces of work from vision and language fields. In Sec 3.1, the proposed feature extraction, quantization, and gated cross-attention modules are all rather straightforward variations of existing work (i.e., the essential mechanisms are not new). Although they do make sense and are reasonable choices for a multimodal framework, I see little novelty here. In Sec 3.2, the three pre-training tasks also resemble well-known techniques, like masked language modeling in NLP and contrastive learning in vision. The alignment of vision and language is also not particularly surprising/insightful and provides little performance gain according to Table 2.

[Quality]

I do not find anything wrong with the proposed methods and models, since they are largely borrowed from well-known existing work. However, the overall technical contribution does not seem exciting/insightful to me. One thing that appears weird to me is the frozen sentence encoders initialized with the weights trained on NLI and STS datasets--if they have been fine-tuned on those tasks, they may capture too corpus-specific features to generalize to other domains (like the new pre-training documents). I don't think the computational constraints are valid excuses if the paper is proposing a new pre-training framework, which should be generic and not task-specific.

[Clarity]

The paper is overall well-structured and the methods are explained clearly.

[Significance]

The effectiveness of the proposed model is evaluated on four downstream tasks, by comparing with a set of baselines. The model outperforms the baselines with large margins. However, I am not sure how significant the results are to the NeurIPS community, because none of the baselines compared, as far as I am aware of, are from ML conferences. I would like to see comparisons with stronger baselines (e.g., there are many better PLMs than BERT from the language field), and ideally how the framework performs on pure language applications (e.g. the GLUE & SQuAD benchmarks) since a lot of applications may not have images and texts simultaneously. If the UniDoc model cannot be effectively used for pure language applications, I am not convinced that this can be called a "unified" framework.

[Summary]

Pros:
* The methods are clearly introduced with reasonable motivation.
* The connection with existing work is described in detail.
* The model has better empirical performance than a set of baselines on some specific applications.

Cons:
* The methods are not novel/insightful as they are largely borrowing from existing work.
* There are concerns about the generalization ability of the model since it uses dataset-specific text encoders.
* The significance of the empirical contribution made is not clear to me (the baselines are not from the ML community and are not strong enough; it's not clear whether the model works for pure language applications as a "unified" framework)
limitations_and_societal_impact: I think the authors have made reasonable responses regarding the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a multi-modal pretraining method for document understanding. It involves a new hierarchical model architecture and three pretraining tasks. It compares with previous SOTA methods on three tasks and achieves stronger performance on two of them. 

The major contribution is the newly proposed multi-modal pretraining tasks and the architecture is a combinations of models from both domains.
main_review: **Originality**: Apart from some novelties of the pretraining objectives, the method reuses some of the existing architectures in a hierarchical way. However, it achieves significant improvements of two of the proposed tasks. 

**Quality and Significance**: Overall, this is a good application method and might be useful for many practical applications. 

**Clarity**: The writing needs to be improved. Many parts of the method are described verbally instead of formal definitions, making the paper hard to read. Some of the terms also need to be defined before using, such as RoI features, semantic regions.
limitations_and_societal_impact: To the best of my knowledge, the authors have identified and discussed the potential social impact of this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents UniDoc as a multimodal pre-trained model for document understanding. UniDoc consists of 3 layers: a feature extraction layer, which extracts sentence embeddings and region embeddings from the original document; a fusion layer, which fuses textual and visual information based on a gated cross-attention mechanism; a task layer, which uses 3 tasks as pre-training objectives to optimize model parameters. Evaluations are conducted on 4 downstream tasks, including Form Understanding, Receipt Understanding, Document Classification and Document Object Detection. Comparing to several SOTA baselines, UniDoc performs better on Form Understanding and Receipt Understanding, but worse on Document Classification. The paper also includes some ablation studies to investigate the effects of V/L/V+L, pre-training tasks and visual backbones, and some interesting future directions.  
main_review: The strengths of the papers: (1) A multimodal pre-trained model for document understanding, where both visual regions and texts extracted from visual regions are used as inputs; (2) 3 specific pre-training tasks designed document understanding are proposed, which show strong gains on some of the downstream tasks.

The weaknesses of the paper: (1) The effects of some factors are not fully discussed (listed in the question part); (2) The result on RVL-CDIP is not SOTA. The paper said that the potential reason could be using a weaker OCR, but not prove it directly. 

In general, I think this is a good paper for document understanding. I give 6 this time but would like to change the score if the questions are addressed in the rebuttal period.

Questions: (1) What is the effect of the gated mechanism used in cross-attention? What the results look like if the gate is removed? (2) Can we replace the gated cross attention layer with a normal Transformer (a single-stream model)? (3) Is it possible to include Masked Language Models as well? If so, will the results become better with this typical pre-training task? (4) Are discrete representations indispensable? For example in Figure 1, can we use the RoI features from other documents in the same batch as negative examples for contrastive learning?
limitations_and_societal_impact: As the paper said: "...the document images used for pretraining may contain sensitive information and therefore the models trained on such data may inappropriately leak some private information...".
review_title: null

Meta-review: This paper presents a new framework for pre-training for document understanding tasks like document and entity classification. The framework combines visual information from the input document image with textual information from OCR output using several self-supervised objectives. Results demonstrate improvements over state-of-the-art document encoding baselines on several downstream tasks. 

The majority of reviewers are in favor of acceptance. Generally, reviewers praised the paper's approach as well-motivated and clearly described, and results as compelling. Several reviewers question the novelty of the ML contribution of this paper. I discount this criticism to some extent given that this is clearly a novel *application* of existing pre-training components, arranged into a new framework, demonstrating gains on important downstream tasks -- this seems squarely fair game for NeurIPS.   

Other more minor criticisms include: 

(1) Concerns about outdated baselines, e.g. why not use more recent MLMs as baselines like RoBERTa and ELECTRA? Rebuttal argues that these are not comparable because they focus on text representation alone. I don't completely buy this since BERT was used as a baseline. 

(2) Some concerns about clarity (more formal mathematical descriptions would help) and ablations (there are many moving parts here, which model components are indispensable?). 

(3) One reviewer was concerned that there were no comparisons with word-level pre-training -- author response points out that region-level often is word-level, but also include additional experiments with word-level MLM for comparison showing worse performance. Thus, I believe this concern was mostly addressed in rebuttal. 

(4) Two reviewers were concerned that there was no improvement on document classification over the top baseline. This has been resolved in author response with new experiments demonstrating the effect of underlying OCR system.

Overall, I agree with the majority of reviewers and recommend acceptance. The issues mentioned by reviewers (and summarized above) absolutely need to be addressed in revision -- e.g. further ablations, improvement to mathematical clarity, comparisons with more recent MLMs, and the inclusion of all additional experiments provided in rebuttal. However, in this case, I do not believe the revisions are so substantial as to warrant another round of review.
Decision: Accept (Poster)