Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: > Summary: 

The paper studies the problem of large-scale pre-training for semi-structured documents. It proposes a new pre-training strategy called BERT relying on Spatiality (BROS) with area-masking and utilizes a graph-based decoder to capture the semantic relation between text blocks to alleviate the serialization problem of LayoutLM. 

It points out that LayoutLM fails to fully utilize spatial information of text blocks and will face difficulties when text blocks cannot be easily serialized.  

The three drawbacks of LayoutLM are listed:
* X-axis and Y-axis are treated individually with point-specific embedding
* Pre-training is identical to BERT so does not consider spatial relations between text blocks
* Suffer from the serialization problem

The proposed three corresponding methods of BROS are:
* Continuous 2D positional encoding 
* Area-masking pre-training on 2D language modeling
* Graph-based decoder for solving EE & EL tasks

> Strength:

* The paper makes incremental advances over past work (LayoutLM) and the proposed BROS models achieves SOTA performance on four EE/EL datasets (i.e., FUNSD, SORIE*, CORD, and SciTSR)

* The paper is generally easy to follow and could be better if provide more important details in Section 3.2 & 3.3

* The experiment and discussion for Section 5.3 are quite convincing. BROS could achieve robust and consistent performances across all the four permuted version datasets, which demonstrates that BROS is adaptive to documents from the practical scenarios. 

> Major concerns:

- For Section 3.2, the author didn’t even provide the pre-training objective for the area-masked language model. I think the author should include the objective and define the exponential distribution explicitly.

* I’m confused about why the performance difference in Table 4 between original LayoutLM and BROS is larger than that in Table 1. In the original LayoutLM, the 2D position encoding method is tied with token embedding. This applies to both Table 1 and Table 4. However, in Table 4 the performance difference on FUNSD EE is 42.46 vs 74.44, while in Table 1 the performance comparison is 78.89 vs 81.21. Could the author give some explanations on this?

- In Table 4, it is better for the author to clearly indicate each ablation’s components. The author should also give the performance of the original LayoutLM and performances after gradually adding each new component to the original LayoutLM: such as original LayoutLM + Sinusoid & Linear, original LayoutLM + Sinusoid & Linear + untied with token embedding, etc.

* For encoder design in Eq. (2), the second term is used to model the spatial dependency given the source semantic representation.  How about adding a symmetric term to model the semantic dependency given the source spatial representation. My question is simply that why not adopt a symmetric design for Eq. (2)?

* Can the author give the reason behind the design of $\mathbf{t}^{ntc}$ in Eq.(4)? I’m not so clear about the function of $\mathbf{t}^{ntc}$.  Does the $\mathbf{f}_{ntc}$ output a distribution of the probability to be the next token over all N tokens? 

* Could the author give a detailed analysis on which strategy contributes the most to BROS’ robustness against permuted order information? From the results of Table 4, it is not the SPADE decoder and the most important factor seems to be calculating semantic and position attentions separately, i.e., untied with token embedding and explicitly modeling semantic/position relations between text blocks. Do the authors agree with my conjecture?

> Minor concerns:

* Although SPADE is not the most important component of BROS, I believe including details of the grade-based decoder will help the readers to understand the model much better.

* I’m curious about the performance of SpanBERT on the four datasets since the author said that area-masking of BROS is inspired by SpanBERT.

* In Table 3, the value of LayoutLM - FUNSD should be 78.89 since all other p-FUNSD & FUNSD related values are consistent with Table 1 & 2.
review_title: A robust and effective pre-training strategy for document understanding and is independent of optimal order information but lacks of some details

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes the pre-trained language model BROS which aims to leverage both text and spatial information to improve information extraction on documents. Using the graph-based decoder from SPADE, BROS achieves the SOTA performance on some entity extraction and entity linking downstream tasks. However, the area-masking strategy does not show significant improvement over the LayoutLM and the graph decoder is proposed in SPADE which is not new. In addition, as most commercial OCR tools have already got very good reading order information, the benefit from the graph decoder might be marginal.

Pros
-	The paper introduces the area-masking pre-training strategy that can be seen as a natural generalization of masking language model in the 2D plane.
-	The authors integrate spatial information into the attention mechanism as a pair-wise bias term, which is reasonable.
-	BROS utilizes the graph-based decoder from SPADE and improves performance on downstream tasks.

Cons
-	The area-masking strategy is to mask small area centered at some tokens, which is actually similar to masking the center token only. Also, given that the FUNSD dataset is small, the area-masking strategy does not show significant improvement over vanilla MLM.
-	This paper shows that sinusoid & linear functions can encode 2D position efficiently. However, it is not reasonable to compare sinusoid & linear and learnable embeddings on small data, since learnable embeddings could leverage large amount of data and get more gains.
-	The graph-based decoder part is identical to which in SPADE so it is not suitable to appear as the contributions of this paper.

In summary, this paper largely overlaps with the previous research work. I do not think it is qualified for the ICLR conference. 

review_title: This paper largely overlaps with pervious research work with minor modifications

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Overall

Authors used BERT alongside to a 2D-position embedding based on a sinusoidal function and a graph-based decoder to improve performance on document information extraction tasks. They do pre-train their model (BROS) on a large dataset with 11M documents, and then used such models to perform downstream tasks in four smaller datasets. Their models achieve better quantitative results when compared to the provided baselines.

### Positive aspects 

* Positional encoder based on sinusoidal function seems to be effective.
* Authors perform experimentally sound experiments, following closely LayoutLM.
* Pre-trained models could be useful.
* Authors reproduced results from their strongest baseline.
* Better results in all downstream tasks.

### Cons and aspects to improve

My main concern is that the overall contribution is seems to be limited.In fact, the original paper of the Transformer approach, already proposed such kind of embedding. It is good to know that it works for 2D-coordinates for the task at hand, though it seems to be more a marginal improvement on existing work rather than a standalone contribution.

It is hard to tell what are the standalone contributions of the paper, and what is coming from other works.

Authors could have provided more in-depth details (visualizations, analysis, examples) to show main differences between the proposed approach and baselines (specially LayoutLM). Also they could visually demonstrate the advantages of their approach.

Authors could have plugged their embedding strategy in LayoutLM to understand the impact of that particular component.

I would like to have seen qualitative examples of model predictions, and more examples from the dataset.

A figure containing the whole process could be helpful to better understand the processing required to train / test such models. Figure from LayoutLM is a good example of that, it comprises the entire process and makes it easier to understand the whole architecture.

* In the abstract, authors say "BROS utilizes a powerful graph-based decoder that can capture the relation between text segment"* Though in the text such a component (that is from other work) is only mentioned twice without further detail.

It is unclear to me:

* How regions of interest are detected in this work? (I assumed authors used the same strategy as LayoutLM).
* OCR seems to be an extra-step in the preprocessing stage. What to do if the user does not have the same OCR. What is the impact of a good OCR for training and testing (prediction of new, unseen documents)?
* "In-house OCR engine was applied" can authors provide more details on that?

This line of work could be much stronger if the models comprised the whole process (detection, text extraction, recognition) in an end-to-end manner.

### Notes on text and style

There are parts of the manuscript that felt somewhat informal and confusing to me. I will provide some details as follows.

* Set a default format for numbers in tables. Table 1 has two distinct decimal number formats.
* Personally, I think it is better to write $5 \times 10^{-5}$ rather than 5e-5.
* In the results section there is a typo: *"performances with a large margins of 2.32pp in"*. Also, text could be more formal. I would avoid using the use of the *pp* abbreviation.
* "By achieving the best, these results prove that BROS" this sentence can be improved.
* "Moreover, it should be noted that BROS achieves higher f1 score than 79.27 of LayoutLM using visual features". I think authors wanted to say that even though BROS does not rely on visual features, it does outperform LayoutLM which, in turn, uses visual features. 

review_title: Official Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The paper provides a novel pretrained language model for document understanding named BROS, which adds spatial layout information and new area-masking strategy.
The authors do some experiments on four public datasets to illustrate the effectiveness of BROS.
The new architecture is well-suited for understanding texts in document, which is valuable.

##########################################################################

Reasons for score: 

 
Overall, I vote for accepting. 
I deem that the pre-trained language model based on BERT that encodes spatial information is useful for 2D document.
Hopefully the authors can address my concern in the rebuttal period (see cons below). 

 
##########################################################################Pros: 

 
1. The paper addresses siome limitations which are very important for document understanding: spatial information, spatial
relation, and the information of text blocks.

 
2.  This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed model.  The entire structure is organized well and the formulas are very detailed.

 
##########################################################################

Cons: 

1. What are the advantages of BROS in terms of speed and resource consumption?
It would be more convincing if the authors can provide more cases in the rebuttal period. 

2. For the Figure 1(b), it would be better to provide more details about it, which seems not very clear to me. Like how to
mask in red area?


 
##########################################################################

Questions during rebuttal period: 

 
Please address and clarify the cons above 

 
#########################################################################
review_title: Recommendation to Accept

Meta-review: The paper proposes a new pre-trained language model for information extraction on documents. It consists of a new pre-training strategy with area-masking and a new graph-based decoder to capture the relationships between text blocks. Experimental results show better performances of the proposed approach.

Pros • The paper is generally clearly written. • Experimental results show better performances on the benchmark datasets.

Cons • Novelty of the work might not be enough. For example, the graph-based decoder is not new. The masking technique is also a natural extension of that in BERT. • Significance of the work might not be enough. For example, the improvement from the area masking is not so significant. • There are additional experiments which can be added, as pointed out by Reviewer 3. • Presentation can be further improved. Some of the issues indicated by the reviewers have been addressed in the rebuttal. We appreciate the authors’ efforts.

During the rebuttal, the authors have addressed the clarity issues pointed out by the reviewers. However, the main issues in novelty and significance still exist. The reviewers think that the quality of the work is still not high enough as an ICLR paper.


Decision: Reject