BROS: A Pre-trained Language Model for Understanding Texts in Document

1 INTRODUCTION . Document intelligence ( DI ) 1 , which understands industrial documents from their visual appearance , is a critical application of AI in business . One of the important challenges of DI is a key information extraction task ( KIE ) ( Huang et al. , 2019 ; Jaume et al. , 2019 ; Park et al. , 2019 ) that extracts structured information from documents such as financial reports , invoices , business emails , insurance quotes , and many others . KIE requires a multi-disciplinary perspective spanning from computer vision for extracting text from document images to natural language processing for parsing key information from the identified texts . Optical character recognition ( OCR ) is a key component to extract texts in document images . As OCR provides a set of text blocks consisting of a text and its location , key information in documents can be represented as a single or a sequence of the text blocks ( Schuster et al. , 2013 ; Qian et al. , 2019 ; Hwang et al. , 2019 ; 2020 ) . Although OCR alleviates the burden of processing images , understanding semantic relations between text blocks on diverse layouts remains a challenging problem . To solve this problem , existing works use a pre-trained language model to utilize its effective representation of text . Hwang et al . ( 2019 ) fine-tunes BERT by regarding KIE tasks as sequence tagging problems . Denk & Reisswig ( 2019 ) uses BERT to incorporate textual information into image pixels during their image segmentation tasks . However , since BERT is designed for text sequences , they artificially convert text blocks distributed in two dimensions into a single text sequence losing spatial layout information . Recently , Xu et al . ( 2020 ) proposes LayoutLM pre-trained on large-scale documents by utilizing spatial information of text blocks . They show the effectiveness of the pretraining approach by achieving high performance on several downstream tasks . Despite this success , LayoutLM has three limitations . First , LayoutLM embeds x- and y-axis individually using trainable parameters like the position embedding of BERT , ignoring the gap between positions in a sequence and 2D space . Second , its pre-training method is essentially identical to BERT that does not explicitly consider spatial relations between text blocks . Finally , in its downstream tasks , LayoutLM only conducts sequential tagging tasks ( e.g . BIO tagging ) that require serialization of text blocks . 1https : //sites.google.com/view/di2019 These limitations indicate that LayoutLM fails not only to fully utilize spatial information but also to address KIE problems in practical scenarios when a serialization of text blocks is difficult . This paper introduces an advanced language model , BROS , pre-trained on large-scale documents , and provides a new guideline for KIE tasks . Specifically , to address the three limitations mentioned above , BROS combines three proposed methods : ( 1 ) a 2D positional encoding method that can represent the continuous property of 2D space , ( 2 ) a novel area-masking pre-training strategy that performs masked language modeling on 2D , and ( 3 ) a combination with a graph-based decoder for solving KIE tasks . We evaluated BROS on four public KIE datasets : FUNSD ( form-like documents ) , SROIE * ( receipts ) , CORD ( receipts ) , and SciTSR ( table structures ) and observed that BROS achieved the best results on all tasks . Also , to address KIE problem under a more realistic setting we removed the order information between text blocks from the four benchmark datasets . BROS still shows the best performance on these modified datasets . Further ablation studies provide how each component contributes to the final performances of BROS . 2 RELATED WORK . 2.1 PRE-TRAINED LANGUAGE MODELS . BERT ( Devlin et al. , 2019 ) is a pre-trained language model using Transformer ( Vaswani et al. , 2017 ) that shows superior performance on various NLP tasks . The main strategy to train BERT is a masked language model ( MLM ) that masks and estimates randomly selected tokens to learn the semantics of language from large-scale corpora . Many variants of BERT have been introduced to learn transferable knowledge by modifying the pre-training strategy . XLNet ( Yang et al. , 2019 ) permutes tokens during the pre-training phase to reduce a discrepancy from the fine-tuning phase . XLNet also utilizes relative position encoding to handle long texts . StructBERT ( Wang et al. , 2020 ) shuffles tokens in text spans and adds sentence prediction tasks for recovering the order of words or sentences . SpanBERT ( Joshi et al. , 2020 ) masks span of tokens to extract better representation for span selection tasks such as question answering and co-reference resolution . ELECTRA ( Clark et al. , 2020 ) is trained to distinguish real and fake input tokens generated by another network for sample-efficient pre-training . Inspired by these previous works , BROS utilizes a new pre-training strategy that can capture complex spatial dependencies between text blocks distributed on two dimensions . Note that LayoutLM is the first pre-trained language model on spatial text blocks but they still employs the original MLM of BERT . 2.2 KEY INFORMATION EXTRACTION FROM DOCUMENTS . Most of the existing approaches utilize a serializer to identify the text order of key information . POT ( Hwang et al. , 2019 ) applies BERT on serialized text blocks and extracts key contexts via a BIO tagging approach . CharGrid ( Katti et al. , 2018 ) and BERTGrid ( Denk & Reisswig , 2019 ) map text blocks upon a grid space , identify the region of key information , and extract key contexts in the pre-determined order . Liu et al . ( 2019 ) , Yu et al . ( 2020 ) , and Qian et al . ( 2019 ) utilize graph convolutional networks to model dependencies between text blocks but their decoder that performs BIO tagging relies on a serialization . LayoutLM ( Xu et al. , 2020 ) is pre-trained on large-scale documents with spatial information of text blocks , but it also conducts BIO tagging for their downstream tasks . However , using a serializer and relying on the identified sequence has two limitations . First , the information represented in two dimensional layout can be lost by improper serialization . Second , there may even be no correct serialization order . A natural way to model key contexts from text blocks is a graph-based formulation that identifies all relationships between text blocks . SPADE ( Hwang et al. , 2020 ) proposes a graph-based decoder to extract key contexts from identified connectivity between text blocks without any serialization . Specifically , they utilize BERT without its sequential position embeddings and train the model while fine-tuning BERT . However , their performance is limited by the amount of data as all relations have to be learned from the beginning at the fine-tuning stage . To fully utilize the graph-based decoder , BROS is pre-trained on a large number of documents and is combined with the SPADE decoder to determine key contexts from text blocks . 3 BERT RELYING ON SPATIALITY ( BROS ) . The main structure of BROS follows BERT , but there are three novel differences : ( 1 ) a spatial encoding metric that reflects the continuous property of 2D space , ( 2 ) a pre-training objective designed for text blocks on 2D space , and ( 3 ) a guideline for designing downstream models based on a graphbased formulation . Figure 1 shows visual description of BROS for document KIE tasks . 3.1 ENCODING SPATIAL INFORMATION INTO BERT . 3.1.1 REPRESENTATION OF A TEXT BLOCK LOCATION . The way to represent spatial information of text blocks is important to encode information from layouts . We utilize sinusoidal functions to encode continuous values of x- and y-axis , and merge them through a linear transformation to represent a point upon 2D space . For formal description , we use p = ( x , y ) to denote a point on 2D space and f sinu : R → RDs to represent a sinusoidal function . Ds is the dimensions of sinusoid embedding . BROS encodes a 2D point by applying the sinusoidal function to x- and y-axis and concatenating them as p̄ = [ f sinu ( x ) ⊕ f sinu ( y ) ] . The ⊕ symbol indicates concatenation . The bounding box of a text block , bbi , consists of four vertices , such as ptli , p tr i , p br i , and p bl i that indicate top-left , top-right , bottom-right , and bottom-left points , respectively . The four points are converted into vectors such as p̄tli , p̄ tr i , p̄ br i , and p̄bli with f sinu . Finally , to represent a spatial embedding , bbi , BROS combines four identified vectors through a linear transformation , bbi = W tlp̄tli + W trp̄tri + W brp̄bri + W blp̄bli , ( 1 ) where W tl , W tr , W br , W bl ∈ RH×2Ds are linear transition metrics and H is a hidden size of BERT . The periodic property of the sinusoidal function can encode continuous 2D coordinates more natural than using point-specific embedding used in BERT and LayoutLM . In addition , by learning the linear transition parameters , BROS provides an effective representation of a bounding box . 3.1.2 ENCODING SPATIAL REPRESENTATION . Position encoding methods affect how models utilize the position information . In BERT , position embedding is tied with the token through a point-wise summation . However , 2D spatial information is richer than 1D sequence due to the their continuous property and the high dimensionality . Moreover , text blocks can be placed over various locations on documents without significant changes in its semantic meaning . For example , locations of page numbers differ over multiple document snapshots even though they are captured from a single document . Therefore , more advanced approach is required to maximally include spatial information during encoding beyond the simple summation approach used in BERT . In BROS , the spatial information is directly encoded during the contextualization of text blocks . Specifically , BROS calculates an attention logit combining both semantic and spatial features . The former is the same as the original attention mechanism in Transformer ( Vaswani et al. , 2017 ) , but the latter is a new component identifying the importance of the target location when the source context and location are given . Our proposed attention logit is formulated as follows , Ai , j = ( W qti ) > ( W ktj ) + ( W qti W sq|qbbi ) > ( W sk|qbbj ) + ( W sqbbi ) > ( W skbbj ) , ( 2 ) where ti and tj are context representations for ith and jth tokens and W q , W k , W sq|q , W sk|q , W sq , W sk are linear transition matrices . The symbol indicates Hadamard product . The first term indicates an attention logit from contextual representations and the third term is from spatial representation . The second term is designed to model the spatial dependency given the source semantic representation , ti . The second and third terms are independently calculated at each layer because spatial dependencies might differ over layers . 3.2 PRE-TRAINING OBJECTIVE : AREA-MASKED LANGUAGE MODEL . Pre-training diverse layouts from unlabeled documents is a key factor for document understanding tasks . To learn effective spatial representation including relationships between text blocks , we propose a novel pre-training objective . Inspired by SpanBERT ( Joshi et al. , 2020 ) , we expand spans of a 1D sequence to consecutive text blocks in 2D space . Specifically , we select a few regions in a document layout , mask all tokens of text blocks in the selected regions , and estimate the masked tokens . The rules for masking tokens in area-masked language model are as the following procedure . ( a ) Select a text block randomly and get the top-left and bottom-right points ( ptl and pbr ) of the block . ( b ) Identify the width , height , and center of the block as ( w , h ) = |ptl − pbr| and c = ( ptl + pbr ) /2 . ( c ) Expand the width and height as ( ŵ , ĥ ) = l ∗ ( w , h ) where l ∼ exp ( λ ) and λ is a distribution parameter . ( d ) Identify rectangular masking area of which top-left and bottom-right are p̂tl = ptl − ( ŵ , ĥ ) , and p̂br = pbr + ( ŵ , ĥ ) , respectively . ( e ) Mask all tokens of text blocks whose centers are allocated in the area . ( f ) Repeat ( a ) – ( e ) until 15 % of tokens are masked . The rationale behind using exponential distribution is to convert the geometric distribution used in SpanBERT for a discrete domain into distribution for a continuous domain . Thus , we set λ = −ln ( 1 − p ) where p = 0.2 used in SpanBERT . In addition , we truncated exponential distribution with 1 to prevent an infinity multiplier covering all space of the document . It should be noted that the masking area is expanded from a randomly selected text block since the area should be related to the text sizes and locations to represent text spans in 2D space . Figure 2 compares token- and area-masking on text blocks . Finally , the loss function for the area-masked language model is formed as ; LAMLM = − ∑ x̂∈A ( x ) log p ( x̂|x\A ( x ) ) , ( 3 ) where x , A ( x ) , and x\A ( x ) denote tokens in given image , masked tokens of which text block is located in masking area , and the rest tokens , respectively . Similar to BERT ( Devlin et al. , 2019 ) , the masked tokens are replaced by [ MASK ] token 80 % of the time , a random token 10 % of the time , or an unchanged token 10 % of the time .

Summary: Authors used BERT alongside to a 2D-position embedding based on a sinusoidal function and a graph-based decoder to improve performance on document information extraction tasks. They do pre-train their model (BROS) on a large dataset with 11M documents, and then used such models to perform downstream tasks in four smaller datasets. Their models achieve better quantitative results when compared to the provided baselines.