# A Structured Recognition Method for Invoices Based on StrucTexT Model
## 2 Related Models and Optimization
### StrucTexT
Compared with the vision_language tasks based on wild pictures, understanding the structured document requires higher semantics to identify the ambiguous entities. Thus, StrucTexT proposes segment ID embedding \(S^{id}\) to allocate a unique number to each text segment, its image, and its text features, which will then create an explicit alignment of cross-modality clues. In addition, StrucTexT adds two other embeddings to the input. The position embedding \(P^{id}\) encodes the indexes from 1 to the maximum sequence length, and the segment embedding \(M^{id}\) denotes the modality for each feature. All of the above embeddings have the same dimensions. In the end, the input of the StrucTexT model is represented as the combination of the embeddings. \[Input=Concat(T,V)+S^{id}+P^{id}+M^{id} \tag{5}\]  
Moreover, StrucTexT appends several [PAD] tags to make the short input sequence a fixed length. An empty bounding box with zeros is assigned to the special [CLS], [SEP], and [PAD] tags. StrucTexT collects multi-modal information from visual segments, text sentences, and position layouts to produce an embedding sequence. We support image_text alignment between different granularities by leveraging the segment IDs mentioned above. At this stage, we introduce a transformer network to encode the embedding sequence and establish deep fusion between modalities and granularities. Crucially, during the pre-training stage, three self-supervised tasks encode the input features to learn task-agnostic joint representations. The details of these tasks are introduced below, and the patterns of all self-supervised tasks are shown in Figure 5. The Chinese characters in the Figure 5 are visual segments in the invoice. In the following section, the three self-supervised tasks of masked visual language modeling, sentence length prediction, and paired box direction are described separately. (a) Masked visual language modeling (MVLM)  
This task is used to facilitate the capture of contextual representations in terms of language. Following the masked multimodal modeling model of ViLBERT, the model selects about 15% of the tokens from the language sequence, masks 80% of them with [MASK] tags, replaces 10% of them with random tokens, and keeps 10% of the tokens unchanged. The model is required to reconstruct the corresponding notation. The model does not follow the masking of ViLBERT's image region precisely, but it retains all information and encourages the model to look for cross-modal cues whenever possible. (b) Sentence length prediction (SLP)  
This task is used to mine fine-grained semantic information on the image side. SLP requires the model to recognize the length of the segment from each visual feature. In this way, the encoder is forced to learn image features using the same segment ID, and more importantly, it is also forced to acquire linguistic sequence knowledge. This information flow accelerates the deep cross-modal fusion of textual, visual, and layout information. In addition, to avoid the interference caused by the generation of sub-words, the model only counts the first occurrence of sub-words in order to keep the length between the language sequence and the image fragment the same. Thus, the model simply and efficiently creates additional alignment between the two granularities. (c) Paired box direction (PBD)  
As a third self-supervised task, PBD aims to exploit global layout information. The purpose of the PBD task is to learn the integrated geometric topology of the document structure by predicting the pairwise spatial relationships between text segments. First, a 360\({}^{\circ}\) region is divided into eight identical buckets. Second, the angle \(\theta_{ij}\) between text segments \(i\) and \(j\) is computed and labeled with one of the buckets. Next, subtraction is performed between two visual features on the image side in order to obtain the result \(\Delta\hat{V}_{ij}\) as input to the PBD:  
\[\Delta\hat{V}_{ij}=\hat{V}_{i}-\hat{V}_{j} \tag{6}\]  
Figure 5: Self-supervised tasks.