# A Structured Recognition Method for Invoices Based on StrucTexT Model
## 2 Related Models and Optimization  
### StrucTexT  
OCR structuring technology often addresses two high-frequency application task types: entity labeling and entity linking. Entity labeling refers to the extraction of text content from OCR recognition results corresponding to predefined entity labels, such as \"Amount\", \"Purchaser Name\", etc. Entity linking refers to analyzing the relationship between text entities, for example, whether they can form key_value pairs or whether they belong to the same row or column in a table. In invoice recognition, invoices are visual, text-rich images with multiple attributes such as text, image, and layout. Multimodal cues are used for modeling, and Chinese and English segment-level multimodal features are incorporated into structured OCR pre-training for feature enhancement. Figure 2 shows a schematic diagram of the overall model framework of the model StrucTexT [5] used in this paper. In a given input image with pre-processed OCR results such as bounding boxes and text segment content, for example, various pieces of information in terms of text, image, and layout are utilized through the feature-embedding phase. The multimodal embedding is then fed back to the pre-trained transformer to obtain rich semantic features. The transformer accomplishes cross-modal fusion by creating interactions between different modal inputs. Figure 3 shows a diagram of the model's pre-training task. Finally, the structured text understanding module receives the encoded features, performs entity recognition for entity labeling, and then performs relationship extraction of entity connections, as shown in Figure 4.  
Given a document image \(I\) with \(n\) text segments, StrucTexT carries out open-source OCR algorithms to obtain the \(i\)-th segment region with the top_left and bottom_right bounding boxes \(b_{i}=(x_{0},y_{0},x_{1},y_{1})\), and its corresponding text sentence \(t_{i}=\left\{c_{1}^{i},c_{2}^{i},\dots,c_{l_{i}}^{i}\right\}\)  
Figure 4: Structured text understanding. Figure 3: Pre-training tasks. Figure 2: An overall illustration of the model framework of StrucTexT. where \(c\) is a word or character and \(l_{i}\) is the length of \(t_{i}\). For every segment or word, StrucTexT uses the encoded bounding boxes as their layout information. \[L=Emb_{l}(x_{0},y_{0},x_{1},y_{1},w,h) \tag{1}\]  
where \(Emb_{l}\) is a layout embedding layer and \(w\) and \(h\) are the shapes of the bounding box \(b\). It is worth mentioning that we estimate the bounding box of a word using its own text segment, considering OCR results that lack word-level information. Following a common method, StrucTexT utilizes WordPiece [6] to tokenize text sentences. After that, all of the text sentences are gathered as a sequence \(S\) by sorting the text segments from the top_left to the bottom_right. Intuitively, a pair of special tags [_CLS_] and [_SEP_] are added at the beginning and end of the sequence, as \(t_{0}=\{[CLS]\}\) and \(t_{n+1}=\{[SEP]\}\). Thus, StrucTexT can define the language sequence \(S\) as follows:  
\[ \begin{array}{ll}S&=\{t_{0},t_{1},\ldots,t_{n},t_{n+1}\}\\ &=\left\{[CLS],c_{1}^{1},\ldots,c_{l_{1}}^{1},\ldots,c_{1}^{n},\ldots,c_{l_{n }}^{n},\{SEP\}\right\}\end{array}  \tag{2}\]  
Then, StrucTexT sums the embedded feature of \(S\) and the layout embedding \(L\) to obtain the language embedding \(T\):  
\[T=Emb_{l}(S)+L \tag{3}\]  
where \(Emb_{l}\) is a text-embedding layer. Within the model's architecture, StrucTexT uses ResNet50 [7] with FPN [8] as the image feature extractor to generate feature maps of \(I\). Afterwards, the image feature of each text segment is extracted from the \(CNN\) maps using RoIAlign [9], according to \(b\). The visual segment embedding \(V\) is computed as  
\[V=Emb_{v}(ROIAlign(CNN(I),b))+L \tag{4}\]  
where \(Emb_{v}\) is the visual embedding layer. Furthermore, the entire feature map of image \(I\) is embedded as \(V_{0}\), in order to introduce global information into image features.