# A Structured Recognition Method for Invoices Based on StrucTexT Model
## 2 Related Models and Optimization
### Optimization  
According to the pre-training task of the \"masked visual language model\" in the invoice-related document images and the StrucTexT model in the domestic scene, the pre-training task is extended to the segment level. By randomly selecting some segments and masking all tokens in the segment, the model is required to restore these tokens in order to learn more complex semantic information. The original model uses the masked visual language model, which can only learn relatively simple semantic information. For rich visual text images such as value-added tax special invoices, the semantic information is relatively complex. By extending the pre-training task using the masked segment prediction, the model's understanding of complex semantic information is improved, as is its ability to understand and recognize various pieces of complex invoice information. Meanwhile, it should be noted in the use of the model that although the StrucTexT model has excellent performance and a fast recognition speed for single-ticket recognition, in practical application scenarios, said tickets are often numerous and disorderly. StrucTexT contains 107 M parameters, and although the model's performance is high, the complexity of the model is also high, and the cost of applying large models in practical scenarios is high. Considering that the final recognition result still needs to be verified by financial personnel, the accuracy and speed of the model must be balanced; the recognition speed should be improved without sacrificing the recognition accuracy. To compress the model and reduce its parameter count, there are several main methods currently available, such as pruning [10], quantization [11], knowledge distillation [12; 13], and so on. Among them, knowledge distillation uses a teacher model to guide the student model to learn specific tasks. As a result, the student model has performance comparable to that of the teacher model but with a significantly reduced parameter count, thereby achieving model compression and acceleration. The method uses the StrucTexT model as the teacher model to guide the student model, StrucTexT_slim, which is essentially mutual supervision between the output and feature maps. The teacher model is needed to load the pre-trained model and fix the parameters. To obtain a softer classification output, the knowledge distillation algorithm adds a temperature coefficient \(T\) to the softmax function in order to soften the category probability of the target, which is calculated as shown:  
\[q_{i}=\frac{exp(z_{i}/T)}{\sum\limits_{j}exp(z_{j}/T)} \tag{7}\]  
where \(z_{i}\) is the content of the output layer. The temperature coefficient \(T\) is taken as 1, which is the original softmax function. As \(T\) decreases, the relative size of the probability of incorrect categories becomes more apparent; as T increases, the probability distribution of the output target category becomes smoother, and the softness of the label increases. Figure 6 shows the algorithm flowchart for knowledge distillation. For the same input vector \(x_{i}\), the predicted output of the teacher model is calculated, divided by the temperature coefficient T, and then transformed via softmax to obtain the soft target, i.e., the softened probability distribution. Afterwards, the predicted output of the student model is calculated, divided by the temperature coefficient T, and transformed via softmax, and cross-entropy loss is calculated using the soft target. The overall loss calculation is as follows:  
\[Loss_{total}=\lambda Loss_{soft}(p_{s},p_{t})+(1-\lambda)Loss_{hard}(p_{s},y) \tag{8}\]  
The overall loss, \(Loss_{total}\), is divided into two parts, \(Loss_{soft}\) and \(Loss_{hard}\), which are calculated as follows:  
\[Loss_{soft}(p_{s},p_{t})=-\sum p_{s}logp_{t} \tag{9}\]  
\[Loss_{hard}(p_{s},y)=-\sum p_{s}logy \tag{10}\]  
where \(p_{s}\) and \(p_{t}\) represent the predicted output of the student model and the teacher model, respectively; \(y\) is the true label and \(\lambda\) is the weighting factor.