# Improving Text Recognition Accuracy for Serbian Legal Documents Using BERT
## 3 A BERT-Augmented Text Recognition Approach  
Building precise and efficient models in any field requires high-quality text corpora. As mentioned in the Introduction of this paper, the Serbian language is low-resourced in the field of digital infrastructure and digitized language resources, especially in Serbian legal heritage documents. Therefore, within this research, we present LexBERT, an approach for generating high-quality legal text corpora by improving text recognition accuracy for Serbian legal heritage documents. LexBERT is designed to recognize text from scanned legal documents written in the Serbian Cyrillic language and return high-quality text corpora. The whole system, presented in Figure 1, combines three main components:  
* Tesseract OCR--optical character recognition engine;
* SrBERTa v2--BERT-based language model for Serbian legal texts;
* Word-level similarity measure. The LexBERT system receives an image of the document as input in the system and extracts the text using the OCR engine. Within our approach, we rely on the Tesseract OCR engine for the Serbian Cyrillic language for extracting the text from an image. The OCR engine returns all the recognized words coupled with information about the word's position in the image, position in the block, paragraph, and line, and its confidence level given in a percentage. Based on the obtained results and information about the block, paragraph, and line, the system reconstructs the text from the image into paragraphs and executes the merging of words that have been split across new lines due to their length. Since OCR accuracy may vary, the core of the LexBERT system is implemented as an autocorrelation mechanism. The diagram for this autocorrelation mechanism is presented in Figure 2. While reconstructing the text, for each of the recognized words, the system analyzes the confidence level and evaluates whether it is necessary to apply the autocorrelation mechanism for that word. The LexBERT system is set to verify and autocorrect each word with an OCR confidence level lower than 90% and each split word that was merged by the system. The threshold of 90% was chosen after a manual check of the OCR output for several pages, during which it was noticed that for words with confidence levels lower than 90%, OCR tended to make errors. This mechanism uses a masked language modeling approach to create suggestions for a particular place in a paragraph. The system uses the developed SrBERTa v2, a BERT-based model for Serbian legal texts, explained in detail in the next section of this paper. For each word that needs to be verified, the system acquires the top 20 suggestions for that word by passing the context that contains the paragraph where that word is masked into the SrBERTa v2 model. Although more than one word in a paragraph can be a candidate for verification, the suggestions are acquired for each word separately. This way, the system obtains a list of the top 20 possible mask values ranked according to score, for the verification and autocorrection of each word. After receiving suggestions from the model, the system calculates the similarities between suggested tokens and the word recognized by OCR and proposes the final value for that position in the paragraph. Since our approach prioritizes words' content and tries to find an exact match for the one in the text, our goal is to determine the suggested token that is most similar to the OCR output with or without the expected punctuation marks. Figure 1: LexBERT architecture—a BERT-augmented text recognition approach. Therefore, the system relies on the Levenshtein distance for similarity calculation between terms and uses it to compare tokens with original OCR output and OCR output trimmed from all non-alphanumeric characters at the beginning and at the end. Furthermore, before the Levenshtein distance is calculated with two combinations of OCR output, each token is trimmed from space characters at the beginning and at the end.