# Improving Text Recognition Accuracy for Serbian Legal Documents Using BERT
## 3 A BERT-Augmented Text Recognition Approach
### SrBERTa v2--BERT-Based Language Model for Serbian Legal Texts Written in Cyrillic  
As part of this research, we present a follow-up study on the SrBERTa model [21]. The primary goal of SrBERTa was to develop a model capable of understanding the formal language used in Serbian legislation. Initially, SrBERTa was trained to comprehend the Serbian language and then fine-tuned for masked language modeling using Cyrillic legal texts from a specially created dataset [32]. Masked language modeling is a downstream task which predicts a masked token in a sequence, and the model performing the task can attend to tokens bidirectionally. This task predicts only the masked words rather than reconstructing the entire input [16]. Figure 2: Diagram for autocorrection mechanism. In this second phase of model development, our objective was to enhance the previously achieved results by using the following approaches:  
* Creating an expanded dataset of legal texts;
* Training a more extensive tokenizer for the Serbian language, optimized for better tokenization of legislative terms;
* Training a larger model on the task of masked language modeling using an increased amount of training data, including a new, legal dataset. For the purpose of training the newer version of the SrBERTa network, we utilized two new datasets. The primary dataset selected for the training phase was the OSCAR [33] dataset, which represents a comprehensive collection of open data, generated through linguistic classification from the Common Crawl corpus [34]. Additionally, we utilized a more recent version of OSCAR, specifically OSCAR 23.01, which includes a greater volume of Serbian language data, comprising 1,677,896 documents and 632,781,822 words and occupying 7.7 GB of memory. The second dataset used within this research was created specifically for this purpose and comprised only legal texts. These texts were gathered from the Legal Information System of the Republic of Serbia, resulting in a total of 1.6 GB of data. Each legal text underwent preprocessing in order to minimize data loss due to truncation, with the maximum input sequence size set to 512 tokens. Consequently, all texts were split into smaller units of 512 tokens and then concatenated using the newline character. This preprocessing ensured the generation of inputs that could be optimally utilized by the network. For the tokenizer training phase, we utilized the Byte-Pair Encoding (BPE) tokenization algorithm provided by the HuggingFace library [35]. Our goal was to train a network specialized in modeling not only the Serbian language but also Serbian legislative language. Therefore, we adopted a novel training approach in comparison to the previous version of the SrBERTa network. Initially, we used a tokenizer trained only on texts collected from the Internet (OSCAR dataset). However, this approach had limitations in tokenizing domain-specific terms found in legal texts. Consequently, during the masked token prediction process, the proposed words often did not represent the best possible options. This issue arose because the network, despite being fine-tuned on legal texts, did not tokenize many legal terms as complete words but rather as sub-words, due to the fact that the tokenizer was not trained using legal texts. To address the previously described issue, we devised a new training approach for the tokenizer. Now, during the tokenizer training phase, we select an equal number of legal texts and texts from the Internet. This strategy aims to create a tokenizer capable of effectively handling both standard spoken language and the specialized terminology used in legal texts. Moreover, in the new version of SrBERTa, we incorporate a larger vocabulary, consisting of 50,256 tokens. In the end, we prove that the network trained with this kind of domain-adapted vocabulary is able to encode more legal terms as whole words and effectively utilize them in masked language modeling tasks.