# Improving Text Recognition Accuracy for Serbian Legal Documents Using BERT
## 3 A BERT-Augmented Text Recognition Approach
### SrBERTa v2--BERT-Based Language Model for Serbian Legal Texts Written in Cyrillic
The configuration of the newly trained SrBERTa tokenizer is shown in Table 1.  
The following example shows the output obtained when we apply the improved version of the SrBERTa tokenizer to a random input sample, selected from the legal corpus \begin{table}
\begin{tabular}{c c c} \hline \hline
**Vocabulary Size** & **Minimum Frequency** & **Special Tokens** \\ \hline
50,256 & 2 & \(<\)s\(>\)\(<\)/s\(>\)\(<\)pad\(>\)\(<\)unk\(>\)\(<\)mask\(>\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: SrBERTa tokenizer configuration. generated out of publicly available legal texts from the Legal Information System of the Republic of Serbia:  
Example input (in Serbian): \"Llamom pravallisia rezulismo je da prisbial tekhnike kontrolole pole,nogo dela projekta za radennikov doznolu toptrybe isernnoost tot dela projekta tako tipo, na pocohoj stranguli due lijanu ys narolche polataika o prishom plinu, odnosio predektennikov koiru ie znymbo tekhnikh kontrolov\". Example input translated into English: \"The article of the rulebook stipulates that the person performing the technical control of a particular part of the project for the building permit confirms the correctness of that part of the project by making a statement on a separate page with information about the legal entity, that is, the entrepreneur who performed the technical control\". The main methods of any tokenizer are tokenizing (splitting strings in sub-word token strings), converting token strings to ids and back, encoding/decoding (tokenizing and converting to integers), adding new tokens to the vocabulary in a way that is independent of the underlying structure, and managing special tokens. The output of a tokenization process is a tensor of IDs belonging to the tokenizer vocabulary generated for a given input sentence. Let us observe the generated tokenized output for the previously shown input sample, shown using two Pytorch tensors: one for the input IDs and the other for the attention masks. The first tensor of the input stores corresponding token IDs from the dictionary and the second stores a mask, which is used by the network to avoid padding token indices. 'input_ids': tensor([[0, 5429, 304, 6295, 14584, 313, 330, 37782, 6904, 3942, 35052, 1190, 3569, 350, 12057, 6564, 7764, 30902, 1883, 1190, 3569, 1068, 6  
maps previously generated contextualized embeddings to the size of the vocabulary, while simultaneously, it generates a score for each possible token in the vocabulary for each position in the input sequence. Another crucial aspect was configuring and setting the training parameters for the new version of the SrBERTa network. During this process, we compared the previously used hyperparameters with the recommended values from the original RoBERTa paper, selecting valid value ranges that aligned with our requirements and hardware capabilities. Notably, for this development phase, we utilized enhanced GPU hardware in comparison to the previous SrBERTa version, specifically the NVIDIA QUADRO RTX 5000, which allowed us to employ a network size twice as large, corresponding to the RoBERTa-base architecture. We continued to use the AdamW optimizer to update the network weights, due to its improved weight decay methods, with the weight decay parameter set to 0.01. Additionally, the mini-batch size was set to 8 because of hardware constraints, but we effectively increased it to 64 using the gradient accumulation method. Table 2 shows the most important configuration parameters of the SrBERTa v2 model in comparison to the previous version. Additionally, during the training phase, we adopted a similar approach for setting the learning rate to that described in the RoBERTa paper. Specifically, the learning rate gradually increased over the first 600 steps to a peak value of \(1\times 10^{-4}\), after which it was linearly decayed.