# Improving Text Recognition Accuracy for Serbian Legal Documents Using BERT
## 4 Evaluation  
The approach presented in this paper was evaluated using scanned legal document corpora gathered from the Official Gazette of the Republic of Serbia. In order to evaluate the model in different domains, we chose issues that do not contain pure legal provisions, but legal provisions from various specific domains. Therefore, we used 99 scanned pages from six issues published in three different epochs between 1970 and 2002. In order to evaluate this approach for different qualities of scanned documents, we chose two issues from each epoch. The worst scan quality was in issues published in the oldest epoch. Within this epoch, we chose 14 pages from an issue originating from the year 1970 that contained the Law on procedure before the Constitutional Court, the Law on Educational Inspection, the Law on realization of special social interest in the field of communal activities, Amendments to the law on republican administration, and a Rulebook on the technical inspection of vehicles. The second issue from this epoch was published in 1983, and we chose 20 pages that contained text regarding the Resolution on socio-economic development and economic policy. For the second epoch, we chose 35 pages from two issues originating from the year 1990. These issues had a better scan quality and contained text regarding the Law on social activities, Rulebook on reimbursement of costs in court proceedings, Law on public information, Law on cultural heritage, Law on library activity, and Law on cultural financing funds. The best scan quality was for 30 pages selected from two issues published in 2001. The selected pages contained text of the rules of procedure and regulations regarding  
Figure 3: SrBERTa-base training graph—the value of the loss function across 45 epochs. the National Assembly of the Republic of Serbia, excise taxes, tax administration, public enterprises, and children's social care. The evaluation was focused on the reconstruction of words written in the Serbian Cyrillic language with low-level accuracy returned by the OCR engine and merged words. Within this evaluation, the system was not tested for the reconstruction of enumeration, numbers, and words written in all capital letters. For each test page, the test examples for evaluation were created based on the output received from the OCR engine. Consequently, we selected 14,604 test examples for all three epochs that were verified manually. The distribution of the number of examples by epoch is shown in Table 3.  
It can be noticed that all three epochs had a similar total number of words. However, the oldest epoch had the most test cases due to it having the worst scan quality. Furthermore, it was noticed that usually, multiple words needed to be verified within one paragraph. Moreover, the lengths of the paragraphs were different, and there were many cases of contexts that contained only a few words. One example of a scanned paragraph with multiple words that needed verification from an issue published in 1990 is presented in Figure 4. In this paragraph, all words framed in green had lower accuracy, and words underlined in orange had to be merged and verified because of the merge. The context returned by OCR for this paragraph is presented in Figure 5. It can be noticed that only one word was wrongly recognized by the OCR, with recognition of the word _suprane_ instead of _suprane_. In this paragraph, the model accurately predicted values for seven out of eight tests, giving the incorrect value for the first word in the paragraph. For the purpose of this evaluation, we defined an accuracy measure that was used for manual verification. The accuracy measure considered a test case to be successful only if the output was the same word that matched the expected value in gender, number, and case. Based on this accuracy measure, the success rate was calculated for all test pages.