## 3 Tokenizer Performance--Models, Dataset, and Performance Measure Insights
### State-of-the-Art Multilingual, Slavic or Serbian-Specific Models  
Developed primarily to understand the formal language found in Serbian legislation, SrBERTa v1 [18] is a language model built on RoBERTa that can be used for different language modeling applications. Initially, the OSCAR dataset, which included 645,747 texts or roughly 150 million words (with a total size of more than 2 GB when stored), was used to train the model and its tokenizer. A vocabulary of 30,522 tokens was produced by the model using the ByteLevelBPETokenizer [53] tokenization technique and its implementation from the HuggingFace library [54]. The second iteration of this model was developed to improve mask language modeling capabilities. Two distinct datasets, the OSCAR dataset and a dataset devoted exclusively to legal texts, were utilized to train SrBERTa v2 [55]. The OSCAR 23.01 dataset, which required 7.7 GB of storage space and had 1,677,896 documents with a total word count of 632,781,822, was used for this training. A total of 1.6 GB of legal documents from the Republic of Serbia's Legal Information System made up the second dataset. Special attention was given to tokenizer development, in terms of the structure of the dataset used for tokenizer training. The RoBERTa network and the tokenisation methods were implemented as ready-made architectures by the HuggingFace community. All these implementations are variations of the transformers library, version 4.17.0. The tokenizer for SrBERTa v2 was created using an equal mixture of legislative texts and texts taken from the OSCAR 23.01 dataset to ensure that more legal phrases are processed as entire words rather than subwords. The adaptation strategy we have chosen belongs to a group of explicit data selection methods, which may introduce domain-specific biases. Therefore, our future work envisions research regarding the optimal balance for data selection. Further, each legislative text was preprocessed to minimize data loss due to truncation, with the maximum input sequence size set to 512 tokens. This meant that all texts were divided into smaller units of 512 tokens and then concatenated using the newline character. This preprocessing ensured the generation of inputs that can be optimally utilizedby the network. SrBERTa v2 tokenizer has a vocabulary containing 50,265 tokens, it used a minimum frequency of 2 and the following special tokens: \"
\", \"
\", \"<pad>\", \"<unk>\", and \"<mask>\". Using an NVIDIA Quadro RTX 5000 GPU, along with AdamW optimiser, the SrBERTa tokenizer and network training took place across 45 epochs and 31 days in total. Eight billion tokens of crawled text from the Croatian, Bosnian, Serbian, and Montenegrin web domains were used to pre-train the transformer model BERTic [16]. The tasks of named entity identification, part-of-speech tagging, geolocation prediction, and commonsense causal reasoning were used to assess this model. The authors have decided to train transformer models using the Electra technique [56]. Further, the authors created a WordPiece vocabulary with a 32,000 token vocabulary size, much like BERT. The HuggingFace tokenizers library was used to train a WordPiece model on a random sample of 10 million paragraphs from the entire dataset. Jerteh-81 [17] is one of the best and most widely used large language models for the Serbian language. It is based on RoBERTa-base architecture with 81 million parameters. Jerteh-81 was trained on publicly available datasets which, all together, contain over 4 billion tokens. When published, Jerteh-81 demonstrated high-quality results on semantic text similarity tasks. Jerteh-81 vocabulary contains 50,265 tokens. XLM-RoBERTa [15] is one of the first models to prove that pretraining multilingual language models at scale leads to significant performance gains. This model is a transformer-based masked language model trained on one hundred languages, using more than two terabytes of filtered CommonCrawl data.