# A Comprehensive Analysis of Various Tokenizers for Arabic Large Language Models
## 2 Related Works  
The literature contains some studies comparing the performance of tokenization algorithms across diverse languages on NLP tasks, including Tunisian Arabic [3], the Turkish language [14], and dialect-agnostic Arabic [1], as well as multilingual applications [15; 16]. For example, [PERSON] et al. [3] compared three tokenization algorithms for the Tunisian Arabic dialect on three machine learning models: conditional random fields, support vector machines, and deep neural networks. The authors used several public datasets such as the Tunisian media corpus [17] and the Tunisian Arabic dialect [18] to compare the model's performance based on their ability to accurately tokenize text, thus considering the challenges posed by Tunisian Arabic's morphological complexity. The three models achieved good performance, but the conditional random fields model achieved the best results, with an F measure of 88.9%. In the Turkish language, [PERSON] et al. [14] compared the performance of three tokenization algorithms. The three used tokenization algorithms were Byte-pair Encoding [19], WordPiece [20], and the Unigram Language Model [10] on the Open Super-large Crawled Aggregated coRpus (OSCAR) [21]. The results showed that WordPiece tends to produce more single-word tokens, along with morphology-compatible tokens. Also, the authors found that increasing the size of the training corpus for a tokenizer leads to producing more and shorter tokens. In the Arabic language, [PERSON] et al. [1]introduced three new tokenization algorithms for Arabic and then compared them to the other three baselines. The three tokenizers used in the comparison were SentencePiece, Character, and Word tokenizer. The evaluation was conducted on three supervised classification tasks: sentiment analysis, news classification, and poetry classification. Four datasets were employed: AJGT [22], Metrec [23], LABR [24], and DSAC [25]. The result showed that the word tokenizer achieved the best results for small vocabulary sizes only for news classification. However, for small and larger vocabulary sizes, the character tokenizer achieved the best results for poetry classification. A growing body of research has investigated the impact of different tokenization methods on NLP tasks in multilingual contexts. [PERSON] et al. [15] conducted several experiments using five different tokenizer algorithms across ten language pairs to evaluate their impact on the quality of the final translation. The five tokenizers employed were SentencePiece, Mecab, StanfordWord Segmenter [26], OpenNMT [27], and Moses [28]. The results showed that tokenization significantly affects the quality of the translation, and the selection of the best tokenizer differs based on the selected language pairs. Also, [PERSON] et al. [16] investigated the performance of multilingual models on nine different languages across five different tasks (NER, SA, QA, Dependency Parsing, and Part-of-Speech Tagging), thus comparing them to their equivalent monolingual model. The paper also investigated the impact of the adopted tokenization approach on the performance of multilingual models on monolingual tasks. The results showed that using a tokenizer that is designed for a particular language significantly improves the model's performance on downstream tasks for almost every task and language. Moreover, [PERSON] et al. [29] also performed a comparison between byte-level and character-level tokenizers in vocabulary building for multilingual pretrained language models. Several case studies were conducted on the multilingual XNLI tasks in seven languages such as English, Arabic, French, and Russian using the NEZHA architecture [30]. The authors found that the NEZHA model trained with byte-level subwords consistently outperformed the multilingual BERT model [5]. Subword tokenizers have also been explored for various languages and applications.