# Employing a Multilingual Transformer Model for Segmenting Unpunctuated Arabic Text
## 3 PDTS: Punctuation Detector for Text Segmentation
We have categorized these 16,281 text examples into short, mid (i.e., number of words between 168 and 512), and long based on their length, see their statistical descriptions in Table 3.

#### 4.1.3 Baselines

We compared the pre-trained mBERT model (i.e., used as a base model for constructing PDTS) against the state-of-the-art Arabic monolingual, as well as multilingual MLM models. ArBERT [19], AraBERT [20], and CAMeLBERT [21] are Arabic-specific transformer-based \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Document Size** & **Documents** & **Sentences** & **Words** \\ \hline Small & 5427 & 6639 & 328,872 \\ Medium & 5427 & 6807 & 491,693 \\ Large & 5427 & 13,404 & 1,057,131 \\ Total & 16,281 & 26,850 & 1,877,696 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Statistical descriptions of text examples used for evaluation. Figure 2: Description of our quantitative evaluation protocol. language models, trained with the same architecture as BERT\({}_{Base}\) configuration but on different datasets. ArBERT is trained on 61 GB of MSA text, while AraBERT is trained on around 23 GB of Arabic text (containing 70 M sentences and 3B words). CAMeLBERT is trained (The model is trained for 1 m epochs which took approximately 4.5 days.) on the largest Arabic dataset, containing different dialectal corpora with a size of about 107 GB. XLM-RoBERTa [33], however, is a state-of-the-art multilingual version of RoBERTa, which is pre-trained on 2.5 TB of filtered CommonCrawl data that consists of 100 languages, including Arabic. #### 4.1.4 Implementation Details

All base models (i.e., mBERT and other pre-trained baselines) are publicly available at the Hugging Face ([[https://huggingface.co/](https://huggingface.co/)]([https://huggingface.co/](https://huggingface.co/)), accessed on 14 October 2022), under the model names: _bert-base-multilingual-uncased_, _UBC-NLP/ARBERT_, _'uabmindlab/bert-base-arabertv02'_, _'bert-base-arabict-camelbert-mix'_, _'xlm-roberta-large'_. We have used the PyTorch ([[https://pytorch.org](https://pytorch.org)]([https://pytorch.org](https://pytorch.org)), accessed on 14 October 2022) framework to construct them and utilized Python-based NLP toolkits for text preprocessing, including NLTK ([[https://www.nltk.org/](https://www.nltk.org/)]([https://www.nltk.org/](https://www.nltk.org/)), accessed on 14 October 2022) and CAMeL ([[https://camel-tools.readthedocs.io/](https://camel-tools.readthedocs.io/)]([https://camel-tools.readthedocs.io/](https://camel-tools.readthedocs.io/)), accessed on 14 October 2022). For a more detailed description of how to use these models, we recommend that researchers refer to Hugging Face Documentations ([[https://huggingface.co/docs](https://huggingface.co/docs)]([https://huggingface.co/docs](https://huggingface.co/docs)), accessed on 14 October 2022). For computation, we used a PC machine equipped with Intel i9-CPU, 64G-RAM, and a single NVIDIA GeForce RTX3070 GPU. ### Performance Evaluation

The performance of PDTS concerning different document sizes is illustrated in Table 4. The overall prediction score looks generally high but relatively resembling between small vs. large document categories. In addition to the lower scores observed with the medium document category, the obtained scores indicate that document size as a factor is independent of performance quality. This means that document size can affect only computational performance. Further, it should be pointed out that the _Accuracy_ score presents the average height score of 0.89 (see Table 4), which results from having a high \(TN\)'s rate, not \(TP\)'s rate. As mentioned, we introduced the parameter \(\alpha\) to reduce computation costs when processing large document sizes. Thus, we conducted several experiments to analyze the sensitivity of \(\alpha\) with respect to both the quality as well as the computational performance, reported in Table 5 and visualized in Figure 3.

Table 5 illustrates the performance of PDTS on six runs, each _with_ and _without_ using our linguistic rules (discussed in Table 2).