# Employing a Multilingual Transformer Model for Segmenting Unpunctuated Arabic Text
## 3 PDTS: Punctuation Detector for Text Segmentation
Through those achieved scores, while paying more attention to the highest _f-measure_ (i.e., \(FM=0.75\)), we can observe that PDTS performs well and demonstrates the best results at \(\alpha=40\). The practical feasibility of applying our linguistic rules is evidently anticipated by an average of around 16% improvement (see the _acc_, _recall_ and _f-measure_ with using vs. without using rules). Moreover, while it is obvious that the execution time would increase by increasing \(\alpha\), this notable improvement (i.e., anticipated to be achieved when applying our rules) is worth sacrificing a few seconds \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Accuracy** & **Precision** & **Recall** & **F-Measure** \\ \hline small & 0.86 & 0.74 & 0.87 & 0.79 \\ medium & 0.87 & 0.67 & 0.67 & 0.65 \\ large & 0.95 & 0.78 & 0.92 & 0.82 \\ \hline avg. & 0.89 & 0.73 & 0.82 & 0.75 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Performance comparison of PDTS with respect to different document sizes: small (\(|X|<168\)), medium (\(168\leq|X|\leq 512\)), and large (\(512<|X|\)). The base model is \(mBERT\), and \(\alpha=40\). For \(\theta\), the best value found in all our experiments is around 0.4.

of performance. This is indicated by giving insight into the computational overhead (see execution time columns). In our experiments, the maximum computational overhead has reached 35 s at \(\alpha=160\) for processing 16,281 documents. In Table 6, we compare different pre-trained transformer-based models (i.e., alternative options rather than mBERT for basing our PDTS). Surprisingly, our considered mBERT outperforms all the state-of-the-art competitor models in the _f-measure_ score, including monolingual Arabic models, for processing Arabic texts. Moreover, while ArbERT [19] is trained on the least amount of data compared to AraBERT [20] and CAMeLBERT [21], it gives next to the best _f-measure_. Concerning the execution time, CAMeLBERT [21] gives the best performance but performs poorly in _f-measure_. ### Human Evaluation

To further evaluate the performance of PDTS, we conducted a qualitative investigation by eliciting human assessments with 54 text documents. These documents were randomly sampled from the 16,281 documents, where the punctuations were added by PDTS using mBERT besides the other competitor base models. We asked three expert consultants in Arabic linguistics (not authors of this paper) to assess these documents on three criteria using a five-point Likert scale (1-5):

* Adequacy (preservation of the source meaning);
* Contextual soundness (in terms of the avoidance of error cases, including (1) the generation of unimportant punctuations and/or (2) the failure in not generating important punctuations);
* Grammaticality (the correctness of the generated punctuations). Each expert is given 18 sample documents to be compared with their three original documents (denoted as gold-standard references). The average number of words in each sample is around 115 words. The experimental results, reported in Table 7, confirm that our PDTS (indicated by \(\ddagger\)) produces the best average ratings of about 3.72 and outperforms all competitor bases models in _contextual soundness_ criteria. We observe that employing ArBERT as an alternative-based model to mBERT is quite convenient, as ArBERT achieves the highest _Adequacy_ and _Grammaticality_ ratings (see 4.17 and 3.67 in the first row of Table 7). This observation is also indicated by the insignificant \(p\)-values obtained with ArBERT (and also AraBERT) against mBERT (including linguistic rules). Apart from the strengths and weaknesses of these base models, this experiment revealed that fine-tuning a comprehensive pre-trained multilingual model (such as mBERT) for the needed Arabic tasks (i.e., in such a low-resource language) is more straightforward and could produce significantly better performance than employing existing monolingual Arabic BERT-style models (particularly, CAMeLBERT that has been trained on the largest Arabic dataset).