# Employing a Multilingual Transformer Model for Segmenting Unpunctuated Arabic Text
## 3 PDTS: Punctuation Detector for Text Segmentation
Nevertheless, not all existing multilingual models are suitable for Arabic as unexpectedly, the worst performance observed was with XLM-RoBERTa [33]. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Base Model** & **Language** & **F-Measure** & **ET (Sec)** \\ \hline ArBERT [19] & Monolingual (Arabic) & 0.6235 & 1286.81 \\ AraBERT [20] & Monolingual (Arabic) & 0.3500 & 1173.98 \\ CAMeLBERT [21] & Monolingual (Arabic) & 0.2121 & 986.17 * \\ XLM-RoBERTa [33] & Multilingual & 0.3035 & 4916.51 \\ \hline mBERT [16] & Multilingual & 0.7523 * & 1986.91 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Performance comparison of PDTS concerning the closely related state-of-the-art pre-trained base models (the asterisk * indicates the best performance), under the settings (\(\alpha=40\) and \(\theta=0.4\)). Here, \(\alpha\) is set according to the best _f-measure_ score found; see Figure 3. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Base Model** & **A** & **C** & **G** & **Avg.** & \(p\)**-Value (\(t\)-Value)** \\ \hline ArBERT [19] & 4.17 * & 3.17 & 3.67 * & 3.67 & 0.925156 (0.11) \\ AraBERT [20] & 3.33 & 2.50 & 3.17 & 3.00 & 0.069934 (2.46) \\ CAMeLBERT [21] & 2.67 & 1.50 & 2.17 & 2.11 & 0.002153 (7.03) \(\ddagger\) \\ XLM-RoBERTa [33] & 1.67 & 1.83 & 1.67 & 1.72 & 0.003126 (6.36) \(\ddagger\) \\ \hline mBERT [16] & 3.83 & 3.33 & 3.30 & 3.50 & 0.468527 (0.81) \\ mBERT [16] + Linguistic rules & 3.83 & 3.83 * & 3.50 & 3.72 * & \(\ddagger\) \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Human evaluation results for the three criteria: **Adequacy**, **C**ontextual soundness, and **G**rammaticality. Base pre-trained models with \(\ddagger\) are significantly different from PDTS’s base model \(\ddagger\), depending on a two-tailed independent student \(t\)-test, at \(p<0.05\). The asterisk * indicates the best result. ## 5 Conclusions and Future Work

This paper investigated whether fine-tuning a pre-trained language model for the _Fill-Mask_ task can be beneficially applied in punctuation detection approaches. We proposed PDTS, a punctuation detector for text splitting, built on top of a multilingual BERT-based model, four generic linguistic rules, and a greedy algorithm for the best subset punctuation selection. We have showcased how PDTS is employed as a text tokenizer for unpunctuated documents, which can be highly useful to text simplification approaches that concentrate on handling large generated audio-to-text documents. Depending on our considered standard quantitative metrics and qualitative human-based evaluation protocols, experimental findings across two well-written Arabic corpora demonstrated that PDTS is practically effective in both performance quality and computational cost. For future work, we are striving to expand our PDTS toward supporting automatic text simplification and summarization problems. Conceptualization, S.A., A.B.A., A.N. and E.N. ; Formal analysis, A.M.A. ; Methodology, A.M.A. ; Writing--review and editing, A.M.A. All authors have read and agreed to the published version of the manuscript. This research was funded by the Deputyship for Research & Innovation at the Ministry of Education in the Kingdom of Saudi Arabia, under Grant IU-Ministry Project (19/20). The experimental corpora and implemented code are publicly available for replicating our experiments ([[https://github.com/[PERSON]/ArSummarizer](https://github.com/[PERSON]/ArSummarizer)]([https://github.com/[PERSON]/ArSummarizer](https://github.com/[PERSON]/ArSummarizer)), accessed on 14 October 2022). The authors declare no conflict of interest. ## References

* _[PERSON] et al. (2021)_ [PERSON]; [PERSON] [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] Leveraging DistilBERT for Summarizing Arabic Text: An Extractive Dual-Stage Approach. _IEEE Access_**2021**, \(9\), 135594-135607. [CrossRef]
* _[PERSON] et al. (2020)_ [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] MUSS: Multilingual unsupervised sentence simplification by mining paraphrases. _arXiv_**2020**, arXiv:2005.00352. * _[PERSON] et al.