# Employing a Multilingual Transformer Model for Segmenting Unpunctuated Arabic Text
## 1 Introduction  
Segmenting unpunctuated textual documents into simple candidate sentences is a vitally important preprocessing task for the success of many natural language processing (NLP) applications. However, this ambiguous segmentation task is intuitively challenging when attempting to tokenize long text that consists of compound sentences but with no proper punctuation (e.g., no obvious clue how to apply sentence-based tokenization when full-stop punctuation is missing). With hard-to-solve NLP applications that require text understanding (such as automatic summarization [1], simplification [2, 3, 4], and question-answering [5]), text segmentation is considered a key preprocessing step for achieving a high-precision performance. Take the example of summarizing a large transcribed audio-to-text document, targeting individuals with low-literacy skills [6] (e.g., non-native readers or children). Here, such an audio document requires complicated or, probably, manual preprocessing to perform the needed tokenization precisely. Lengthy unpunctuated texts are a stumbling block to processing any low-resource languages. The Arabic language is no exception [7] due to its high linguistic complexity and shortage in having parsing and tokenizing resources, including complete lexicons. Specifically, some of the linguistic hurdles related to text splitting have resulted due to theambiguity in identifying (1) word-stemming/lemmatization (as the Arabic language has a significantly rich derivational morphology) and (2) POS-tagging precisely (i.e., in the case the diacritics are missing, which is usually the case) [1]. Approaches to tackling text segmentation problems (i.e., under the low-resource setting) can be distinctly classified into either traditional or transfer-based paradigms. In this context, the traditional paradigm would include approaches that focus on, e.g., constructing hand-crafted rules for text segmentation or extensively designing/training a specific monolingual model. Transfer-based approaches, however, strive to take advantage of languages with rich resources (such as English) in two ways: transferring resources (i.e., cross-lingual transfer learning) or adopting a pre-trained multilingual model for transferring the needed NLP capabilities [8]. Concerning the Arabic language, different text segmentation methods have been put forward, focusing only on the traditional techniques [1; 9; 10; 11; 12; 13; 14; 15], and to our knowledge, no transfer-based method has been proposed yet. In this paper, we demonstrate insights into the capabilities of adopting a multilingual model for Arabic text segmentation. We consider Multilingual BERT (mBERT [[https://github.com/google-research/bert/blob/master/multilingual.md](https://github.com/google-research/bert/blob/master/multilingual.md)]([https://github.com/google-research/bert/blob/master/multilingual.md](https://github.com/google-research/bert/blob/master/multilingual.md)), accessed on 14 October 2022), an extended language understanding model from the original monolingual BERT [16] (Bidirectional Encoder Representations from Transformers [17]). In more detail, mBERT is pre-trained on a tremendous amount of Wikipedia text from the top 104 languages using a Masked-Language Modeling (MLM) objective [18]. It principally focuses on improving different cross-lingual NLP tasks, including the _Mask-fill_ task (i.e., allows predicting masked words according to the context of a given input text without reconstructing the entire text). Its stacked Transformer architecture consists of 12-layer, 768-hidden, and 12-heads, which gives 110 M parameters in total. Although mBERT allows sharing vocabulary and weights parameters across these 104 languages, the training samples are designed based on the monolingual method, and no cross-lingual objectives are considered. However, mBERT generalizes well from the monolingual training data and surprisingly gives state-of-the-art results on cross-lingual benchmarks [18].