# Employing a Multilingual Transformer Model for Segmenting Unpunctuated Arabic Text
## 2 Related Literature  
Text segmentation is rooted as a preprocessing phase in many needed solutions that focuses on philosophical understandings of natural languages. As a consequence, one can review various text segmentation approaches with different objectives within NLP-literary circles (see the pre-2018 approaches surveyed in [22] and, more recently, in [23]). Furthermore, existing approaches endeavor to split a given text into subtests while tackling different levels of language complexities. These levels range from identifying morphological and lexical cohesion boundaries within word-to-small text segments [1; 4; 13] (e.g., words, phrases, and simple discourse (e.g., Elementary Discourse Units (EDUs)) sentences) to higher-level syntactic and thematic structures [24; 25; 26] (such as paragraphs, passages, chapters, or textual documents containing many chapters). In terms of the underlying technique used, existing text segmentation approaches also vary according to their chronological development, which can be classified into three genres:  
* The early discourse segmentation approaches (some approaches date back to the previous decade), which often depend on extracting linguistic annotation features, including treebank structure and POS tags [4; 9; 10; 24];
* The statistical and machine learning methods, which often manipulate text similarity metrics and distances; and recently [12; 13; 14; 26];
* The natural language understanding (NLU) approaches depending on artificial neural networks, particularly those adopted transformer-based models (e.g., [27]). In this paper, we follow the third genre for generating subtests that potentially represent independent clauses by surrounding them with proper punctuation predicted. To date, tremendous NLP work has made a productive contribution to only a few Indo-European languages (mainly, the English language, which is an extremely high-resource). In contrast, work on low-resource languages, such as Arabic, is still rudimentary. Unfortunately, a handful of related text segmentation approaches are suggested for Arabic [1; 9; 10; 11; 12; 13; 14; 15]. Therefore, in the remaining of this section, we limit the discussion by reviewing these related approaches only. Word-level segmentation solutions are presented wider in the literature (e.g., [10; 13; 14]) than the other higher and linguistically more complex levels (such as phrases/sentences and (in)dependent clauses, e.g., [1; 9; 11; 12; 15]). In the former segmentation, ref. [10] suggested an improved clitic segmentation model (at a character-based level) targeting formal and informal dialect Egyptian text. In particular, the authors extend the labeled features of an existing linear-chain Markov model and then retrain it on some annotated Egyptian corpus. Nonetheless, the proposed extension does not seem to generalize well to other Arabic dialects. In a similar line with [10], ref. [13] proposed (Farasa, which means _insight_ in Arabic) a more robust character-level segmentation model (implemented using a linear SVM classifier) for anticipating segmentations of a word. However, it depends heavily on extracted lexicions and morphological features. As yet, Farasa is one of the state-of-the-art word-level segmentation methods, which outperforms other well-known similar approaches, including Madamira [28] and Stanford-Arabic (Stanford-Arabic NLP [[https://nlp.stanford.edu/projects/arabic.shtml](https://nlp.stanford.edu/projects/arabic.shtml)]([https://nlp.stanford.edu/projects/arabic.shtml](https://nlp.stanford.edu/projects/arabic.shtml)), accessed on 14 October 2022). A follow-up attempt to improve existing word-level segmentation approaches, depending on SVM and Bi-directional LSTM machine learning models, is suggested by [14]. Their models are trained on authors-created corpora containing naturally occurring text from social media. Although [14] reports an improved performance compared to [13], ref. [13] seems much lighter and more straightforward to adopt.