# Employing a Multilingual Transformer Model for Segmenting Unpunctuated Arabic Text
## 3 PDTS: Punctuation Detector for Text Segmentation
### A Greedy Strategy for Selecting the Best Subset Punctuations

Up to line 15 of Algorithm 1, the discussion concerning the predicted masked tokens has been relatively abstract. Now we introduce our greedy strategy for selecting the best subset punctuations from \(\mathit{PunTokens}\) (implemented in Algorithm 2 and presented in Figure 1D), such that \(\mathit{Pun}^{best}\in\mathit{PunTokens}\). This strategy aims to iteratively validate/accept \(pu_{i}^{m}\) according to Equation (3) besides our defined linguistic rules, starting in each iteration with \(pu_{i}^{m}\) that has the highest accuracy score (i.e., \(pu_{i}^{m}\leftarrow\mathit{getPredWithMaxScore}(\mathit{PunTokens})\)). Given two text segments \(SB\) and \(SA\) (i.e., text segment before and after the punctuation \(pu_{i}^{m}\), see lines 6-10 of Algorithm 2), extracted at a particular iteration, our linguistic rules validate if \(SB\) and/or \(SA\) contains simple sentences to be separated. More generally, regarding the Arabic language grammar, simple Arabic sentences are linguistically categorized into either: nominal-sentence (noun+noun) or verbal-sentence (typically, a verb followed by a noun); each is formulated with a minimum of two words. Depending on the parts-of-speech of the first word, the type of sentence can be determined [31]. In this context, and without delving into the complex subcategories of Arabic sentences, we apply four straightforward rules to validate if \(SB\) and/or \(SA\) include at least one simple sentence. Let \(s\) be the text segment that we want to validate, we define our linguistic rules in Table 2, such that \(R1\) checks the minimum number of words allowed in a simple sentence; \(R2\) and \(R3\) check whether \(s\) is a nominal or a verbal sentence; and \(R4\) is a special case rule applicable when full-stop punctuation precedes \(s\). As explained, the analysis of the collected punctuations in _PunTokens_ (i.e., obtained in the first step by mBERT) performs in a greedy-like strategy, which iteratively selects a \(pu_{i}^{m}\) with the highest prediction score. In principle, this subset selection problem is known to be an NP-hard problem [32] (i.e., it cannot be solved in an exact polynomial time). However, a greedy-like strategy often provides a well-approximated solution to the target optimization problem and is practically implemented in many applications, including, e.g., sparse learning and feature data mining selection. In our problem, the size of the best subset selection \(|Pun^{best}|\) is unknown, and the search for selection terminates once no \(pu_{i}^{m}\) is satisfying the rules presented in Table 2. Formally, our PDTS attempts to optimize the subset selected punctuations (i.e., \(Pun^{best}\)) as expressed in Equation (4):

\[ \begin{split}\underset{Pun^{best}}{\text{maximize}}& f(Pun^{ selected})\\ \text{subject to}& Pun^{selected}\in PunTokens.\end{split}  \tag{4}\]

where \(f\) represents our objective function (described in Algorithm 2 lines 3-14) that aims to maximize the number of splits in \(x\) by adding the best punctuations from _PunTokens_. ## 4 Experiments and Discussion

The ultimate overarching purpose behind our proposed PDTS is to tokenize long-given texts into simple candidate sentences, employing two procedures: (1) identifying punctuation-split delimiters and (2) validating them using our suggested linguistic rules. Thus, to explore the overall effectiveness of PDTS, we conducted several experiments to evaluate these procedures, elaborated as follows. First, we measured the overall performance of PDTS concerning different input sizes (\(|X|\)) along with different settings for \(\alpha\). Here, we shed light on the comparison with the closely related state-of-the-art pre-trained base models. Second, we studied the humanistic point of view in eliciting and evaluating the performance of PDTS (i.e., by focusing on both aspects: linguistic sentence rules and punctuation prediction depending on the MLM method).