# Employing a Multilingual Transformer Model for Segmenting Unpunctuated Arabic Text
## 3 PDTS: Punctuation Detector for Text Segmentation
for example:\(\mathit{Pun}[]\leftarrow\{\mathit{colon}\;,\mathit{question}-\mathit{mark}\;,\;\mathit{exclamation}-\mathit{mark}\;,\;\mathit{full}-\mathit{stop}\;,\;\mathit{comma}\;\}\)
2 PunTokens \(\leftarrow\emptyset\)
3 iBeginIndex = 0 \(\mathit{wx}\leftarrow\texttt{wordTokenizer}(X)\) if\(\alpha>|X|\)then\(\alpha\leftarrow|X|\) if\(\alpha>512\)then\(\alpha\leftarrow512\) // This is limited by some BERT-based models. for\(i\gets 0\)to\(|\mathit{wx}|\)do if\(\alpha/2\leq i\leq(|\mathit{wx}|-\alpha)\)then\(iBeginIndex\,+=1\) \(Fragment^{i}\leftarrow\texttt{subStringFromTokens}\) (\(iBeginIndex\), \(\alpha\)) \(Fragment^{i}\leftarrow\texttt{addMaskTokenAtIndex}\) (\(Fragment^{i}\), \(i\)) \(Pred\leftarrow\mathit{MLM.FillMask}\) (\(Fragment^{i}\)) // In PDTS, we use \(mBERT\) if\(pred.Vocab\in\mathit{Pun}\)and\(\theta\leq pred.Score\)then \(\mathit{PunTokens}.\texttt{append}(i,\mathit{pred.Vocab},\mathit{pred.Score})\) return\(X\)and\(\mathit{PunTokens}\)
```

**Algorithm 1**The proposed procedure for predicting punctuations in an unpunctuated document using an MLM model. Figure 1: Overview of our proposed PDTS that detects missing punctuations in a complex Arabic text \(X\) with the purpose of chunking \(X\) into concatenated simple sentences \(Y\). The translation of the mentioned Arabic text \(X\) is _Smile is the right word without letters. It enriches the one who receives but does not diminish the one who gives. It does not last more than a jiffy to behold, but its memory remains long. No one can do without it, and no one does not own it. It is the signing of the charter of respect between friends, a relief from the tiredness, and a ray of hope for the miserable. It is the most beautiful consolation for the deleted. It is neither purchasable, borovable, nor lodable. It’s the smile. ```
Function:\(Y\leftarrow\texttt{PDTS}(X,\textit{PunTokens})\) input :\(X\) and \(\textit{PunTokens}\) are the return from Algorithm 1 output :\(Y\) the text in \(X\) with the best detected punctuations added (i.e., \(\textit{Punselected}\in\textit{PunTokens}\)). 1\(Y\gets X\)
2\(\textit{Punselected}[]\leftarrow\emptyset\)
3 repeat
4\(\textit{maxPred}\leftarrow\texttt{getPredWithMaxScore}\) (\(\textit{PunTokens}\))
5// Recursively, we extract the text segments before (\(SB\)) and after (\(SA\)) the predicted maxPred.Vocab. These surrounding segments are determined based on the addent punctuations to \(Y\) at each iteration
6\(SB\), \(SA\leftarrow\texttt{getSurroundingTextSegments}\) (\(Y,\textit{maxPred}\))
7\(SB^{v}\leftarrow[R1(SB)\) and (\(R2(SB)\) or \(R3(SB)\))]
8\(SA^{v}\leftarrow[R1(SA)\) and \((R2(SA)\) or \(R3(SA)))\) or \(R4(SA,\textit{maxPred.Vocab})]\)
9// Rules (R1-R4) are presented in Table 2.
10 if\(SB^{v}==SA^{v}==\textit{true}\)then
11\(Y\leftarrow\texttt{addPunctuation}\) (\(Y\), \(\textit{maxPred.index}\), \(\textit{maxPred.Vocab}\))
12\(\textit{Punselected}\).append (\(\textit{maxPred}\))
13\(\textit{PunTokens}.remove\) (\(\textit{maxPred}\))
14 untilPunTokens\(==\emptyset\)
15// Return \(Y\) that contains \(X\) plus the added punctuations. 16// Here, we can straightforwardly tokenize \(Y\) based on \(\textit{Punselected}\) as split delimiters (i.e., \(Y.split(\textit{Punselected})\)). 17 return\(Y\)
```

**Algorithm 2**The proposed procedure for validating the predicted punctuations in Algorithm 1 and selecting the best ones to be added in \(X\). These added punctuations can be then used as split delimiters for segmenting \(X\). ### mBERT: Multilingual Masked Language Model

Instead of extensively training a large language understanding model on a specific non-English language (such as ArBERT [19], AraBERT [20], and CAMeLBERT [21] for the Arabic language), a more straightforward alternative is to fine-tune a comprehensive pre-trained multi-lingual model for the needed task that could give highly competitive results. We discuss the performance of some of these models (cf.