# Employing a Multilingual Transformer Model for Segmenting Unpunctuated Arabic Text
## 3 PDTS: Punctuation Detector for Text Segmentation  
Encouraged by the MLM method that particularly enables learning complex linguistic patterns of a natural language in a self-supervised objective, we base our proposed punctuation detector for text splitting (PDTS) on top of a robust pre-trained MLM. Given a very large complex text \(X\) (typically a document composed of sentences but without full-stop punctuation) that consists of a sequence of words \(X=\left\{x_{1},x_{2},\cdots,x_{\left|n\right|}\right\}\), PDTS attempts to segment \(X\) into a set of simple sentences \(Y\gets PDTS(X)\). Each simple sentence \(Y_{i}\) represents an independent-clause segmentation without rephrasing. Intuitively, the problem that needs to be tackled lies in detecting the missing punctuations (mainly, full-stop, comma, semicolon, colon, and exclamation/question marks). Considering Figure 1, PDTS begins with detecting the missing punctuations using mBERT (described in A, B, and C), then it applies some linguistic rules for validating them (described in D and E). For a more detailed description of the elemental BERT architecture, including its input and output embedding representations (mentioned in B and C), we recommend that researchers refer to the original works [16,17]. In the following sections, we detail the underlying steps of our PDTS (expressed outrightly in Algorithms 1 and 2) and use Figure 1 for illustration. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Language**} & \multirow{2}{*}{**Reference**} & \multirow{2}{*}{**Segmentation Level**} & \multicolumn{3}{c}{**Method**} & \multirow{2}{*}{**Dialect-**} & \multirow{2}{*}{**Language**} & \multirow{2}{*}{**Transfer**} \\ \cline{3-4} \cline{6-9}  & & & **A** & & & & & **B** & **C** & **D** \\ \hline \multirow{4}{*}{Arabic} & Rhetorical SVM & Phrase to a simple & & & & & & & \\  & classifier [9] & sentence & & & & & & & \\ \cline{2-9}  & SVM and & & & & & & & & \\  & Bi-directional & & Word & & & & & & \\  & LSTM [14] & & & & & & & & \\ \cline{2-9}  & DJAZI (Chunking & & & & & & & \\  & procedure) [15] & & & & & & & & \\  & & & & & & & & & \\ \cline{2-9}  & Regular rule-based & & Simple sentence & & & & & & \\  & expressions [1] & (independent clause) & & & & & & & \\ \cline{2-9}  & Our PDTS & & & & & & & & \\  & & (in)dependent clause & & & & & & Multilingual & \\ \hline \multirow{4}{*}{English} & SegBot [29,30] & Sentence and & & & & & & & \\  & & document & & & & & & \\ \cline{2-9}  & DisSim [24] & Discourse sentence & & & & & & \\ \cline{2-9}  & Three BERT-style & & & & & & & & \\  & _models_[27] & & & & & & & & \\ \cline{2-9}  & Context-preserving & & & & & & & & \\  & approach [4] & & & & & & & \\ \cline{2-9}  & TopicDiff-LDA & & & & & & & \\  & Latent Dirichlet & & & & & & & \\  & Allocation [26] & & & & & & & \\ \hline \hline \end{tabular}  
* A: Chunking procedure based on pre-defined split-delimiters. B: Document/discourse segmentation approaches depending on extracting linguistic annotation features. C: Statistical/classical machine learning methods. D: Deep learning/transformer-based models  
\end{table} 
Table 1: _Cont._```
Function:\(\mathit{PunTokens}\leftarrow\texttt{predPunctuations}(X,\alpha,\theta)\) input :\(X\) an unpunctuated document, consisting of a sequence of words. \(\alpha\) a positive integer parameter for fragmenting \(X\) into smaller pieces, such that \((1\leq\alpha\leq 512)\). \(\theta\) a probability-threshold for temporary accepting the predicted punctuations. output :\(\mathit{PunTokens}\) a set of punctuations predicted by an MLM model (e.g., \(mBERT\)) to be added into \(X\). 1//Declarer a set of user-defined punctuations in \(\mathit{Pun}[]\).