# Employing a Multilingual Transformer Model for Segmenting Unpunctuated Arabic Text
## 3 PDTS: Punctuation Detector for Text Segmentation
Section 4), but for designing our approach, we follow the latter (i.e., the base of our approach is mBERT, which is fine-tuned for the _Mask-fill_ task) as it gives better performance for understanding the Arabic sentences in general. The input and output representations of mBERT (like any other pre-defined encoder-based models) are grounded to token embeddings at different layers (i.e., word token embedding, segment embedding, transformer positional embedding, and contextual token embedding for the output). We consider all these embeddings in our approach except segment embedding (i.e., recognized by the specific BERT token \([SEP]\)) as our input text is a single large complex sentence (i.e., assumed to have no full-stop punctuation). For fine-tuning mBERT on downstream _Mask-fill_ task, we consider the output softmax dense layer and inspect only the contextual token-to-vocab vectors that are mapped with the input token \([MASK]\), see (B and C) of Figure 1. In other words, to predict a masked vocab from contextual-mBERT vectors (i.e., at the output embeddings layers), we implement Equation (1):

\[p(v|X;\theta)=Softmax(W^{T}c) \tag{1}\]

where \(W\in\mathbb{R}^{|X|\times|V|}\) is the weight matrix of the last staked FFNN layer (i.e., used to project the contextual vector (\(c\)) of the specified masked token into mBERT vocabulary (\(V\))), \(v\) is the predicted masked vocab (i.e., has the highest softmax probability score), and \(\theta\) is a decision-threshold parameter. ### Arabic Text Fragments and Masked Token-to-Vocab Prediction

For tokenizing \(X\) into a set of independent clauses, PDTS requires invoking mBERT \(|X|\) times to detect whether the space between every two sequential tokens should include proper punctuation. Let \(X=\langle t_{1},t_{2},t_{i},\cdots,t_{n}\rangle\) be a sequence of \(n\) tokens, \(i\) is the index of token-sequence, we generate \(M=\{t_{1}^{m},t_{2}^{m},t_{i}^{m},\cdots,t_{n}^{m}\}\) by invoking mBERT \(n\) times, such that \(t_{i}^{m}\) represents the temporarily predicted masked vocab between (\(t_{i}\) and \(t_{i+1}\)) when \(i<n\) or after \(t_{i}\) when \(i=n\). To clarify more, we generate \(M\) by (1) inserting \([MASK]\) into \(X\) at \(i\) index (denoted as \(X_{i}^{t}\leftarrow\mathit{insertMASK}(X,i),\forall i\in X\)) and (2) we collect the set of mBERT's outputs, expressed in Equation (2):

\[ \begin{split} t_{1}^{m}&\gets mBERT(\langle t_{1 },[MASK],t_{2},t_{i},\cdots,t_{n}\rangle)\\ t_{i}^{m}&\gets mBERT(\langle t_{1},t_{2},t_{i},[MASK], \cdots,t_{n}\rangle)\\ t_{n}^{m}&\gets mBERT(\langle t_{1},t_{2},t_{i}, \cdots,t_{n},[MASK]\rangle)\end{split}  \tag{2}\]

Intuitively, invoking mBERT many times can affect the runtime efficiency of PDTS, particularly when applying it for a large-scale \(X\) that might contain a long sequence of sentences (e.g., a document containing \(|X|>5\)k). We address scaling PDTS by reducing the size of the input sequence from \(|X|\) down to \(\alpha\) without significantly sacrificing the detection accuracy of \(t_{i}^{m}\) (\(\alpha\) is a positive integer pre-defined parameter). In Algorithm 1, we describe the steps of invoking mBERT with a fixed size of input sequence \(\alpha\) (i.e., supposed to be smaller than \(|X|\)), see lines 6-11. To use PDTS as expressed in Equation (3), the user must first declare a set \(\mathit{Pun}\) containing their desired punctuations to be added in \(X\) (see line 2 of Algorithm 1) besides a decision-threshold \(\theta\) for temporary-accepting/ignoring the predicted punctuation:

\[p(pu_{i}^{m}|t_{i}^{m};\mathit{pun},\theta)=PDTS(X_{i}^{t}) \tag{3}\]

where \(pu_{i}^{m}\) represents the accepted mBERT's output, \(\mathit{pun}\) and \(\theta\) are model parameters used to filter out \(t_{i}^{m}\) (i.e., obtained by Equation (2)). In lines 12-13 of Algorithm 1, we implement Equation (3) to filter out the predicted masked tokens (\(t_{i}^{m}\)), and temporarily store \(pu_{i}^{m}\) in \(\mathit{PunTokens}\) to be validated next by our linguistic rules.