## 3 Tibetan SBD Based on BERT (BPE)
### BERT Model Training  
[PERSON] et al. (2019) proposed a Transformer-based Bidirectional Encoder Model (BERT) in 2019, masking several words with unique tokens before predicting them and preprocessing the bidirectional representation of unlabeled text by jointly regulating left and right up and down in all layers. In this study, the Tibetan BERT pre-trained language model based on two subword methods, Sentencepiece and BPE, is applied in the SBD model. The Tibetan SBD task is investigated based on the BERT + deep learning model. The BERT uses the encoder part of the Transformer, which consists of multiple layers of bi-directional Transformer ([PERSON] et al., 2019) encoder stacked together. Each layer of the Transformer's encoder consists of a multi-head attention mechanism and a feed-forward network with layer normalization. The feed-forward network consists of two fully connected layers and a nonlinear activation function RELU; Figure 2 shows the structure of the multi-head attention mechanism. The multi-head attention mechanism is an extension of the attention mechanism, which independently calculates attention weights in multiple representation subspaces, allowing the model to learn different attention patterns in different subspaces, thereby improving the model's representation ability. The attention mechanism creates three vectors before input: Query vector \((Q)\), Key vector \((K)\), and Value vector \((V)\). Attention is computed in three general steps:  
(1) Calculate the weights; firstly, \(Q\) and all \(K\) calculate the weights using the similarity method. \[f(Q,K_{i})=\left\{ \begin{aligned} & Q^{T}K_{i}\\ & Q^{T}WK_{i}\end{aligned} \right. \tag{1}\]  
Normalization, using the Softmax function to normalize \(f(Q,K_{i})\):  
(2) Sum the normalized weights with \(V\) weighting to obtain Attention:  
\[Attention=(Q,K,V)=\sum_{i}a_{i}V_{i} \tag{2}\]  
where \(Q\), \(K\), and \(V\) denote the query, index, and the value obtained from the query, respectively, all obtained from the input word vectors by linear transformation, and then the value of the Softmax function is solved by Equation (2), after calculating the attentional weights of each position to the other positions, the self-attention mechanism will generate a weighted vector for each position based on these weights, i.e., multiplying each position with the attentional weights of the other positions, and then summing up these products to obtain the weighted vector for that position, i.e., the attentional representation of that position ([PERSON] et al., 2019). In the multi-head attention mechanism, i.e., for a given query \(Q\), essential \(K\), and value \(V\), each attention head \(h_{i}\) (\(i=1,\ 2,\ \dots,\ h\)) is computed as follows:  
\[head_{i}=Attention\Big{(}QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}\Big{)} \tag{3}\]  
Figure 2: Multi-head attention mechanism. The multi-head attention mechanism is calculated as follows:  
\[MultiHead(Q,K,V)=concat(head_{1},head_{2},\dots,head_{n})W^{Q} \tag{4}\]  
In this study, the specific process of sentence boundary disambiguation based on the Tibetan BERT pre-trained language model is as follows:  
(1) Sentences are first segmented, and [CLS] markers are added to the beginning of each sentence, and [SEP] markers are added to the end of each sentence. (2) The labeled sentences are input to the embedding layer: marker embedding, fragment embedding, and position embedding. Token embedding adds [CLS] tokens to the beginning of each sentence while separating each sentence fragment by [SEP]; fragment embedding is used to differentiate between multiple sentence fragments given, mapping the tokens to EA in odd-indexed sentences, and mapping the tokens to EB in even-indexed sentences. Positional embedding encodes the positional information of words in the input, as shown in Equations (5) and (6).