Semi-supervised Thai Sentence Segmentation Using Local and Distant Word Representations

A sentence is typically treated as the minimal syntactic unit used for extracting valuable information from a longer piece of text. However, in written Thai, there are no explicit sentence markers. We proposed a deep learning model for the task of sentence segmentation that includes three main contributions. First, we integrate n-gram embedding as a local representation to capture word groups near sentence boundaries. Second, to focus on the keywords of dependent clauses, we combine the model with a distant representation obtained from self-attention modules. Finally, due to the scarcity of labeled data, for which annotation is difficult and time-consuming, we also investigate and adapt Cross-View Training (CVT) as a semi-supervised learning technique, allowing us to utilize unlabeled data to improve the model representations. In the Thai sentence segmentation experiments, our model reduced the relative error by 7.4% and 10.5% compared with the baseline models on the Orchid and UGWC datasets, respectively. We also applied our model to the task of pronunciation recovery on the IWSLT English dataset. Our model outperformed the prior sequence tagging models, achieving a relative error reduction of 2.5%. Ablation studies revealed that utilizing n-gram presentations was the main contributing factor for Thai, while the semi-supervised training helped the most for English.

section_name: Introduction
paragraphs: Automatic summarization, machine translation, question answering, and semantic parsing operations are useful for processing, analyzing, and extracting meaningful information from text. However, when applied to long texts, these tasks usually require some minimal syntactic structure to be identified, such as sentences BIBREF0 , BIBREF1 , BIBREF2 , which always end with a period (“.”) in English BIBREF3 .
However, written Thai does not use an explicit end-of-sentence marker to identify sentence boundaries BIBREF4 . Prior works have adapted traditional machine learning models to predict the beginning position of a sentence. The authors of BIBREF5 , BIBREF6 , BIBREF7 proposed traditional models to determine whether a considered space is a sentence boundary based on the words and their part of speech (POS) near the space. Meanwhile, Zhou N. et al. BIBREF8 considered Thai sentence segmentation as a sequence tagging problem and proposed a CRF-based model with n-gram embedding to predict which word is the sentence boundary. This method achieves the state-of-the-art result for Thai sentence segmentation and achieves greater accuracy than other models by approximately 10% on an Orchid dataset BIBREF9 .
Several deep learning approaches have been applied in various tasks of natural language processing (NLP), including the long short-term memory BIBREF10 , self-attention BIBREF11 , and other models. Huang Z. et al. BIBREF12 proposed a deep learning sequence tagging model called Bi-LSTM-CRF, which integrates a conditional random field (CRF) module to gain the benefit of both deep learning and traditional machine learning approaches. In their experiments, the Bi-LSTM-CRF model achieved an improved level of accuracy in many NLP sequence tagging tasks, such as named entity recognition, POS tagging and chunking.
The CRF module achieved the best result on the Thai sentence segmentation task BIBREF8 ; therefore, we adopt the Bi-LSTM-CRF model as our baseline. This paper makes the following three contributions to improve Bi-LSTM-CRF for sentence segmentation.
First, we propose adding n-gram embedding to Bi-LSTM-CRF due to its success in BIBREF8 and BIBREF12 . By including n-gram embedding, the model can capitalize on both approaches. First, the model gains the ability to extract past and future input features and sentence level tag information from Bi-LSTM-CRF; moreover, with the n-gram addition, it can also extract a local representation from n-gram embedding, which helps in capturing word groups that exist near sentence boundary. Although Jacovi A. et al. BIBREF13 reported that a convolutional neural network (CNN) can be used as an n-gram detector to capture local features, we chose n-gram embedding over a CNN due to its better accuracy, as will be shown in Section SECREF8 .
Second, we propose adding incorporative distant representation into the model via a self-attention mechanism, which can focus on the keywords of dependent clauses that are far from the considered word. Self-attention has been used in many recent state-of-the-art models, most notably the transformer BIBREF11 and BERT BIBREF14 . BERT has outperformed Bi-LSTM on numerous tasks, including question answering and language inference. Therefore, we choose to use self-attention modules to extract distant representations along with local representations to improve model accuracy.
Third, we also apply semi-supervised learning BIBREF15 , allowing us to employ unlimited amounts of unlabeled data, which is particularly important for low-resource languages such as Thai, for which annotation is costly and time-consuming. Many semi-supervised learning approaches have been proposed in the computer vision BIBREF16 , BIBREF17 and natural language processing BIBREF18 , BIBREF19 , BIBREF20 fields. Our choice for semi-supervised learning to enhance model representation is Cross-View Training (CVT) BIBREF20 . Clark K. et al. BIBREF20 claims that CVT can improve the representation layers of the model, which is our goal. However, CVT was not designed to be integrated with self-attention and CRF modules; consequently, we provide a modified version of CVT in this work.
Based on the above three contributions, we pursue two main experiments. The first experiment was conducted on two Thai datasets, Orchid and UGWC BIBREF21 , to evaluate our Thai sentence segmentation model. In this case, our model achieves F1 scores of 92.5% and 88.9% on Orchid and UGWC, respectively, and it outperforms all the baseline models. The second experiment was executed on the IWSLT dataset BIBREF22 and involves an English-language punctuation restoration task. This experiment demonstrates that our model is generalizable to different languages. Our model, which does not require pretrained word vectors, improved the overall F1 score by 0.9% compared to the baselines, including a model that uses pretrained word vectors.
There are five sections in the remainder of this paper. Section SECREF2 reviews the related works on Thai sentence segmentation, English punctuation restoration and introduces the original CVT. Section SECREF3 describes the proposed model architecture and the integration of cross-view training. The datasets, implementation process and evaluation metrics are explained in Section SECREF4 . The results of the experiments are discussed in Section SECREF5 . Finally, Section SECREF6 concludes the paper.
section_name: Related Works
paragraphs: This section includes three subsections. The first subsection concerns Thai sentence segmentation, which is the main focus of this work. The task of English punctuation restoration, which is similar to our main task, is described in the second subsection. The last subsection describes the original Cross-View Training initially proposed in BIBREF20 .
section_name: Thai sentence segmentation
paragraphs: In Thai, texts do not contain markers that definitively identify sentence boundaries. Instead, written Thai usually uses a space as the vital element that separates text into sentences. However, there are three ways that spaces are used in this context BIBREF23 : before and after an interjection, before conjunctions, and before and after numeric expressions. Therefore, segmenting text into sentences cannot be performed simply by splitting a text at the spaces.
Previous works from BIBREF5 , BIBREF6 , BIBREF7 have focused on disambiguating whether a space functions as the sentence boundary. These works extract contextual features from words and POS around the space. Then, the obtained features around the corresponding space are input into traditional models to predict whether space is a sentence boundary.
Although a space is usually considered essential as a sentence boundary marker, approximately 23% of the sentences end without a space character in one news domain corpus BIBREF8 . Hence, Zhou N. et al. BIBREF8 proposed a word sequence tagging CRF-based model in which all words can be considered as candidates for the sentence boundary. A space is considered as only one possible means of forming a sentence boundary. The CRF-based model BIBREF24 , which is extracted from n-grams around the considered word, achieves a F1 score of 91.9%, which is approximately 10% higher than the F1 scores achieved by other models BIBREF5 , BIBREF6 , BIBREF7 on the Orchid dataset, as mentioned in BIBREF8 .
In this work, we adopt the concept of word sequence tagging and compare it with two baselines: the CRF-based model with n-gram embedding, which is currently the state-of-the-art for Thai sentence segmentation, and the Bi-LSTM-CRF model, which is currently the deep learning state-of-the-art approach for sequence tagging.
section_name: English punctuation restoration
paragraphs: Most languages use a symbol that functions as a sentence boundary; however, a few do not use sentence markers including Thai, Lao and Myanmar. Thus, few studies have investigated sentence segmentation in raw text. However, studies on sentence segmentation, which is sometimes called sentence boundary detection, are still found in the speech recognition field BIBREF25 . The typical input to speech recognition model is simply a stream of words. If two sentences are spoken back to back, by default, a recognition engine will treat it as one sentence. Thus, sentence boundary detection is also considered a punctuation restoration task in speech recognition because when the model attempts to restores the period in the text, the sentence boundary position will also be defined.
Punctuation restoration not only provides a minimal syntactic structure for natural language processing, similar to sentence boundary detection but also dramatically improves the readability of transcripts. Therefore, punctuation restoration has been extensively studied. Many approaches have been proposed for punctuation restoration that use different features, such as audio and textual features. Moreover, punctuation restoration is also considered to be a different machine learning problem, such as word sequence tagging and machine translation.
A combination of audio and textual features were utilized in BIBREF26 , BIBREF27 , BIBREF28 to predict and restore punctuation, including pitch, intensity and pause duration, between words. We ignore these features in our experiment because our main task—Thai sentence segmentation— does not include audio features.
Focusing only on textual features, there are two main approaches, namely, word sequence tagging and machine translation. For the machine translation approach, punctuation is treated as just another type of token that needs to be recovered and included in the output. The methods in BIBREF29 , BIBREF30 , BIBREF31 restore punctuation by translating from unpunctuated text to punctuated text. However, our main task, sentence segmentation, is an upstream task in text processing, unlike punctuation restoration, which is considered a downstream task. Therefore, the task needs to operate rapidly; consequently, we focus only on the sequence tagging model, which is less complex than the machine translation model.
In addition to those machine translation tasks, both traditional approaches and deep learning approaches must solve a word sequence tagging problem. Of the traditional approaches, contextual features around the considered word were used to predict following punctuation in the n-gram BIBREF32 and CRF model approaches BIBREF33 , BIBREF34 . Meanwhile, in the deep learning approaches, a deep convolutional neural network BIBREF35 , T-LSTM (Textual-LSTM) BIBREF26 and a bidirectional LSTM model with an attention mechanism, called T-BRNN BIBREF36 , have been adopted to predict a punctuation sequence from the word sequence. T-BRNN BIBREF36 was proposed to solve the task as a word-sequence tagging problem, and it is currently the best model that uses the word sequence tagging approach. Tilk O. et al. BIBREF36 also proposed a variant named T-BRNN-pre, which integrates pretrained word vectors to improve the accuracy.
To demonstrate that our model is generalizable to other languages, we compare it with other punctuation restoration models, including T-LSTM, T-BRNN, and T-BRNN-pre. These models adopt a word sequence tagging approach and do not utilize any prosodic or audio features.
section_name: Cross-View Training
paragraphs: CVT BIBREF20 is a semi-supervised learning technique whose goal is to improve the model representation using a combination of labeled and unlabeled data. During training, the model is trained alternately with one mini-batch of labeled data and INLINEFORM0 mini-batches of unlabeled data.
Labeled data are input into the model to calculate the standard supervised loss for each mini-batch and the model weights are updated regularly. Meanwhile, each mini-batch of unlabeled data is selected randomly from the pool of all unlabeled data; the model computes the loss for CVT from the mini-batch of unlabeled data. This CVT loss is used to train auxiliary prediction modules, which see restricted views of the input, to match the output of the primary prediction module, which is the full model that sees all the input. Meanwhile, the auxiliary prediction modules share the same intermediate representation with the primary prediction module. Hence, the intermediate representation of the model is improved through this process.
Similar to the previous work, we also apply CVT to a sequence tagging task. However, our model is composed of self-attention and CRF modules, which were not included in the sequence tagging model in BIBREF20 . The previous CVT was conducted on an LSTM using the concepts of forward and backward paths, which are not intuitively acquired by the self-attention model. Moreover, the output used to calculate CVT loss was generated by the softmax function, which does not operate with CRF. Thus, in our study, both the primary and auxiliary prediction modules needed to be constructed differently from the original ones.
architecture/Semi
As discussed in Section SECREF3 , CVT requires primary and auxiliary prediction modules for training with unlabeled data to improve the representation. Thus, we construct both types of prediction modules for our model. The flow of unlabeled data, which is processed to obtain a prediction by each module, is shown in Fig. . The output of each prediction module is transformed into the probability distribution of each class by the softmax function and then used to calculate INLINEFORM0 , as shown in cvtloss. DISPLAYFORM0 
The INLINEFORM0 value is based on the Kullback–Leibler divergence (KL divergence) between the probability distribution of the primary INLINEFORM1 output and those of two auxiliary modules, INLINEFORM2 and INLINEFORM3 , where INLINEFORM4 . The KL divergence at each timestep is averaged when the timesteps are dropped timesteps D, which is described in Section UID25 . The details of the primary and auxiliary prediction modules, which are used in the INLINEFORM5 calculation, are described in the following subsections.
In BIBREF20 , the output of the primary prediction module is acquired from the last layer and used to predict tags. However, our model uses a CRF layer to decode the tags, instead of the softmax function, whose input is the output from the last fully connected layer. Thus, the probability distribution of a primary prediction module should be the marginal probability acquired from the CRF layer. Nevertheless, the forward-backward algorithm for the marginal probability calculation is time-consuming with a time complexity is INLINEFORM0 , where INLINEFORM1 is the sequence length, and INLINEFORM2 is the number of tags. To reduce the training time, the probability distribution of the primary prediction module INLINEFORM3 is instead obtained from the output of the Softmax function, whose input is a virtual logit vector INLINEFORM4 , as shown in primary. DISPLAYFORM0 
Two auxiliary views are included to improve the model. The first view is generated from a recurrent representation vector INLINEFORM0 to acquire the local probability distribution INLINEFORM1 , where INLINEFORM2 . The second view is generated from the low-level distant representation vectors INLINEFORM3 to acquire the probability distribution of a distant structure in the low-level module INLINEFORM4 , where INLINEFORM5 . By generating the views from these representation vectors separately, the local and distant structures in the low-level module can improve equally.
Although both representation vectors are used separately to create auxiliary views, the input of each structure is still not restricted, unlike BIBREF20 , where the input is restricted to only previous or future tokens. Because BERT, which is trained by the masked language model, outperforms OpenAI GPT, which uses an autoregressive approach for training as reported in BIBREF14 , we adopt the concept of the masked language model BIBREF37 to obtain both auxiliary views. This approach allows the representation to fuse the left and the right context, which results in a better representation. By using the masked language model, some tokens at each timestep are randomly dropped and denoted as removed tokens INLINEFORM0 ; then, the remaining tokens are used to obtain auxiliary predictions in the dropped timesteps INLINEFORM1 , as shown in Fig. . The details of both auxiliary prediction modules are described below.
architecture/Masklanguage
For recurrent representation vectors, if one of the tokens is dropped, the related n-gram tokens that include the dropped tokens will also be dropped. For example, if INLINEFORM0 is dropped, INLINEFORM1 and INLINEFORM2 will also be dropped as removed tokens in the case of a bigram. The remaining n-gram tokens are then used to obtain the recurrent representation vectors at the dropped timesteps. Then, the vectors are provided as an input to the softmax function to obtain the probability distribution of the first auxiliary prediction module, as shown in auxlocal. DISPLAYFORM0 
In the other auxiliary prediction module, a sequence of the low-level distant representation vectors is generated and some tokens are dropped. This sequence of vectors is also input into the Softmax function, just as in the first auxiliary prediction module, and the output is another probability distribution, which is the second auxiliary prediction, as shown in auxdistant. DISPLAYFORM0 
section_name: Proposed method
paragraphs: In this section, we describe our proposed method in two subsections. The first subsection specifies the model architecture and the details of each module. Our first and second contributions, which are local and distant representations, are mainly described in this subsection. Meanwhile, the second subsection expounds on how the model is trained with unlabeled data through the modified CVT, which is our third contribution.
section_name: Model architecture
paragraphs: In this work, the model predicts the tags INLINEFORM0 for the tokens in a word sequence INLINEFORM1 where INLINEFORM2 is the sequence size and INLINEFORM3 , INLINEFORM4 denote the token and its tag at timestep INLINEFORM5 , respectively. Each token INLINEFORM6 consists of a word, its POS and its type. There are five defined word types: English, Thai, punctuation, digits, and spaces.
The tag set INLINEFORM0 is populated based on the considered task. In Thai sentence segmentation, the assigned tags are INLINEFORM1 and INLINEFORM2 ; INLINEFORM3 denotes that the corresponding word is a sentence boundary considered as the beginning of a sentence, while and INLINEFORM4 denotes that the word is not a sentence boundary. Meanwhile, there are four tags in the punctuation restoration task. Words not followed by any punctuation are tagged with INLINEFORM5 . Words that are followed by a period “.”, comma “,” or question mark “?” are tagged to INLINEFORM6 , INLINEFORM7 , and INLINEFORM8 , respectively.
architecture/MainArchitecture
Our model architecture is based on Bi-LSTM-CRF, as shown in Fig. . The model is divided into three modules. The first, low-level module, consists of two separate structures: local and distant structures. The second, high-level module, contains a sequence of stacked bidirectional LSTM and self-attention layers. The final module, the prediction module, is responsible for predicting the tags INLINEFORM0 . Each module is described more completely in the next three subsections.
A sequence of word tokens is input into the low-level module. The input tokens pass through two structures. The first structure generates a sequence of local representation vectors INLINEFORM0 , and the second structure generates low-level distant representation vectors INLINEFORM1 . After obtaining both sequences of representation vectors, the local representation vectors are fed to the Bi-LSTM to obtain the recurrent representation vectors INLINEFORM2 , as shown in recurrentrep. Then, the recurrent and distant representation vectors are concatenated to form the low-level representation vector INLINEFORM3 , as shown in low-level: DISPLAYFORM0 DISPLAYFORM1 
This structure is shown as the left submodule of the low-level module in Fig. . It extracts the local representation vectors INLINEFORM0 . Its input tokens are used to create n-gram tokens, which are unigrams INLINEFORM1 , bigrams INLINEFORM2 , and trigrams INLINEFORM3 . Each n-gram token is represented as an embedding vector, which is classified as a unigram embedding vector INLINEFORM4 , a bigram embedding vector INLINEFORM5 or a trigram embedding vector INLINEFORM6 . Each vector INLINEFORM7 is mapped from a token by gram embedding INLINEFORM8 , which is a concatenated vector of the word embedding INLINEFORM9 , POS embedding INLINEFORM10 and type embedding INLINEFORM11 , as shown in eq:embedding: DISPLAYFORM0 
Each n-gram token at timestep INLINEFORM0 is generated by the previous, present and next token and embedded into vectors as shown in uni,bi,tri. The unigram embedding at timestep INLINEFORM1 is a unigram embedding of the current token INLINEFORM2 . The bigram embedding vector at timestep INLINEFORM3 is a bigram embedding of the previous and present tokens INLINEFORM4 , and the trigram embedding vector at timestep INLINEFORM5 is a trigram embedding of the previous, present and next tokens INLINEFORM6 : DISPLAYFORM0 DISPLAYFORM1 
At each timestep INLINEFORM0 , a local representation vector INLINEFORM1 is combined from the n-gram embedding vectors generated from the context around INLINEFORM2 . A combination of embedding vectors, which is used to construct a local representation vector, is shown in n-gramcombination. A combination consists of the unigram, bigram, and trigram embedding vectors at timesteps INLINEFORM3 , INLINEFORM4 and INLINEFORM5 and it is a concatenation of all the embedding vectors: DISPLAYFORM0 
The distant structure, which is a self-attention module, is shown in Fig. on the right side of the low-level module. The structure extracts low-level distant representation vectors INLINEFORM0 from a sequence of unigram embedding vectors INLINEFORM1 , as shown in distantattention. In this case, the self-attention module is a scaled dot-product attention BIBREF11 , where key, query, and value vectors are the linear projections of the unigram embedding vectors shown in Fig. . The linear transformations for key, query, and value are learned separately and updated in the model through backpropagation. The output vector, which is the scaled dot-product attention at each timestep, is concatenated with the input vector INLINEFORM2 and projected by a linear transformation. That projected vector is the output vector of a self-attention module, which is a low-level distant representation vector.
architecture/selfattention DISPLAYFORM0 
The low-level representation vectors INLINEFORM0 are used as the input for this module, which outputs the high-level representation vectors INLINEFORM1 whose calculation is shown in high-level. The high-level module, as shown in Fig. , is composed of a stacked bidirectional LSTM and a self-attention modules. A stacked bidirectional LSTM contains K layers of bidirectional LSTMs in which the output from the previous bidirectional LSTM layer is the input of the next bidirectional LSTM layer. The self-attention part of this structure is the same as that in the low-level distant structure. The self-attention module helps to generate the high-level distant representation vectors that are output by the high-level module. DISPLAYFORM0 
The prediction module is the last module. It includes two layers: a fully connected layer and a CRF layer. In the fully connected layer, the output vectors from the high-level module are projected by a linear transformation as shown in virtual. The purpose of this layer is to create the virtual logit vectors INLINEFORM0 , which represent the probability distribution for CVT, as discussed in Section SECREF21 . Therefore, the number of dimensions of logits equals the number of possible tags in each task: DISPLAYFORM0 
The CRF layer is responsible for predicting the tag INLINEFORM0 of a token at each timestep, as shown in crf. The layer receives a sequence of virtual logit vectors ( INLINEFORM1 ) as input and then decodes them to a sequence of tags INLINEFORM2 using the Viterbi algorithm. DISPLAYFORM0 
section_name: Datasets
paragraphs: Three datasets are used in the experiments as described in the following subsections. We use two datasets for Thai sentence segmentation, and the third dataset is used for English punctuation restoration. The statistics of the preprocessed data are shown in Table TABREF31 , including the number of sequences and the number of vocabulary words in each dataset. We also calculate the average number of words per passage in the unlabeled data that do not appear in the labeled data, as shown in Table TABREF32 .
This dataset is a Thai part-of-speech-tagged dataset containing 10,864 sentences. In the corpus, text was separated into paragraphs, sentences, and words hierarchically by linguists. Each word was also manually assigned a POS by linguists. These data include no unlabeled data with the same word segmentation and POS tag set. Hence, we do not execute CVT on this dataset.
Our data preprocessing on the ORCHID corpus was similar to that in BIBREF8 : all the comments are removed, and the data are partitioned into 10 parts containing equal numbers of sentences to support 10-fold cross-validation. Each training set is split into one part used for validation and the rest is used for model training. Subsequently, all the words in each dataset are concatenated and then separated into sequences with 200 words per instance. Each sequence always begins with the first word of a sentence. If a sequence ends with an unfinished sentence, the next sequence starts with that complete sentence.
This Thai dataset includes many types of labeled data useful in sentence segmentation tasks. The raw text was generated by users having conversations in the financial domain and were acquired mainly by crawling social sites. The labeled data for sentence segmentation were manually annotated by linguists using the definitions in BIBREF21 .
At the time of this study, the dataset was extended from that in BIBREF21 ; the data were collected from January 2017 to December 2017. The labeled dataset includes 48,374 passages. To support semi-supervised learning, the first 3 months of data (96,777 passages) are unlabeled.
Because the data stem from social media, some text exists that cannot be considered as part of any sentence, such as product links, symbols unrelated to sentences, and space between sentences. These portions were not originally annotated as sentences by the linguists. However, in this work, we treat these portions as individual sentences and tag the first word of each fraction as the sentence boundary.
For evaluation purposes, the collection of passages in this dataset is based on 5-fold cross-validation, similar to the previous work BIBREF21 . The passages are treated as input sequences for the model. For each passage, word segmentation and POS tagging are processed by the custom models from this dataset.
We adopted this English-language dataset to enable comparisons with models intended for other languages. The dataset is composed of TED talk transcripts. To compare our model with those of previous works, we selected the training dataset for the machine translation track in IWSLT2012 and separated it into training and validation sets containing 2.1 million and 295 thousand words, respectively. The testing dataset is the IWSLT2011 reference set, which contains 13 thousand words. To acquire unlabeled data for semi-supervised learning, we adopted the IWSLT2016 machine translation track training data; duplicate talks that also appear in IWSLT2012 are discarded.
The data preprocessing follows the process in BIBREF36 . Each sequence is generated from 200 words, of which beginning is always the first word in a sentence. If a sentence is cut at the end of a sequence, that sentence is copied in full to the beginning of the next sequence.
To use our model, the POS of each word is required. However, the IWSLT dataset contains only the raw text of transcripts and does not include POS tags. Thus, we implement POS tagging using a special library BIBREF38 to predict the POS of each word.
section_name: Implementation Detail
paragraphs: Before mapping each token included in the unigram, bigram, and trigram to the embedding vector, we limit the minimum frequency of occurring words that are not marked as an unknown token. There are 2 parameters set for the unigram INLINEFORM0 and the remaining INLINEFORM1 , respectively. We found that model accuracy is highly sensitive to these parameters. Therefore, we use a grid search technique to find the best value for both parameters for the model.
We apply two optimizers used in this work: Adagard BIBREF39 and Adam BIBREF40 , whose learning rates are set to 0.02 and 0.001 for the Thai and English datasets, respectively. To generalize the model, we also integrate L2 regularization with an alpha of 0.01 to the loss function for model updating. Moreover, dropout is applied to the local representation vectors, recurrent representation vectors, between all bidirectional LSTMs and enclosed by the self-attention mechanism in the high-level module.
During training, both the supervised and semi-supervised models are trained until the validation metrics stop improving; the metrics are (1) sentence boundary F1 score and (2) overall F1 score for Thai sentence segmentation and English punctuation restoration, respectively.
CVT has three main parameters that impact model accuracy. The first is the drop rate of the masked language model, which determines the number of tokens that are dropped and used for learning auxiliary prediction modules as described in Section SECREF21 . The second is the number of unlabeled mini-batches INLINEFORM0 used for training between supervised mini-batches. Third, rather than using the same dropout rate for the local representation vectors, a new dropout rate is assigned.
The details of hyperparameters such as the hidden size of each layer and dropout rate are given in Section SECREF7 .
section_name: Evaluation
paragraphs: During the evaluation, each task is assessed using different metrics based on previous works. For Thai sentence segmentation, three metrics are used in the evaluation: sentence boundary F1 score, non-sentence boundary F1 score, and space correct BIBREF8 . In this work, we mainly focus on the performance of sentence boundary prediction and not non-sentence boundary prediction or space prediction. Therefore, we make comparisons with other models regarding only their sentence boundary F1 scores. The equation for the sentence boundary F1 score metric is shown in f1sb. In calculating the F1 score, the positive class is defined as the sentence boundary, and the negative class is defined as the non-sentence boundary. INLINEFORM0 INLINEFORM1 
For English punctuation, the evaluation is measured on each type of punctuation and overall F1 score. For the punctuation restoration task, we care only about the performance of the samples belonging to the classes that are tagged to words followed by punctuation; therefore class INLINEFORM0 , which represents words not immediately followed by punctuation, is ignored in the evaluation. Consequently, the overall F1 score does not include INLINEFORM1 as the positive class in f1overall. INLINEFORM2 INLINEFORM3 
To compare the performance of each punctuation restoration model in a manner similar to sentence segmentation, the 2-class F1 score is calculated to measure model accuracy, as shown in f12class. The calculation of this metric is the same as that used in BIBREF35 . The metric considers only where the punctuation position is and ignores the type of restored punctuation. Therefore, this measure is similar to the metric sentence boundary F1, which only considers the position of the missing punctuation. INLINEFORM0 INLINEFORM1 
section_name: Results and discussions
paragraphs: We report and discuss the results of our two tasks in four subsections. The first and second subsections include the effect of local representation and distant representation, respectively. The impact of CVT is explained in the third subsection. The last subsection presents a comparison of our model and all the baselines. Moreover, we also conduct paired t-tests to investigate the significance of the improvement from each contribution, as shown in Section SECREF9 .
section_name: Effect of local representation
paragraphs: To find the effect of local representation, we compare a standard Bi-LSTM-CRF model using our full implementation to the model that includes n-gram embedding to extract local representation. In tab:thairesult,tab:engresult, the standard Bi-LSTM-CRF model is represented as Bi-LSTM-CRF (row (e)), while the models with local features are represented as INLINEFORM0 (row (f)).
The results in Table TABREF45 show that using n-gram to obtain the local representation improves the F1 score of the model from 90.9% (row (e)) to 92.4% (row (f)) on the Orchid dataset and from 87.6% (row (e)) to 88.7% (row (f)) on the UGWC dataset. These results occur because many word groups exist that can be used to signal the beginning and end of a sentence in Thai. Word groups always found near sentence boundaries can be categorized into 2 groups. The first group consists of final particles, e.g., “thaiนะ|คะ ” (na | kha), “thaiนะ|ครับ ” (na | khrạb), “thaiเลย|ครับ ” (ley | khrạb), “thaiแล้ว|ครับ ” (laêw | khrạb), and others. These word groups are usually used at the ends of sentences to indicate the formality level. For instance, the model with local representation can detect the sentence boundary at “thaiครับ ” (khrạb) that is followed by “thaiแล้ว ” (laêw), as shown in Fig. , while the model without local representation cannot detect the word as a sentence boundary. The second group consists of conjunctions that are always used at the beginnings of sentences, e.g., “thaiจาก|นั้น (after that) ”, “thaiไม่|งั้น (otherwise) ” and others. The model that uses n-gram to capture word group information is better able to detect word groups near sentence boundaries. Thus, this model can identify these sentence boundaries easily in the Thai language.
result/cherrypickngram
In contrast, for the English dataset, local representation using n-gram drops the overall F1 score of punctuation restoration from 64.4% (row (e)) to 63.6% (row (f)), as shown in Table TABREF47 . However, the 2-class F1 score increases slightly from 81.4% (row (e)) to 81.8% (row (f)) when compared to the Bi-LSTM-CRF model, which does not integrate n-gram embedding. Common phrases such as ”In spite of”, ”Even though” and ”Due to the fact” might provide strong cues for punctuation; however, such phrases can be found at both the beginnings and in the middle of sentences. Because such phrases can be used in both positions, they may follow commas when they are in the middle of the sentence or periods when they are at the beginning of a sentence. However, they still follow either a period or a comma; consequently, such phrases can still help identify whether the punctuation should be restored, which increases the 2-class F1 score, which considers only the positions of missing punctuation. Moreover, English does not use the concept of a final particle usually found at the end of the sentence—similar to the Thai word group mentioned earlier—including “thaiนะ|คะ ”(na | kha), “thaiนะ|ครับ ”(na | khrạb), “thaiเลย|ครับ ” (ley|khrạb), “thaiแล้ว|ครับ ” (laêw | khrạb) and others. Therefore, the word groups captured by n-gram can only help to identify where punctuation should be restored but they do not help the model determine the type of punctuation that should be restored.
section_name: Effect of distant representation
paragraphs: The effect of this contribution can be found by comparing the model that integrates the distant representation and the model that does not. The model with distant features integrated is represented as INLINEFORM0 (row (g)) in both tables. In this case, the distant representation is composed of the self-attention modules in both the low- and high-level modules, as shown in Fig. .
From the combination of local and distant representation, the results in tab:thairesult,tab:engresult show that the distant feature improves the accuracy of the model on all datasets compared to the model with no distant representation. The F1 scores of the sentence segmentation models improved slightly, from 92.4% and 88.7% (row (f)) to 92.5% and 88.8% (row (g)) on the Orchid and UGWC datasets, respectively. For the IWSLT dataset, the distant feature can recover the overall F1 score of punctuation restoration, which is degraded by the n-gram embedding; it improves from 63.6% (row (f)) to 64.5% (row (g)). The reason is that the self-attention modules focus selectively on certain parts of the passage. Thus, the model focuses on the initial words of the dependent clauses, which helps in classifying which type of punctuation should be restored. An example is shown in Fig. : the model with distant representation classifies the punctuation after ”her” as a ”COMMA” because ”Before” is the word that indicates the dependent clause. Meanwhile, the model without distant representation predicts the punctuation as a ”PERIOD” because there is no self-attention module; therefore, it does not focus on the word ”Before”. Overall, the model that includes both local and distant representation can generally be used for both sentence segmentation and punctuation restoration, and it outperforms both baseline models.
result/cherrypickdistant
section_name: Effect of Cross-View Training (CVT)
paragraphs: To identify the improvement from CVT, we compared the models that use different training processes: standard supervised training ( INLINEFORM0 ) and CVT ( INLINEFORM1 ). The model trained with CVT improves the accuracy in terms of the F1 score on both Thai and English datasets, as shown in tab:thairesult,tab:engresult (row (g) vs row (h)).
This experiment was conducted only on the UGWC dataset because no unlabeled data are available in the Orchid dataset, as mentioned in Section UID33 . The model improves the F1 score slightly, from 88.8% (row (g)) to 88.9% (row (h)) on the UGWC dataset. This result occurs because both the labeled and unlabeled data in the UGWC dataset are drawn from the same finance domain. The average number of new words found in a new unlabeled data passage is only 0.650, as shown in Table TABREF32 . Therefore, there is little additional information to be learned from unlabeled data.
CVT also improved the model on the IWSLT dataset, from an overall F1 score of 64.5% (row (g)) to 65.3% (row (h)) and from a 2-class F1 score of 81.7% to 82.7%. Because both the labeled and unlabeled data were collected from TED talks, the number of vocabulary words grows substantially more than in the UGWC dataset because the talks cover various topics. In this dataset, average 1.225 new words found in each new unlabeled data passage, as shown in Table TABREF32 ; consequently the model representation learns new information from these new words effectively.
section_name: Comparison with baseline models
paragraphs: For the Thai sentence segmentation task, our model is superior to all the baselines on both Thai sentence segmentation datasets, as shown in Table TABREF45 . On the Orchid dataset, the supervised model that includes both local and distant representation was adopted for comparison to the baseline model. Our model improves the F1 score achieved by CRF-ngram, which is the state-of-the-art model for Thai sentence segmentation in Orchid, from 91.9% (row (d)) to 92.5% (row (g)). Meanwhile, in the UGWC dataset, our CVT model (row (h)) achieves an F1 score of 88.9%, which is higher than the F1 score of both the baselines (CRF-ngram and Bi-LSTM-CRF (rows d and e, respectively)). Thus, our model is now the state-of-the-art model for Thai sentence segmentation on both the Orchid and UGWC datasets.
Our model outperforms all the sequence tagging models. T-BRNN-pre (row (c)) is the current state-of-the-art model, as shown in Table TABREF47 . The CVT model improves the overall F1 score from the 64.4% of T-BRNN-pre to 65.3% (row (h)), despite the fact that T-BRNN-pre integrates a pretrained word vector. Moreover, our model also achieves a 2-class F1 score 1.3% higher than that of Bi-LSTM-CRF (row (e)).
section_name: Conclusions
paragraphs: In this paper, we propose a novel deep learning model for Thai sentence segmentation. This study makes three main contributions. The first contribution is to integrate a local representation based on n-gram embedding into our deep model. This approach helps to capture word groups near sentence boundaries, allowing the model to identify boundaries more accurately. Second, we integrate a distant representation obtained from self-attention modules to capture sentence contextual information. This approach allows the model to focus on the initial words of dependent clauses (i.e., ”Before”, ”If”, and ”Although”). The last contribution is an adaptation of CVT, which allows the model to utilize unlabeled data to produce effective local and distant representations.
The experiment was conducted on two Thai datasets, Orchid and UGWC, and one English punctuation restoration dataset, IWSLT. English punctuation restoration is similar to our Thai sentence segmentation. On the Thai sentence segmentation task, our model achieves F1 scores of 92.5% and 88.9% on the Orchid and UGWC datasets, constituting a relative error reduction of 7.4% and 10.5%, respectively. On the English punctuation task, the 2-class F1 score reached 82.7% when considering only two punctuation classes (making the task similar to sentence segmentation in Thai). Moreover, our model outperforms the model integrated with pretrained word vectors in terms of the overall F1 score on the IWSLT dataset. Based on our contributions, the local representation scheme has the highest impact on the Thai corpus, while the distant representation and CVT result in strong improvements on the English dataset.
Moreover, our model can also be applied to elementary discourse unit (EDU) segmentation, which is used as the minimal syntactic unit for downstream tasks such as text summarization and machine translation. However, no experiments have been conducted to determine how different sentences and EDUs affect downstream tasks. Therefore, the evaluation of downstream tasks from different sources needs to be studied.
section_name: Acknowledgment
paragraphs: This paper was supported by KLabs at Kasikorn Business Technology (KBTG), who provided facilities and data. The procedures that were conducted based on social data are visible to the public, and ethical issues that can arise from the use of such data were addressed. We would like to thank the linguists Sasiwimon Kalunsima, Nutcha Tirasaroj, Tantong Champaiboon and Supawat Taerungruang for annotating the UGWC dataset used in this study.
section_name: Hyperparameters
paragraphs: The hyperparameter values were determined through a grid search to find their optimal values on the different datasets. All the hyperparameters for each dataset are shown in Table TABREF55 . The optimal values from the grid search depend on the task. For Thai sentence segmentation, the hyperparameters are tuned to obtain the highest sentence boundary F1 score, while the overall F1 score is used to tune the parameters for English punctuation restoration.
section_name: Comparison of CNN and n-gram models for local representation
paragraphs: Jacovi A. et al. BIBREF13 proposed that a CNN can be used as an n-gram detector to capture local text features. Therefore, we also performed an experiment to compare a CNN and n-gram embedded as local structures. The results in Table TABREF56 show that the model using the embedded n-gram yields greater improvement than the one using an embedded CNN on the Orchid and UGWC datasets.
section_name: Statistical Tests for Thai sentence segmentation
paragraphs: To prove the significance of the model improvements, we compared the cross-validation results using paired t-tests to obtain the p-values, which are shown in Table TABREF57 for the Orchid dataset and Table TABREF58 for the UGWC dataset.

Questions and Answers:
question: Which deep learning architecture do they use for sentence segmentation?
answers: answer: unanswerable: no
extractive_spans: Bi-LSTM-CRF
evidence: The CRF module achieved the best result on the Thai sentence segmentation task BIBREF8 ; therefore, we adopt the Bi-LSTM-CRF model as our baseline. This paper makes the following three contributions to improve Bi-LSTM-CRF for sentence segmentation.
answer: unanswerable: no
extractive_spans: Bi-LSTM-CRF
evidence: Several deep learning approaches have been applied in various tasks of natural language processing (NLP), including the long short-term memory BIBREF10 , self-attention BIBREF11 , and other models. Huang Z. et al. BIBREF12 proposed a deep learning sequence tagging model called Bi-LSTM-CRF, which integrates a conditional random field (CRF) module to gain the benefit of both deep learning and traditional machine learning approaches. In their experiments, the Bi-LSTM-CRF model achieved an improved level of accuracy in many NLP sequence tagging tasks, such as named entity recognition, POS tagging and chunking.
The CRF module achieved the best result on the Thai sentence segmentation task BIBREF8 ; therefore, we adopt the Bi-LSTM-CRF model as our baseline. This paper makes the following three contributions to improve Bi-LSTM-CRF for sentence segmentation.
question: How do they utilize unlabeled data to improve model representations?
answers: answer: unanswerable: no
extractive_spans: During training, the model is trained alternately with one mini-batch of labeled data and INLINEFORM0 mini-batches of unlabeled data.
evidence: CVT BIBREF20 is a semi-supervised learning technique whose goal is to improve the model representation using a combination of labeled and unlabeled data. During training, the model is trained alternately with one mini-batch of labeled data and INLINEFORM0 mini-batches of unlabeled data.
Labeled data are input into the model to calculate the standard supervised loss for each mini-batch and the model weights are updated regularly. Meanwhile, each mini-batch of unlabeled data is selected randomly from the pool of all unlabeled data; the model computes the loss for CVT from the mini-batch of unlabeled data. This CVT loss is used to train auxiliary prediction modules, which see restricted views of the input, to match the output of the primary prediction module, which is the full model that sees all the input. Meanwhile, the auxiliary prediction modules share the same intermediate representation with the primary prediction module. Hence, the intermediate representation of the model is improved through this process.
As discussed in Section SECREF3 , CVT requires primary and auxiliary prediction modules for training with unlabeled data to improve the representation. Thus, we construct both types of prediction modules for our model. The flow of unlabeled data, which is processed to obtain a prediction by each module, is shown in Fig. . The output of each prediction module is transformed into the probability distribution of each class by the softmax function and then used to calculate INLINEFORM0 , as shown in cvtloss. DISPLAYFORM0
FLOAT SELECTED: Figure 3: Two auxiliary predictions (~pt,local and ~pt,distant) are obtained from the local and global structures in the low-level module. The primary prediction ~pt,primary is obtained from the virtual logit vector ~gt

Figure and Table Captions:
caption: Figure 1: Model architecture that integrates local and distant representation. This model is composed of three main modules: a low-level module, a high-level module and a prediction module. In the low-level module, two structures (local and distant) are responsible for extracting different features.
caption: Figure 2: The architecture of a self-attention module. This module mainly contains Scaled-Dot Product Attention, which requires three inputs: Key, Query and Value. Those inputs are generated from the same input sequence but projected by different linear transformations.
caption: Figure 3: Two auxiliary predictions (~pt,local and ~pt,distant) are obtained from the local and global structures in the low-level module. The primary prediction ~pt,primary is obtained from the virtual logit vector ~gt
caption: Figure 4: Words are dropped (denoted as 〈REMOV ED〉) randomly. Those positions are used to calculate LossCV T and update the auxiliary prediction modules to improve the model.
caption: Table 1: The number of passages and vocabulary words in each dataset. The labeled and unlabeled data are separately counted and shown in the rows.
caption: Table 3: The result of Thai sentence segmentation for each model. For the Orchid dataset, we report the average of each metric on 10-fold cross-validation. Meanwhile, average metrics from 5-fold cross-validation are shown for the UGWC dataset.
caption: Figure 5: An example of sentence boundary prediction by a normal Bi-LSTM-CRF and by the model with local representation (+local). Here, sb indicates that the word is predicted as the sentence boundary.
caption: Table 4: The result of English punctuation restoration by each model. Each model is evaluated by the F1 score of each class of punctuation, the overall F1 score and the 2-class F1 score.
caption: Figure 6: An example of punctuation prediction of the model with distant representation (+ local + distant) and without distant representation (+ local). The "COMMA" indicates that the word is followed by a comma (,) and "PERIOD" indicates that a period (.) is restored at that position.
caption: Table 5: Model hyperparameters for each dataset
caption: Table 8: The improvement of each contribution on the UGWC dataset results shown as p-values from paired t-tests