Question: What are the three modules of the model the author uses?
Evidence: In this work, the model predicts the tags INLINEFORM0 for the tokens in a word sequence INLINEFORM1 where INLINEFORM2 is the sequence size and INLINEFORM3 , INLINEFORM4 denote the token and its tag at timestep INLINEFORM5 , respectively. Each token INLINEFORM6 consists of a word, its POS and its type. There are five defined word types: English, Thai, punctuation, digits, and spaces.
The tag set INLINEFORM0 is populated based on the considered task. In Thai sentence segmentation, the assigned tags are INLINEFORM1 and INLINEFORM2 ; INLINEFORM3 denotes that the corresponding word is a sentence boundary considered as the beginning of a sentence, while and INLINEFORM4 denotes that the word is not a sentence boundary. Meanwhile, there are four tags in the punctuation restoration task. Words not followed by any punctuation are tagged with INLINEFORM5 . Words that are followed by a period “.”, comma “,” or question mark “?” are tagged to INLINEFORM6 , INLINEFORM7 , and INLINEFORM8 , respectively.
architecture/MainArchitecture
Our model architecture is based on Bi-LSTM-CRF, as shown in Fig. . The model is divided into three modules. The first, low-level module, consists of two separate structures: local and distant structures. The second, high-level module, contains a sequence of stacked bidirectional LSTM and self-attention layers. The final module, the prediction module, is responsible for predicting the tags INLINEFORM0 . Each module is described more completely in the next three subsections.
A sequence of word tokens is input into the low-level module. The input tokens pass through two structures. The first structure generates a sequence of local representation vectors INLINEFORM0 , and the second structure generates low-level distant representation vectors INLINEFORM1 . After obtaining both sequences of representation vectors, the local representation vectors are fed to the Bi-LSTM to obtain the recurrent representation vectors INLINEFORM2 , as shown in recurrentrep. Then, the recurrent and distant representation vectors are concatenated to form the low-level representation vector INLINEFORM3 , as shown in low-level: DISPLAYFORM0 DISPLAYFORM1 
This structure is shown as the left submodule of the low-level module in Fig. . It extracts the local representation vectors INLINEFORM0 . Its input tokens are used to create n-gram tokens, which are unigrams INLINEFORM1 , bigrams INLINEFORM2 , and trigrams INLINEFORM3 . Each n-gram token is represented as an embedding vector, which is classified as a unigram embedding vector INLINEFORM4 , a bigram embedding vector INLINEFORM5 or a trigram embedding vector INLINEFORM6 . Each vector INLINEFORM7 is mapped from a token by gram embedding INLINEFORM8 , which is a concatenated vector of the word embedding INLINEFORM9 , POS embedding INLINEFORM10 and type embedding INLINEFORM11 , as shown in eq:embedding: DISPLAYFORM0 
Each n-gram token at timestep INLINEFORM0 is generated by the previous, present and next token and embedded into vectors as shown in uni,bi,tri. The unigram embedding at timestep INLINEFORM1 is a unigram embedding of the current token INLINEFORM2 . The bigram embedding vector at timestep INLINEFORM3 is a bigram embedding of the previous and present tokens INLINEFORM4 , and the trigram embedding vector at timestep INLINEFORM5 is a trigram embedding of the previous, present and next tokens INLINEFORM6 : DISPLAYFORM0 DISPLAYFORM1 
At each timestep INLINEFORM0 , a local representation vector INLINEFORM1 is combined from the n-gram embedding vectors generated from the context around INLINEFORM2 . A combination of embedding vectors, which is used to construct a local representation vector, is shown in n-gramcombination. A combination consists of the unigram, bigram, and trigram embedding vectors at timesteps INLINEFORM3 , INLINEFORM4 and INLINEFORM5 and it is a concatenation of all the embedding vectors: DISPLAYFORM0 
The distant structure, which is a self-attention module, is shown in Fig. on the right side of the low-level module. The structure extracts low-level distant representation vectors INLINEFORM0 from a sequence of unigram embedding vectors INLINEFORM1 , as shown in distantattention. In this case, the self-attention module is a scaled dot-product attention BIBREF11 , where key, query, and value vectors are the linear projections of the unigram embedding vectors shown in Fig. . The linear transformations for key, query, and value are learned separately and updated in the model through backpropagation. The output vector, which is the scaled dot-product attention at each timestep, is concatenated with the input vector INLINEFORM2 and projected by a linear transformation. That projected vector is the output vector of a self-attention module, which is a low-level distant representation vector.
architecture/selfattention DISPLAYFORM0 
The low-level representation vectors INLINEFORM0 are used as the input for this module, which outputs the high-level representation vectors INLINEFORM1 whose calculation is shown in high-level. The high-level module, as shown in Fig. , is composed of a stacked bidirectional LSTM and a self-attention modules. A stacked bidirectional LSTM contains K layers of bidirectional LSTMs in which the output from the previous bidirectional LSTM layer is the input of the next bidirectional LSTM layer. The self-attention part of this structure is the same as that in the low-level distant structure. The self-attention module helps to generate the high-level distant representation vectors that are output by the high-level module. DISPLAYFORM0 
The prediction module is the last module. It includes two layers: a fully connected layer and a CRF layer. In the fully connected layer, the output vectors from the high-level module are projected by a linear transformation as shown in virtual. The purpose of this layer is to create the virtual logit vectors INLINEFORM0 , which represent the probability distribution for CVT, as discussed in Section SECREF21 . Therefore, the number of dimensions of logits equals the number of possible tags in each task: DISPLAYFORM0 
The CRF layer is responsible for predicting the tag INLINEFORM0 of a token at each timestep, as shown in crf. The layer receives a sequence of virtual logit vectors ( INLINEFORM1 ) as input and then decodes them to a sequence of tags INLINEFORM2 using the Viterbi algorithm. DISPLAYFORM0 
The model is divided into three modules. The first, low-level module, consists of two separate structures: local and distant structures. The second, high-level module, contains a sequence of stacked bidirectional LSTM and self-attention layers. The final module, the prediction module, is responsible for predicting the tags ~
Answer: Low-level module,high-level module and the prediction module.