# Enhance Text-to-Text Transfer Transformer with Generated Questions for Thai Question Answering
## 3 Proposed Method
### Model Evaluation  
We used the F1 score and Exact Match (EM), which are widely used in span extraction Question Answering tasks [19] to evaluate the performance of models. The Exact Match measures how much the model is able to retrieve the exact ground truth span correctly in the whole dataset. The F1 score is a harmonic mean of precision and recall of prediction compared to the ground truth. Originally, to measure the precision and the recall, we count the number of words found in both the prediction and the ground truth.  
To calculate the F1 score, the equations below were used. \(TP\) refers to 'True Positive', that counts the tokens appearing in both the prediction and the ground truth. \(FP\) refers to 'False Positive' that counts the tokens that appear only in the prediction. \(FN\) refers to 'False Negative', which means the number of the tokens that appear only in the ground truth. The F1 score of the dataset is an average of the F1 score of every sample.  
\[Precision=\frac{TP}{TP+FP} \tag{1}\]  
\[Recall=\frac{TP}{TP+FN} \tag{2}\]  
\[F1=\frac{2\times Precision\times Recall}{Precision+Recall} \tag{3}\]  
\[F1=\frac{\sum_{i=1}^{N}F1_{i}}{N} \tag{4}\]  
Due to the imperfections of the Thai word tokenizer, measuring at the word-level might not be sufficient. In English, there are space separators between words that make English easier to be tokenized into words. On the other hand, the Thai language is more ambiguous as there is no space between words. Thus, the Word-level F1 depends on the quality of the tokenizer used. To overcome this, we also calculated the F1 score at the syllable-level.  
The Syllable-level F1 score, the F1 score that calculates based on syllable tokens, is a more appropriate metric than the Word-level F1 score for a language ambiguous to segment because of the following reasons. First, due to the quality of word tokenizers, using different word tokenizers may result in different F1 scores and cause the score to be unable to be compared with other works. Secondly, because of the imperfection of word tokenizers, there are still mistakes when segmenting some similar words. Lastly, due to the ambiguity of the Thai language, some Thai words can be tokenized in many ways, especially the proper nouns. In contrast, using syllables to calculate scores is less ambiguous because there is only a way to segment a word into syllables that maintains a unit of pronunciation.  
In this experiment, we used the 'newmm' tokenizer [18] that is currently one of the fastest and the most reliable word tokenizers for the Thai language. However, as shown in the example in Table 2, the 'newmm' tokenizer could not tokenize the word into a  
Figure 9: Illustration of the training flow of the Merge Strategy.  
proper form. To address this problem, we evaluated the predictions with Syllable-level F1 along with Word-level F1. Using syllable tokens to calculate the F1 score could obtain a more accurate score to the linguistic word segmentation than word tokens because the syllable tokenizer can extract overlapping words into pieces of syllables while the word tokenizer cannot.  
Similar to the English language, Thai words can have one or more syllables. Tokenizing a word into syllables means dividing the word by a unit of pronunciation that has one vowel sound. For example, in Table 2, the word \"_\"J@7@7@\"_\" (Malay language in the short term) can be pronounced as /malaju:/ which has three syllables as \"_\"N / #7 / %\"_; each piece can be pronounced as /ma/, /la:/ and /ju:/ respectively. Another example is the word \"_\"_\"_\" (language), which can be pronounced as /pa:sa:/. This word has two syllables as \"_\"_\" / #7\"_; each piece can be pronounced as /pa:/ and /sa:/ sequentially.