# A Benchmark for Morphological Segmentation in Uyghur and Kazakh
## 2 Related Work
### Result Analysis
#### 4.6.1 OOV and IV Analysis  
Discovering new words is a core challenge in the field of NLP. Out-of-vocabulary (OOV) or out-of-set words refer to new words encountered in the validation or test set. The OOV recall rate in this paper can be understood as evaluating the model's ability to identify new morphologies correctly. In-vocabulary (IV) words, also known as in-set words, usually refer to words encountered in the validation or test set that have appeared in the training set. The IV recall rate in this paper can be understood as evaluating the model's ability to identify morphologies that appear in the dictionary correctly. In this paper, the morphologies that appear in the training set are used as the dictionary for evaluation. Table 7 shows the out-of-vocabulary rate of the validation and test sets, and Table 8 shows the situation of different models in identifying OOV and IV. Table 8 shows that, among the models based on unsupervised learning, MMSeg has a higher ability to recall out-of-vocabulary and in-vocabulary words than other methods. On the Uyghur test set, the OOV and IV rates reach 46.69% and 50.55%, respectively; on the Kazakh test set, the OOV and IV rates reach 8.52% and 48.11%, respectively. Although the OOV rate in the Kazakh dataset is lower than that in the Uyghur, the ability of unsupervised models to recall OOV words in Kazakh is severely lower than that in Uyghur. Through experimental analysis, it is found that this problem is caused by the following two reasons: (1) In the unsupervised learning models, due to the introduction of the n-gram correlation assumption, the morpheme length after model training and prediction will be much smaller than N. The relatively long morphemes in the dataset are stems. In the training set, the average length of stems in Uyghur is 6.3, and the average length of stems in Kazakh is 5.9; in the test set, the average length of stems in Uyghur is 6.3, and the average length of stems in Kazakh is 6.1. The average stem length of Kazakh has changed significantly, with longer stems in the test set. Therefore, the model's segmentation granularity will be finer during \begin{table}
\begin{tabular}{l c c} \hline \hline
**Lang.** & **Uyghur** & **Kazakh** \\ \hline MMSeg & **47.55** & **35.22** \\ w/o MAM & 42.53 & 33.67 \\ \hline FEMSeg-CRF & **92.69** & **92.84** \\ w/o CRF(FEMSeg) & 92.04 & 92.34 \\ w/o Transformer-encoder & 90.89 & 90.56 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Experimental results of ablation studies (%). Bold represents the best result in the same group of experiments. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Lang.** & **Uyghur** & **Kazakh** \\ \hline Validation & 13.44\% & 7.27\% \\ Test & 14.62\% & 8.60\% \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Out-of-vocabulary (OOV) rate in datasets (%). testing, breaking the stems' integrity. (2) In Kazakh, morphological ambiguity is a common situation. Therefore, the ambiguity problem is also a reason for the low OOV recall rate. In the models based on supervised learning, FEMSeg or FEMSeg-CRF has higher or nearly the best OOV and IV recall rates compared to other models. FEMSeg achieved OOV and IV recall rates of 87.67% and 93.93%, respectively, on the Uyghur test set, which is close to the highest recall rate (BiLSTM: 88.91%); on the Kazakh test set, the OOV and IV recall rates reached 81.51% and 93.57%, respectively. FEMSeg-CRF has achieved OOV and IV recall rates of 87.37% and 94.06%, respectively, on the Uyghur test set, and OOV and IV recall rates of 78.83% and 94.39%, respectively, on the Kazakh test set. It is worth noting that the FEMSeg has achieved higher out-of-vocabulary (OOV) and in-vocabulary (IV) recall rates compared to BiLSTM and BiGRU on both datasets, demonstrating stronger OOV recognition capabilities on the Kazakh dataset. Similarly, the FEMSeg-CRF model also achieved higher OOV recall rates on the Kazakh dataset. There is also a situation where the OOV recall rate is relatively low in all models.