# A Benchmark for Morphological Segmentation in Uyghur and Kazakh
## 2 Related Work
### Result Analysis
#### 4.6.2 Average Edit Distance and Morphological Error Rate Analysis  
The above analyses evaluated the model's predictive ability from the morphological perspective. Next, this paper evaluates the model's ability from the word perspective. The average edit distance evaluates the difference between the model's segmentation results and the ground truth from the character level. This paper uses Levenshtein distance to calculate the similarity between morphemes, mainly by calculating the distance between two sequences through morphemes' insertion, deletion, and substitution operations. Figure 5 shows the average edit distance of different models on the Uyghur and Kazakh test sets. The shorter the edit distance, the more similar the morphemes are. In unsupervised learning, MMSeg obtains a lower score on the Uyghur, reducing it by 0.42 compared to the Morfessor model and 0.97 compared to the BPE model, but it is insignificant in Kazakh. This phenomenon also appears in the evaluation of morphological error rate. This paper posits that the reason is related to the n-gram assumption of the unsupervised model, which will be analyzed with examples in the case study. In supervised learning, the BiLSTM-ART-CRF and FEMSeg-CRF achieve the smallest edit distance of 0.12 on the Uyghur, and FEMSeg-CRF achieves the smallest edit distance of 0.14 on the Kazakh. The morphological error rate evaluates the model's ability to correctly predict all morphemes that appear in words from the word perspective. Figure 6 shows the morphological error rates of different models on Uyghur and Kazakh validation and test sets. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multicolumn{1}{c}{**Lang. **} & \multicolumn{4}{c}{**Uyghur**} & \multicolumn{4}{c}{**Kazakh**} \\ \hline \multirow{3}{*}{**Models**} & \multicolumn{2}{c}{**Validation**} & \multicolumn{2}{c}{**Test**} & \multicolumn{2}{c}{**Validation**} & \multicolumn{2}{c}{**Test**} \\ \cline{2-9}  & **OOV** & **IV** & **OOV** & **IV** & **OOV** & **IV** & **OOV** & **IV** \\ \hline Morfessor & 5.88 & 39.10 & 4.78 & 40.37 & 2.33 & 31.36 & 1.95 & 31.33 \\ BPE & 1.80 & 19.16 & 1.70 & 21.98 & 2.04 & 28.35 & 3.65 & 25.46 \\ MMSeg & **45.75** & **50.04** & **46.69** & **50.55** & **6.98** & **46.24** & **8.52** & **48.11** \\ \hline BiLSTM & **83.82** & 91.58 & **88.91** & 92.03 & 64.54 & **93.25** & 70.56 & 93.54 \\ BiGRU & 80.88 & 92.59 & 85.52 & 92.93 & 63.15 & 92.46 & 67.40 & 92.38 \\ FEMSeg & 82.35 & **93.33** & 87.67 & **93.93** & **81.98** & 92.87 & **81.51** & **93.57** \\ \hline HMM & 29.41 & 50.42 & 31.43 & 50.45 & 17.73 & 48.04 & 23.84 & 49.26 \\Figure 6 shows that, when evaluating the morphological segmentation error rate of different models, the error rates of unsupervised learning models and HMM models are relatively high. Among the unsupervised learning models, MMSeg obtains a relatively low error rate in the Uyghur, whereas the error rate is high in the Kazakh. Combining the experimental results in Table 4, it can be found that on the Uyghur, MMSeg has certain advantages, whether from the morphological or word perspective. However, in Kazakh, although MMSeg can predict more correct morphemes from the morphological perspective, the morphemes predicted by the model are not higher than the Morfessor or BPE model in terms of word or morphological integrity. In supervised learning, regardless of whether a CRF layer is added to the model, FEMSeg has a lower error rate than other models in the same group. The error rate of FEMSeg-CRF is reduced by 1.53% and 0.90% on the Uyghur and Kazakh test sets, respectively. The error rate of FEMSeg is reduced by 0.53% and 2.10% on the Uyghur and Kazakh test datasets compared to the BiLSTM model. The error rate of FEMSeg-CRF is reduced by 0.63% and 2.35% on the Uyghur and Kazakh test datasets compared to the BiLSTM-Att-CRF model. Compared to Uyghur, the error rate reduction of FEMSeg-CRF is relatively large compared to other methods in Kazakh.