# Word Sense Disambiguation Using Clustered Sense Labels
## 5 Discussion  
**Training size effect:** We tested Korean training datasets of three different sizes. As the size increased, the performance increased both in the uncompressed vocabulary (threshold 0.0) and in the compressed vocabulary. Table 5 shows the change in performance on the Small, Medium, and Large datasets: 95.7%, 96.0%, and 96.1% for the uncompressed vocabulary at threshold 0.0, and 97.2%, 97.3%, and 97.5% for the compressed vocabulary. This is because more training data increases the performance by decreasing the quantity of unseen data on testing. Table 6 shows that the percentage of untrained data decreases from 3.3% to 1.8% and 1.2% as the training data size increases. This confirms that more training data are also useful for the compressed vocabulary. **Performance difference between English and Korean data:** The performance on Korean data is much higher than on the English in the highest average F1 value: 97.3% versus 69.1%. We conjecture that this difference is caused by the following:  
1. The ambiguity of English is higher than Korean. It ranges from 6.9 to 10.9 in English and from 5.2 to 5.3 in Korean (shown in Table 3). This is because English senses are categorized more specifically than Korean, as shown in Table 2; 206 K versus 114 K senses. This means that English sense labeling prediction should be more precise than Korean, which makes the performance gain relatively difficult. 2. The configuration of test corpus is different. English test corpus used the data of different domains: SemCor for training, Senseval and Semeval for testing, whereas Korean used the shuffled data of the same corpus (Sejong) for 3-fold cross validation testing. It is verified by Table 6, showing that the ratio of untrained data in English test data is much higher than Korean, 16.3% versus 1.2% to 3.3%. This fact benefits the performance of Korean data. 3. For Korean BERT input, normalized words are used, whereas raw words are used for English input. English language is inflectional and does not have many variations in word form, but Korean language is agglutinative and has many variations in word ending [34; 35]. \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Data** & **Eng All** & **Kor Small** & **Kor Medium** & **Kor Large** \\ \hline Trained & 6072 (83.7\%) & 3972 (96.7\%) & 7674 (98.2\%) & 11,918 (98.8\%) \\ Untrained & 1181 (16.3\%) & 136 (3.3\%) & 142 (1.8\%) & 142 (1.2\%) \\ Total & 7253 (100.0\%) & 4108 (100.0\%) & 7816 (100.0\%) & 12,060 (100.0\%) \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Number of sense vocabularies in the trained and untrained test data. Figure 4: F1 performance at various threshold values of English Avg., Korean Small, Medium, and Large. Note that Korean performance and English performance are scaled differently on the left-hand and right-hand sides of the Y axis. Because these productive variations of Korean language degrade the performance of the language model, some Korean language models are trained with preprocessing the input words to avoid the degradation. We used KorBERT-morp model for Korean BERT, which is the Korean language model using normalized input words through the preprocessing of morphological analysis and part of speech tagging. We conjecture that this normalized word with part of speech tag rather benefits the performance of Korean dataset, because it gives more information to the BERT model than a simple word. **Performance increase ratio:** The performance of English data increased by 6.2 percentage points, from 62.9% to 69.1%, whereas that of Korean data increased by 1.4 percentage points, changing from 95.9% to 97.3%. There is a much higher performance improvement on English data. This is because the baseline performance for English data is significantly lower than for Korean data, which allows greater improvements more easily than a high baseline performance. **Compression level:** Table 7 shows the comparison of (a) the model with a thesaurus and (b) the model without a thesaurus.