# Word Sense Disambiguation Using Clustered Sense Labels
## 4 Experiment and Result
### Experiment Result  
For evaluating the English data, we followed the evaluation methods used in Semeval, where the SemCor corpus is used for training, and the Senseval and Semeval corpus are used for testing. For a fair comparison with the English data, the Korean Sejong sense annotated corpus was randomly sampled to be the same size as the English data and named the Small set. To see the effect of corpus size, we sampled the Medium and Large sets, respectively, to be twice and thrice the size of the Korean Small set. All the experiments for the Korean data were 3-fold cross validated. The results are shown in Table 5. Generally, the score increases as the threshold value increases to a certain point, then decreases thereafter. Figure 4 shows the performance changes at each threshold point. The English data showed the best performance with a 69.1% average at threshold point 0.5 with an 89% compression ratio. The Korean data produced a 97.3% average at point 0.3 with a 33% compression ratio. The performances are much better than the baseline at threshold 0.0, which does not use a compressed sense vocabulary. Note that the baseline uses the first senses as default labels for the untrained \begin{table}
\begin{tabular}{l l} \hline \hline  & **(a)**\(k\)-means \\ \hline init & \(k\)-means++ \\ max iter & 300 \\ tol & \(1\times 10^{-4}\) \\ dist & Euclidean \\ \hline \hline  & **(b) affinity propagation** \\ \hline damping & 0.7 \\ max iter & 50,000 \\ converg iter & 15 \\ dist & Euclidean \\ \hline \hline  & **(c) BERT-WSD model** \\ \hline learning rate & \(1\times 10^{-4}\) \\ embed size & 768 \\ dropout rate & 0.1 \\ trans layer num & 6 \\ trans layer dim & 2048 \\ atten head num & 8 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: System hyper-parameter values. senses as done in [18]. The average performance increased from 62.9% to 69.1% on English data and from 95.9% to 97.3% on Korean data. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & **Threshold** & **0.0** & **0.1** & **0.3** & **0.5** & **0.7** & **0.9** & **1.0** \\ \hline \multirow{3}{*}{Eng} & Senseval2 & 66.3 & 70.9 & **71.1** & 70.6 & 70.6 & 69.8 & 67.4 \\ \cline{2-8}  & Senseval3 & 65.6 & 71.2 & **72.7** & 71.6 & 71.4 & 72.5 & 66.8 \\ \cline{2-8}  & Senseval07 & 56.0 & 62.2 & **63.3** & 62.9 & 63.1 & 61.8 & 54.7 \\ \cline{2-8}  & Senseval13 & 64.2 & 71.2 & 71.0 & **71.8** & 71.5 & 71.7 & 70.3 \\ \cline{2-8}  & Senseval15 & 62.2 & 66.5 & 66.8 & **68.1** & 67.3 & 66.6 & 64.7 \\ \cline{2-8}  & Avg. & 62.9 \(\pm\) 3.4 & 68.4 \(\pm\) 3.3 & 69.0 \(\pm\) 3.1 & **69.1**\(\pm\) 3.0 & 68.8 \(\pm\) 2.9 & 68.5 \(\pm\) 3.6 & 64.8 \(\pm\) 4.9 \\ \hline \multirow{3}{*}{Kor} & Small & 95.7 \(\pm\) 0.2 & **97.2**\(\pm\) 0.1 & 97.1 \(\pm\) 0.1 & 96.7 \(\pm\) 0.0 & 96.1 \(\pm\) 0.1 & 95.4 \(\pm\) 0.2 & 94.1 \(\pm\) 0.0 \\ \cline{2-8}  & Medium & 96.0 \(\pm\) 0.1 & 97.0 \(\pm\) 0.1 & **97.3**\(\pm\) 0.0 & 96.9 \(\pm\) 0.1 & 96.5 \(\pm\) 0.1 & 95.9 \(\pm\) 0.1 & 95.0 \(\pm\) 0.1 \\ \cline{2-8}  & Large & 96.1 \(\pm\) 0.1 & 97.1 \(\pm\) 0.1 & **97.5**\(\pm\) 0.0 & 96.9 \(\pm\) 0.2 & 96.8 \(\pm\) 0.1 & 96.1 \(\pm\) 0.0 & 95.1 \(\pm\) 0.1 \\ \cline{2-8}  & Avg. & 95.9 \(\pm\) 0.1 & 97.1 \(\pm\) 0.1 & **97.3**\(\pm\) 0.1 & 96.8 \(\pm\) 0.1 & 96.5 \(\pm\) 0.2 & 95.8 \(\pm\) 0.3 & 94.7 \(\pm\) 0.4 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: WSD F1-score with standard deviation of various thresholds and datasets. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**POS** & **Noun** & **Verb** & **Adj** & **Adv** & **Total** \\ \hline Eng & 146,312 & 25,047 & 30,002 & 5580 & 206,941 \\ Kor & 91,457 & 16,381 & 4317 & 2667 & 114,822 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Number of senses in the dictionaries: WordNet 3.0 and Standard Korean Language Dictionary.