# Capsule Network Improved Multi-Head Attention for Word Sense Disambiguation
## 4 Experiments
### Results  
#### 4.3.1 Overall Results  
Table 2 reports the F1 scores of our model and compares against previous different types of methods. We group systems by method type. * **Knowledge-based systems**: The first four systems are knowledge-based methods, among which, MFS and WordNet S1 are two strong knowledge-based baselines. They select the most frequent sense (MFS) in the training dataset and in WordNet, respectively. Lesk+ext,emb [8] is an extended version of Lesk algorithm, which calculates the definition-context overlap to measure semantic similarity. Babelfy [42] builds a unified graph-based architecture that exploits BabelNet as the semantic network. WSD-TM [43] leverage the formalism of topic model to design a WSD system. * **Traditional supervised systems**: IMS [10] and IMS+emb [11] are two traditional word expert supervised methods training an SVM classifier for WSD. The latter explores different approaches to incorporate word embeddings as features on the basis of the former using local features. The results show that word embeddings provide significant improvement. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & **Dev** & \multicolumn{4}{c}{**Test**} & \multicolumn{4}{c}{**Concatenation**} \\ \hline
**System** & **SE07** & **SE2** & **SE3** & **SE13** & **SE15** & **Noun** & **Verb** & **Adj** & **Adv** & **All** \\ \hline Knowledge-based & & & & & & & & & & \\ MFS baseline & 54.5 & 65.6 & 66.0 & 63.8 & 67.1 & 67.7 & 49.8 & 73.1 & 80.5 & 65.5 \\ WordNet S1 & 55.2 & 66.8 & 66.2 & 63.0 & 67.8 & 67.6 & 50.3 & 74.3 & 80.9 & 65.2 \\ Lesk+ext,emb & 56.7 & 63.0 & 63.7 & 66.2 & 64.6 & 70.0 & 51.1 & 51.7 & 80.6 & 64.2 \\ Babelfy & 51.6 & 67.0 & 63.5 & 66.4 & 70.3 & 68.9 & 50.7 & 73.2 & 80.5 & 65.5 \\ WSD-TM & 55.6 & 69.0 & 66.9 & 65.3 & 69.6 & 69.7 & 51.2 & 76.0 & 80.9 & 66.9 \\ \hline Traditional & & & & & & & & & & \\ Supervised & & & & & & & & & & \\ IMS & 61.3 & 70.9 & 69.3 & 65.3 & 69.5 & 70.5 & 55.8 & 75.6 & 82.9 & 68.9 \\ IMS+emb & 62.6 & 72.2 & 70.4 & 65.9 & 71.5 & 71.9 & 56.6 & 75.9 & 84.7 & 70.1 \\ \hline Neural-based & & & & & & & & & & \\ Bi- & 64.8 & 72.0 & 69.1 & 66.9 & 71.5 & 71.5 & 57.5 & 75.0 & 83.8 & 69.9 \\ LSTM+att,LEX,POS & & & & & & & & & & \\ GAS\({}_{\text{ext}}\)(concatenation) & - & 72.2 & 70.5 & 67.2 & 72.6 & 72.2 & 57.7 & 76.6 & 85.0 & 70.6 \\ EWISE & 67.3 & 73.8 & 71.1 & 69.4 & 74.5 & 74.0 & 60.2 & 78.0 & 82.1 & 71.8 \\ LMMS\({}_{2348}\)(BERT) & 68.1 & 76.3 & 75.6 & 75.1 & 77.0 & - & - & - & - & 75.4 \\ GlossBERT & 72.5 & 77.7 & 75.2 & 76.1 & 80.4 & 79.8 & 67.1 & 79.6 & 87.4 & 77.0 \\ CapsDecE2S\({}_{\text{large}}\) & 68.7 & 78.9 & 77.4 & 75.6 & 77.1 & - & - & - & - & 76.9 \\ CapsDecE2S\({}_{\text{large}}\)+LMMS & 73.8 & 78.8 & **80.7** & 76.6 & 79.4 & - & - & - & - & 78.6 \\ \hline
**Ours** & **75.2** & **79.6** & 78.4 & **79.9** & **81.9** & **81.7** & **69.5** & **83.7** & **88.2** & **79.5** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Reports of F1-score (%) on all-words word sense disambiguation (WSD) task, including SE07 (Dev), SE2, SE3, SE13, SE15, and ALL (concatenation of four test datasets) as well as every part-of-speech type (Noun, Verb, Adj, and Adv). Knowledge-based, traditional supervised, and neural-based methods and at last our method are listed. The best results are marked in bold and underlined numbers denotes previous state-of-the-art results. * **Neural-based systems**: We list several recent neural-based methods. Bi-LSTM+att,LEX,POS [17] converts WSD to a sequence learning task. GASext (concatenation) [19] jointly encodes the context and glosses of the target word and extending gloss knowledge. EWISE [20] uses BiLSTM to train the context encoder and knowledge graph embedding to train the definition encoder. LMMS\({}_{2348}\) (BERT) [21] focuses on making full use of WordNet knowledge to create sense-level embeddings. GlossBERT [16] constructs context-gloss pairs, thus treating WSD task as a sentence-pair classification problem to fine-tune the pre-trained BERT model.