# DAWE: A Double Attention-Based Word Embedding Model with Sememe Structure Information
## 5 Experiments and Results
### Word Analogy  
In this section, we examine the quality of word embeddings by the performance of the models in the word analogy reasoning task. In the Chinese word analogy reasoning task, each analogy sample consists of two-word pairs \((\mathrm{w}_{1},\mathrm{w}_{2})\) and \((\mathrm{w}_{3},\mathrm{w}_{4})\), which satisfy: \(\mathrm{w}_{2}-\mathrm{w}_{1}\cong\mathrm{w}_{4}-\mathrm{w}_{3}\), ie \(\mathrm{w}_{2}-\mathrm{w}_{1}+\mathrm{w}_{3}\cong\mathrm{w}_{4}\).Therefore, in the word analogy reasoning task, the score of the candidate word is calculated by replacing \(\mathrm{w}_{4}\) with the candidate word \(\mathrm{w}\) and by the following formula:  
\[\mathrm{S}_{\mathrm{A}}(\mathrm{w})=\cos(\mathrm{w}_{2}-\mathrm{w}_{1}+\mathrm{ w}_{3},\mathrm{w}). \tag{16}\]  
After obtaining the ranking of all candidate words, the experiment chose top-ranked words and evaluated the performance of the model by calculating accuracy and mean rank metrics. The higher is the accuracy and the lower is the mean rank, the better is the model. The results of the word analogy reasoning task are shown in Table 4. From the evaluation results of the word analog task, we can conclude that:  
(1) In the word analogy reasoning task, our models are significantly better than the previous models. The accuracy of DAC is 2% higher than that of SAC, and the accuracy of DAT is 3% higher than that of SAT. DAC has increased more than 4% compared to the SAC model and DAT has increased more than 3% compared to the SAT model of mean rank. The experimental results show that both DAC and DAT are more conducive to the accurate description of senses by distinguishing the internal sememes. (2) Our models perform well in the class of Capital, which is the collection of groups of capital and country around the world. Most of the words of the capital names have distinct meanings in various contexts, such as the word \"Washington\" may be the name of a capital city, a state, a university, a hotel, or a people. In the training process, the proposed model can dynamic adjustment the weights of both senses and sememes by the \"double-attention\" mechanism, hence offering more powerful ability on the embedding of those words. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{4}{c}{**Accuracy**} & \multicolumn{4}{c}{**Mean Rank**} \\ \cline{2-9}  & **Capital** & **City** & **Relationship** & **All** & **Capital** & **City** & **Relationship** & **All** \\  & **677** & **175** & **272** & **1124** & **677** & **175** & **272** & **1124** \\ \hline CBOW & 45.05 & 86.85 & 84.19 & 61.03 & 60.28 & 1.43 & 41.87 & 46.66 \\ GloVe & 62.03 & 83.42 & 82.35 & 70.28 & 17.09 & 1.77 & 14.28 & 14.02 \\ Skip-gram & 60.26 & 96.00 & 77.57 & 70.01 & 78.67 & 1.05 & 2.98 & 48.27 \\ SSA & 72.67 & 80.00 & 74.63 & 74.28 & 21.05 & 7.21 & 2.74 & 14.45 \\ SAC & 66.24 & 92.28 & 71.87 & 71.66 & 40.86 & 5.74 & 2.56 & 13.51 \\ SAT & 71.64 & 87.14 & 74.44 & 74.73 & 14.79 & 2.07 & 2.34 & 9.80 \\ \hline DAC & 68.53 & 93.14 & 72.24 & 73.26 & 14.10 & 1.15 & 2.74 & 9.34 \\ DAT & 74.00 & 91.42 & 75.36 & 77.04 & 8.87 & 1.71 & 2.58 & 6.23 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Evaluation results of word analogy task. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **Wordsim-240** & **Wordsim-297** \\ \hline CBOW & 57.987 & 62.063 \\ GloVe & 57.618 & 57.107 \\ Skip-gram & 55.279 & 60.565 \\ SSA & 60.410 & 60.167 \\ SAC & 57.574 & 57.825 \\ SAT & 60.480 & 62.280 \\ \hline DAC & 57.157 & 59.671 \\ DAT & 61.162 & 63.327 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Evaluation results of word similarity tasks. (3) Although the performances of our models are not the best in the classes of City and Relationship, our models are more robust in the overall performance of accuracy and mean rank. (4) DAWE models are significantly improved in the performance of the word analogy reasoning task, but only a small increase in performance in the word similarity task.