# DAWE: A Double Attention-Based Word Embedding Model with Sememe Structure Information
## 4 Methodology
### Double Attention over Context Model  
As shown in Figure 6, DAC consists of two parts: encoding part and training part, which correspond to the DAWE encoding framework and Skip-gram training framework respectively.  
For each context word \(\text{w}_{\text{c}}\in\text{C}(\text{w})\) (\(\text{C}(\text{w})=[\text{w}_{\text{t-k}},\dots,\text{w}_{\text{t-1}},\text{ w}_{\text{t+1}},\dots,\text{w}_{\text{t+k}}]\), where k is the size of the context window), we have:  
\[\text{w}_{\text{c}}=\sum_{\text{i}=1}^{|\text{S}^{(\text{w}_{\text{c}})}|} \text{att}\Big{(}\hat{\text{s}}_{\text{i}}^{(\text{w}_{\text{c}})},\text{w} _{\text{t}}\Big{)}\hat{\text{s}}_{\text{i}}^{(\text{w}_{\text{c}})}, \tag{8}\]  
where \(\text{att}\Big{(}\hat{\text{s}}_{\text{i}}^{(\text{w}_{\text{c}})},\text{w} _{\text{t}}\Big{)}\) denotes that the target word \(\text{w}_{\text{t}}\) is used as attention to calculating the weight of the ith sense of the context word \(\text{w}_{\text{c}}\), as follows:  
\[\text{att}\Big{(}\hat{\text{s}}_{\text{i}}^{(\text{w}_{\text{c}})},\text{w} _{\text{t}}\Big{)}=\frac{\exp\Big{(}\hat{\text{s}}_{\text{i}}^{(\text{w}_{ \text{c}})}\cdot\text{w}_{\text{t}}\Big{)}}{\sum_{\text{j}=1}^{|\text{w}_{ \text{c}}|}\exp\Big{(}\hat{\text{s}}_{\text{j}}^{(\text{w}_{\text{c}})}\cdot \text{w}_{\text{t}}\Big{)}}, \tag{9}\]  
where \(\hat{\text{s}}_{\text{i}}^{(\text{w}_{\text{c}})}\) denotes the value used in the calculation of weight, which is obtained by the sememe embeddings of the sememe set \(\text{X}_{\text{i}}^{(\text{w}_{\text{c}})}\) corresponding to the sense \(\text{s}_{\text{i}}^{(\text{w}_{\text{c}})}\). It can be formalized by the following:  
\[\hat{\text{s}}_{\text{i}}^{(\text{w}_{\text{c}})}=\sum_{\text{j}=1}^{|\text{X} _{\text{i}}^{(\text{w}_{\text{c}})}|}\text{att}\Big{(}\hat{\text{x}}_{\text{j }}^{(\text{s}_{\text{i}})},\text{w}_{\text{t}}\Big{)}\hat{\text{x}}_{\text{j}}^ {(\text{s}_{\text{i}})}. \tag{10}\]  
Similar to \(\text{att}\Big{(}\hat{\text{s}}_{\text{i}}^{(\text{w}_{\text{c}})},\text{w} _{\text{t}}\Big{)}\), \(\text{att}\Big{(}\hat{\text{x}}_{\text{j}}^{(\text{s}_{\text{i}})},\text{w}_{ \text{t}}\Big{)}\) indicates that the target word \(\text{w}_{\text{t}}\) is used as attention to calculating the weight of the jth sememe in the ith sense of the context word \(\text{w}_{\text{c}}\), as follows:  
\[\text{att}\Big{(}\hat{\text{x}}_{\text{j}}^{(\text{s}_{\text{i}})},\text{w}_{ \text{t}}\Big{)}=\frac{\exp\Big{(}\hat{\text{x}}_{\text{j}}^{(\text{s}_{\text{ i}})}\cdot\text{w}_{\text{t}}\Big{)}}{\sum_{\text{k}=1}^{|\text{X}_{\text{k}}^{( \text{w}_{\text{c}})}|}\exp\Big{(}\hat{\text{x}}_{\text{k}}^{(\text{s}_{\text{ i}})}\cdot\text{w}_{\text{t}}\Big{)}}. \tag{11}\]  
DAWE is a two-layer encoding framework. The first layer is sense encoding, which corresponds to Equation (10) and Equation (11). In the first layer, the sememe embeddings are used as input, and then the sense embeddings are obtained through sememe-level attention. The second layer is word encoding, corresponding to Equation (8) and Equation (9). In the second layer, the sense embeddingsobtained by Equation (10) are used as input, and then the word embeddings are obtained through sense-level attention. In DAC, the target word \(\mathrm{w_{t}}\) is used to guide the generation of word vectors of context words. Under this attention mechanism, if the sememe vectors and sense vectors of the context word are more relevant to the target word vectors, the corresponding sememes and senses will get higher weight. This is similar to the idea in Word2 Vec that the more similar words are closer in the semantic space. In this way, the different senses of context words can be disambiguated too.  
Figure 6: Double attention over context model (DAC).  
Figure 7: Double attention over target model (DAT).