# DAWE: A Double Attention-Based Word Embedding Model with Sememe Structure Information
## 3 Related Works
### Classical Word Embeddings  
Word embedding aims to embed words into continuous low-dimensional, high-density semantic space. Early models usually use an NLM (neural language model) to generate word vectors (word-level word embedding vectors). The typical representative of them is the Word2 Vec, which includes a CBOW model (continuous bag-of-words model) and a Skip-gram model (continuous skip-gram model), as shown in Figure 4. The key idea of Word2 Vec is that the words with similar text contexts (or those words appearing in the same window that slides through the text with a size of k) should be close to each other in the semantic space, that is, their word vectors should be similar. As shown in Figure 3, the words \"first\", \"second\" and \"third\" are close to each other in the semantic space because they have the same context \"This is the \(\sim\) sentence\". Figure 4: Several typical works of word embeddings. Skip-gram is a model which predicts the context words (surrounding words) given a target word (the center word). It intends to maximize the likelihood function as follows:  
\[\mathrm{L}(\mathrm{H})=\sum_{\mathrm{t=k}}^{\mathrm{n-k}}\mathrm{logP}(\mathrm{ w}_{\mathrm{t-k}},\dots,\mathrm{w}_{\mathrm{t+k}}|\mathrm{w}_{\mathrm{t}})\cong \sum_{\mathrm{t=k}}^{\mathrm{n-k}}\mathrm{log}\prod_{\mathrm{w}_{\mathrm{c}} \in\mathrm{C}(\mathrm{w}_{\mathrm{t}})}\mathrm{P}(\mathrm{w}_{\mathrm{c}}| \mathrm{w}_{\mathrm{t}}). \tag{4}\]  
where \(\mathrm{n}\) is the size of the text corpus, that is, the number of words contained in the corpus. \(\mathrm{P}(\mathrm{w}_{\mathrm{t-k}},\dots,\mathrm{w}_{\mathrm{t+k}}|\mathrm{ w}_{\mathrm{t}})\) denotes the probability of the context \([\mathrm{w}_{\mathrm{t-k}},\dots,\mathrm{w}_{\mathrm{t+k}}]\) being predicted by the target word \(\mathrm{w}_{\mathrm{t}}\), \([\mathrm{w}_{\mathrm{t-k}},\dots,\mathrm{w}_{\mathrm{t+k}}]\) is the set of the first and last k words of the current word \(\mathrm{w}_{\mathrm{t}}\) in the text sequence and k is the size of the context window. For example, for the text sequence, \"I twisted an apple off the tree,\" when the target word is \"apple\" and k = 2, then \([\mathrm{w}_{\mathrm{t-k}},\dots,\mathrm{w}_{\mathrm{t+k}}]\) = [twisted, an, off, tree]. Based on the assumption of context independence, the probability of predicting context \([\mathrm{w}_{\mathrm{t-k}},\dots,\mathrm{w}_{\mathrm{t+k}}]\) by the target word \(\mathrm{w}_{\mathrm{t}}\) can be converted to the product of the probability of predicting each word \(\mathrm{w}_{\mathrm{c}}\) in the context (the co-occurrence probability of the target word and the context word): \(\mathrm{P}(\mathrm{w}_{\mathrm{t-k}},\dots,\mathrm{w}_{\mathrm{t+k}}|\mathrm{ w}_{\mathrm{t}})\cong\prod_{\mathrm{w}_{\mathrm{c}}\in\mathrm{C}(\mathrm{w}_{\mathrm{t}})} \mathrm{P}(\mathrm{w}_{\mathrm{c}}|\mathrm{w}_{\mathrm{t}})\), where \(\mathrm{C}(\mathrm{w}_{\mathrm{t}})\) = \([\mathrm{w}_{\mathrm{t-k}},\dots,\mathrm{w}_{\mathrm{t+k}}]\). By introducing the negative sampling [12] method, the co-occurrence probability of each context word and the target word can be formalized by the following:  
\[\mathrm{P}(\mathrm{w}_{\mathrm{c}}|\mathrm{w}_{\mathrm{t}})\cong\sigma\Big{(} \mathrm{w}_{\mathrm{c}}^{\mathrm{T}}\cdot\mathrm{w}_{\mathrm{t}}\Big{)}\prod_{ \mathrm{w}_{\mathrm{t}}^{\prime}\in\mathrm{NEG}(\mathrm{w}_{\mathrm{t}})} \Big{(}1-\sigma\Big{(}\mathrm{w}_{\mathrm{c}}^{\mathrm{T}}\cdot\mathrm{w}_{ \mathrm{t}}^{\prime}\Big{)}\Big{)}, \tag{5}\]  
where \(\sigma(\cdot)\) denotes the sigmoid function and \(\mathrm{NEG}(\mathrm{w}_{\mathrm{t}})\) is the negative word set for the target word \(\mathrm{w}_{\mathrm{t}}\). The objective of negative sampling is to make the context word \(\mathrm{w}_{\mathrm{c}}\) as close as possible to the target word \(\mathrm{w}_{\mathrm{t}}\) in the semantic space and as far away as possible from the negative sample \(\mathrm{w}_{\mathrm{t}}^{\prime}\).