# DAWE: A Double Attention-Based Word Embedding Model with Sememe Structure Information
## 3 Related Works
### Word Embeddings with External Semantic Information  
A lot of semantic information related to words is now emerging, such as images with text labels, as well as some semantic knowledge bases including WordNet [22], BabelNet [23], ConceptNet [24] and HowNet [11]. These semantic data should help us improve the accuracy of word vectors.  
A large and growing body of literature has researched on joining external semantic information for word embedding. [PERSON] et al. [25] proposed a character-level embedding model that attempts to capture the common structure between characters from visual features by using morphological images corresponding to characters. [PERSON], [PERSON] and [PERSON] [26] proposed a word-level embedding model, which uses images from the real world as a complement to text semantics, rather than directly replacing text semantic information with visual feature information. In terms of considering external semantic knowledge base, [PERSON] and [PERSON] [9] used Tongyici Cilin [27] whose purpose is to make the words with the same semantic classification in the Tongyici Cilin close to each other (Tongyici Cilin is a Chinese semantic knowledge base based on synonym sets, which can classify words according to their semantics). [PERSON], [PERSON], [PERSON] and [PERSON] [6] used BabelNet to annotate the different senses of words and then performed joint learning to get word and sense embeddings. [PERSON], [PERSON] and [PERSON] [8] introduced the concepts of \"strong pairs\" and \"weak pairs\" from dictionary entries, so as to better distinguish the relative intensity of word pairs in the semantic space. [PERSON] et al. [28] proposed a knowledge-enabled language representation model with knowledge graphs (KGs), in which KG triples are injected into the sentences as domain knowledge. [PERSON], [PERSON], [PERSON] and [PERSON] [7] proposed the sememe-encoded word representation learning (SE-WRL) model. The SE-WRL model embeds words by encoding sememe in word-sense-sememe knowledge of HowNet. Since the word-sense-sememe is an intuitive form of organizing words, it is easily organized and interpretable [11] and has a wide range of potential uses.  
Specifically, three SE-WRL models are mentioned: simple sememe aggregation model (SSA), sememe attention over context model (SAC) and sememe attention over target model (SAT).  
(1) The SSA model simply represents vector \(w\) of each word as the average of all its sememe vectors, as shown in Equation (6).  
\[w=\frac{1}{m}\sum_{s_{i}^{(w)}\in S^{(w)}}\sum_{\chi_{i}^{(s)}\in\chi_{i}^{(w)}} \chi_{i}^{(s_{i})}, \tag{6}\]  
where \(m\) is the number of sememes of the word \(w\).  
(2) Based on the SSA model, [PERSON], [PERSON], [PERSON] and [PERSON] [7] developed a SAC model and a SAT model that can distinguish different word meanings:  
\[att\Big{(}s_{j}^{(w_{u})},w_{v}\Big{)}=\frac{\exp\!\left(w_{v}s_{j}^{(w_{u})} \right)}{\sum_{k=1}^{[w_{u}]}\exp\!\left(w_{v}s_{k}^{(w_{u})}\right)}. \tag{7}\]  
In the above formula, if \(w_{v}\) represents the target word \(w_{t}\) and \(w_{u}\) represents the context \(w_{c}\), then it is SAC model, while, if \(w_{v}\) represents context \(w_{c}\) and \(w_{u}\) represents the target word \(w_{t}\), then it is the SAT model.  
In this study, we used both sememe and word-sense-sememe structure as external supplements to refine the process of word embedding. Different from SE-WRL, our model captures the weight changes of different sememes under the same sense over different contexts, while using sememe to encode words.